Oct 30 18:33:40.904590 ip-10-0-137-135 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Oct 30 18:33:40.904602 ip-10-0-137-135 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Oct 30 18:33:40.904611 ip-10-0-137-135 systemd[1]: kubelet.service: Failed with result 'resources'.
Oct 30 18:33:40.904914 ip-10-0-137-135 systemd[1]: Failed to start Kubernetes Kubelet.
Oct 30 18:33:51.074010 ip-10-0-137-135 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Oct 30 18:33:51.074033 ip-10-0-137-135 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 12c44ce9f5c7478ea77c65ed7915990b --
Oct 30 18:35:36.415970 ip-10-0-137-135 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 18:35:36.860910 ip-10-0-137-135 kubenswrapper[2573]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 18:35:36.860910 ip-10-0-137-135 kubenswrapper[2573]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 18:35:36.860910 ip-10-0-137-135 kubenswrapper[2573]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 18:35:36.860910 ip-10-0-137-135 kubenswrapper[2573]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 18:35:36.864650 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.864552    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:35:36.864650 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.864645    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:35:36.864650 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.864651    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:35:36.864650 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.864655    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:35:36.864650 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.864659    2573 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:35:36.864957 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.864663    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:35:36.864957 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.864667    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:35:36.864957 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.864671    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:35:36.864957 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.864694    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:35:36.864957 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.864699    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:35:36.864957 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.864702    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:35:36.864957 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.864708    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:35:36.864957 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.864711    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:35:36.864957 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.864715    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:35:36.864957 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.864719    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:35:36.864957 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.864723    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:35:36.864957 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.864727    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:35:36.864957 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.864730    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:35:36.864957 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.864734    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:35:36.864957 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.864738    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:35:36.864957 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.864744    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:35:36.864957 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.864748    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:35:36.864957 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.864753    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:35:36.864957 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.864757    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:35:36.864957 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.864760    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:35:36.865778 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.864764    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:35:36.865778 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.864768    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:35:36.865778 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.864772    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:35:36.865778 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.864776    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:35:36.865778 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.864781    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:35:36.865778 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.864785    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:35:36.865778 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.864789    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:35:36.865778 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.864793    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:35:36.865778 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.864797    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:35:36.865778 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.864803    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:35:36.865778 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.864811    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:35:36.865778 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.864816    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:35:36.865778 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.864821    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:35:36.865778 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.864826    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:35:36.865778 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.864831    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:35:36.865778 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.864835    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:35:36.865778 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.864839    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:35:36.865778 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.864844    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:35:36.865778 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.864848    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:35:36.866441 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.864853    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:35:36.866441 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.864857    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:35:36.866441 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.864862    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:35:36.866441 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.864866    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:35:36.866441 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.864871    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:35:36.866441 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.864875    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:35:36.866441 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.864880    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:35:36.866441 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.864884    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:35:36.866441 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.864889    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:35:36.866441 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.864894    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:35:36.866441 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.864898    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:35:36.866441 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.864903    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:35:36.866441 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.864919    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:35:36.866441 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.864924    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:35:36.866441 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.864929    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:35:36.866441 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.864933    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:35:36.866441 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.864938    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:35:36.866441 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.864942    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:35:36.866441 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.864947    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:35:36.866441 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.864952    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:35:36.866978 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.864956    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:35:36.866978 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.864960    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:35:36.866978 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.864967    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:35:36.866978 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.864972    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:35:36.866978 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.864978    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:35:36.866978 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.864982    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:35:36.866978 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.864987    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:35:36.866978 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.864991    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:35:36.866978 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.864996    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:35:36.866978 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.865001    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:35:36.866978 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.865005    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:35:36.866978 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.865010    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:35:36.866978 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.865015    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:35:36.866978 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.865020    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:35:36.866978 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.865024    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:35:36.866978 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.865029    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:35:36.866978 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.865034    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:35:36.866978 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.865038    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:35:36.866978 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.865045    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:35:36.867481 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.865049    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:35:36.867481 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.865054    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:35:36.867481 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.865058    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:35:36.867481 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.865064    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:35:36.867481 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.865068    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:35:36.867481 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.865072    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:35:36.867481 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.865077    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:35:36.867481 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.865081    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:35:36.867481 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.865086    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:35:36.867481 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.865090    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:35:36.867481 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.865094    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:35:36.867481 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.865099    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:35:36.867481 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.865103    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:35:36.867481 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.865108    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:35:36.867481 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.865113    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:35:36.867481 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.865117    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:35:36.867481 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.865121    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:35:36.867481 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.865125    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:35:36.867481 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.865130    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:35:36.867481 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.865135    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:35:36.868308 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.865139    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:35:36.868308 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.865143    2573 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:35:36.868308 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.865148    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:35:36.868308 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866453    2573 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:35:36.868308 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866468    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:35:36.868308 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866472    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:35:36.868308 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866476    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:35:36.868308 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866479    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:35:36.868308 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866481    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:35:36.868308 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866484    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:35:36.868308 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866487    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:35:36.868308 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866490    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:35:36.868308 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866493    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:35:36.868308 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866496    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:35:36.868308 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866499    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:35:36.868308 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866502    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:35:36.868308 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866505    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:35:36.868308 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866508    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:35:36.868308 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866510    2573 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:35:36.868308 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866513    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:35:36.868912 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866517    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:35:36.868912 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866522    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:35:36.868912 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866525    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:35:36.868912 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866528    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:35:36.868912 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866531    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:35:36.868912 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866534    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:35:36.868912 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866537    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:35:36.868912 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866540    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:35:36.868912 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866543    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:35:36.868912 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866546    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:35:36.868912 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866549    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:35:36.868912 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866553    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:35:36.868912 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866555    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:35:36.868912 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866558    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:35:36.868912 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866561    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:35:36.868912 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866563    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:35:36.868912 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866568    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:35:36.868912 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866571    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:35:36.868912 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866573    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:35:36.868912 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866576    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:35:36.869638 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866579    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:35:36.869638 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866582    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:35:36.869638 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866584    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:35:36.869638 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866587    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:35:36.869638 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866589    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:35:36.869638 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866592    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:35:36.869638 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866595    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:35:36.869638 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866598    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:35:36.869638 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866601    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:35:36.869638 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866603    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:35:36.869638 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866606    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:35:36.869638 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866609    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:35:36.869638 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866611    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:35:36.869638 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866614    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:35:36.869638 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866616    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:35:36.869638 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866619    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:35:36.869638 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866621    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:35:36.869638 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866624    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:35:36.869638 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866627    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:35:36.870131 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866630    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:35:36.870131 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866633    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:35:36.870131 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866636    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:35:36.870131 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866638    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:35:36.870131 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866641    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:35:36.870131 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866644    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:35:36.870131 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866647    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:35:36.870131 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866649    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:35:36.870131 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866652    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:35:36.870131 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866655    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:35:36.870131 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866658    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:35:36.870131 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866660    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:35:36.870131 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866663    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:35:36.870131 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866666    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:35:36.870131 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866668    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:35:36.870131 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866671    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:35:36.870131 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866673    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:35:36.870131 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866702    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:35:36.870131 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866706    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:35:36.870131 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866708    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:35:36.870638 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866711    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:35:36.870638 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866714    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:35:36.870638 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866718    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:35:36.870638 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866721    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:35:36.870638 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866723    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:35:36.870638 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866726    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:35:36.870638 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866729    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:35:36.870638 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866732    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:35:36.870638 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866734    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:35:36.870638 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866737    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:35:36.870638 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866740    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:35:36.870638 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866742    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:35:36.870638 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866745    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:35:36.870638 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866748    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:35:36.870638 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866751    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:35:36.870638 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866754    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:35:36.870638 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866757    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:35:36.870638 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866759    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:35:36.870638 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866762    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:35:36.870638 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866764    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:35:36.871140 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866769    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:35:36.871140 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866773    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:35:36.871140 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866776    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:35:36.871140 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866780    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:35:36.871140 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866783    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:35:36.871140 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866785    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:35:36.871140 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866788    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:35:36.871140 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866790    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:35:36.871140 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866793    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:35:36.871140 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.866795    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:35:36.871140 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.867510    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:35:36.871140 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.867529    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:35:36.871140 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.867539    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:35:36.871140 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.867547    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:35:36.871140 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.867562    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:35:36.871140 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.867570    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:35:36.871140 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.867578    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:35:36.871140 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.867585    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:35:36.871140 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.867593    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:35:36.871579 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.867600    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:35:36.871579 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.867606    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:35:36.871579 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.867614    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:35:36.871579 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.867621    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:35:36.871579 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.867628    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:35:36.871579 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.867636    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:35:36.871579 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.867646    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:35:36.871579 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.867662    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:35:36.871579 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.867671    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:35:36.871579 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.867695    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:35:36.871579 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.867704    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:35:36.871579 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.867712    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:35:36.871579 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.867719    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:35:36.871579 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.867726    2573 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:35:36.871579 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.867734    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:35:36.871579 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.867742    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:35:36.871579 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.867750    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:35:36.871579 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.867757    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:35:36.871579 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.867765    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:35:36.872221 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.867777    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:35:36.872221 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.867784    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:35:36.872221 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.867792    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:35:36.872221 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.867800    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:35:36.872221 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.867808    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:35:36.872221 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.867815    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:35:36.872221 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.867822    2573 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:35:36.872221 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.867829    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:35:36.872221 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.867836    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:35:36.872221 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.867844    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:35:36.872221 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.867851    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:35:36.872221 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.867858    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:35:36.872221 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.867865    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:35:36.872221 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.867900    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:35:36.872221 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.867917    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:35:36.872221 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.867923    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:35:36.872221 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.867928    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:35:36.872221 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.867932    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:35:36.872221 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.867938    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:35:36.872221 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.867947    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:35:36.872862 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.867952    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:35:36.872862 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.867957    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:35:36.872862 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.867961    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:35:36.872862 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.867966    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:35:36.872862 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.867971    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:35:36.872862 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.867981    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:35:36.872862 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.867985    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:35:36.872862 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.867990    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:35:36.872862 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.867994    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:35:36.872862 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.867998    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:35:36.872862 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.868003    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:35:36.872862 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.868008    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:35:36.872862 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.868012    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:35:36.872862 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.868019    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:35:36.872862 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.868024    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:35:36.872862 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.868028    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:35:36.872862 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.868032    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:35:36.872862 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.868041    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:35:36.872862 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.868045    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:35:36.872862 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.868049    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:35:36.873339 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.868054    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:35:36.873339 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.868059    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:35:36.873339 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.868064    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:35:36.873339 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.868068    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:35:36.873339 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.868072    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:35:36.873339 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.868077    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:35:36.873339 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.868084    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:35:36.873339 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.868090    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:35:36.873339 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.868095    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:35:36.873339 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.868101    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:35:36.873339 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.868111    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:35:36.873339 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.868116    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:35:36.873339 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.868120    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:35:36.873339 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.868124    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:35:36.873339 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.868128    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:35:36.873339 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.868133    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:35:36.873339 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.868138    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:35:36.873339 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.868142    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:35:36.873339 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.868146    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:35:36.873879 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.868150    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:35:36.873879 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.868155    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:35:36.873879 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.868159    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:35:36.873879 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.868163    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:35:36.873879 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.868172    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:35:36.873879 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.868177    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:35:36.873879 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.868181    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:35:36.873879 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.868187    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:35:36.873879 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.868191    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:35:36.873879 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.868195    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:35:36.873879 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.868200    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:35:36.873879 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.868204    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:35:36.873879 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.868209    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:35:36.873879 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.868213    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:35:36.873879 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.868218    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:35:36.873879 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.868222    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:35:36.873879 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.868231    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:35:36.873879 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.868235    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:35:36.873879 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.868243    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:35:36.873879 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869279    2573 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 18:35:36.874356 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869487    2573 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 18:35:36.874356 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869499    2573 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 18:35:36.874356 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869506    2573 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 18:35:36.874356 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869511    2573 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 18:35:36.874356 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869515    2573 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 18:35:36.874356 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869520    2573 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 18:35:36.874356 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869524    2573 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 18:35:36.874356 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869528    2573 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 18:35:36.874356 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869531    2573 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 18:35:36.874356 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869534    2573 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 18:35:36.874356 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869538    2573 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 18:35:36.874356 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869541    2573 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 18:35:36.874356 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869544    2573 flags.go:64] FLAG: --cgroup-root=""
Oct 30 18:35:36.874356 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869547    2573 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 18:35:36.874356 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869550    2573 flags.go:64] FLAG: --client-ca-file=""
Oct 30 18:35:36.874356 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869554    2573 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 18:35:36.874356 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869557    2573 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 18:35:36.874356 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869563    2573 flags.go:64] FLAG: --cluster-domain=""
Oct 30 18:35:36.874356 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869566    2573 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 18:35:36.874356 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869569    2573 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 18:35:36.874356 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869573    2573 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 18:35:36.874356 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869577    2573 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 18:35:36.874356 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869589    2573 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 18:35:36.874356 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869592    2573 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 18:35:36.874944 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869596    2573 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 18:35:36.874944 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869599    2573 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 18:35:36.874944 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869602    2573 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 18:35:36.874944 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869606    2573 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 18:35:36.874944 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869609    2573 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 18:35:36.874944 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869613    2573 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 18:35:36.874944 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869616    2573 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 18:35:36.874944 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869621    2573 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 18:35:36.874944 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869625    2573 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 18:35:36.874944 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869628    2573 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 18:35:36.874944 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869631    2573 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 18:35:36.874944 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869634    2573 flags.go:64] FLAG: --enable-server="true"
Oct 30 18:35:36.874944 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869637    2573 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 18:35:36.874944 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869642    2573 flags.go:64] FLAG: --event-burst="100"
Oct 30 18:35:36.874944 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869646    2573 flags.go:64] FLAG: --event-qps="50"
Oct 30 18:35:36.874944 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869649    2573 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 18:35:36.874944 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869652    2573 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 18:35:36.874944 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869655    2573 flags.go:64] FLAG: --eviction-hard=""
Oct 30 18:35:36.874944 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869660    2573 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 18:35:36.874944 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869664    2573 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 18:35:36.874944 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869667    2573 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 18:35:36.874944 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869670    2573 flags.go:64] FLAG: --eviction-soft=""
Oct 30 18:35:36.874944 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869673    2573 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 18:35:36.874944 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869687    2573 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 18:35:36.874944 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869691    2573 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 18:35:36.875593 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869694    2573 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 18:35:36.875593 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869697    2573 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 18:35:36.875593 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869701    2573 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 18:35:36.875593 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869704    2573 flags.go:64] FLAG: --feature-gates=""
Oct 30 18:35:36.875593 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869708    2573 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 18:35:36.875593 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869711    2573 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 18:35:36.875593 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869715    2573 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 18:35:36.875593 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869718    2573 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 18:35:36.875593 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869722    2573 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 18:35:36.875593 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869725    2573 flags.go:64] FLAG: --help="false"
Oct 30 18:35:36.875593 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869729    2573 flags.go:64] FLAG: --hostname-override="ip-10-0-137-135.us-east-2.compute.internal"
Oct 30 18:35:36.875593 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869732    2573 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 18:35:36.875593 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869735    2573 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 18:35:36.875593 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869739    2573 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 18:35:36.875593 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869742    2573 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 18:35:36.875593 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869746    2573 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 18:35:36.875593 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869749    2573 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 18:35:36.875593 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869752    2573 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 18:35:36.875593 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869755    2573 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 18:35:36.875593 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869758    2573 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 18:35:36.875593 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869761    2573 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 18:35:36.875593 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869764    2573 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 18:35:36.875593 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869767    2573 flags.go:64] FLAG: --kube-reserved=""
Oct 30 18:35:36.875593 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869770    2573 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 18:35:36.876179 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869772    2573 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 18:35:36.876179 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869775    2573 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 18:35:36.876179 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869778    2573 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 18:35:36.876179 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869781    2573 flags.go:64] FLAG: --lock-file=""
Oct 30 18:35:36.876179 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869784    2573 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 18:35:36.876179 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869787    2573 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 18:35:36.876179 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869790    2573 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 18:35:36.876179 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869796    2573 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 18:35:36.876179 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869799    2573 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 18:35:36.876179 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869802    2573 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 18:35:36.876179 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869805    2573 flags.go:64] FLAG: --logging-format="text"
Oct 30 18:35:36.876179 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869808    2573 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 18:35:36.876179 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869812    2573 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 18:35:36.876179 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869814    2573 flags.go:64] FLAG: --manifest-url=""
Oct 30 18:35:36.876179 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869818    2573 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 18:35:36.876179 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869823    2573 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 18:35:36.876179 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869826    2573 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 18:35:36.876179 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869831    2573 flags.go:64] FLAG: --max-pods="110"
Oct 30 18:35:36.876179 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869834    2573 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 18:35:36.876179 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869837    2573 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 18:35:36.876179 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869840    2573 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 18:35:36.876179 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869843    2573 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 18:35:36.876179 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869847    2573 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 18:35:36.876179 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869850    2573 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 18:35:36.876179 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869853    2573 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 18:35:36.876795 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869859    2573 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 18:35:36.876795 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869862    2573 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 18:35:36.876795 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869865    2573 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 18:35:36.876795 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869868    2573 flags.go:64] FLAG: --pod-cidr=""
Oct 30 18:35:36.876795 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869871    2573 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 18:35:36.876795 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869874    2573 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 18:35:36.876795 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869877    2573 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 18:35:36.876795 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869881    2573 flags.go:64] FLAG: --port="10250"
Oct 30 18:35:36.876795 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869884    2573 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 18:35:36.876795 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869887    2573 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0e74aff2ec6d34e9d"
Oct 30 18:35:36.876795 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869891    2573 flags.go:64] FLAG: --qos-reserved=""
Oct 30 18:35:36.876795 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869894    2573 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 18:35:36.876795 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869897    2573 flags.go:64] FLAG: --register-node="true"
Oct 30 18:35:36.876795 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869900    2573 flags.go:64] FLAG: --register-with-taints=""
Oct 30 18:35:36.876795 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869905    2573 flags.go:64] FLAG: --registry-burst="10"
Oct 30 18:35:36.876795 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869907    2573 flags.go:64] FLAG: --registry-qps="5"
Oct 30 18:35:36.876795 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869910    2573 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 18:35:36.876795 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869913    2573 flags.go:64] FLAG: --reserved-memory=""
Oct 30 18:35:36.876795 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869918    2573 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 18:35:36.876795 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869921    2573 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 18:35:36.876795 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869926    2573 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 18:35:36.876795 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869929    2573 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 18:35:36.876795 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869932    2573 flags.go:64] FLAG: --runonce="false"
Oct 30 18:35:36.876795 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869935    2573 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 18:35:36.876795 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869939    2573 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 18:35:36.876795 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869942    2573 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 18:35:36.876795 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869945    2573 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 18:35:36.877496 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869948    2573 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 18:35:36.877496 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869951    2573 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 18:35:36.877496 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869954    2573 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 18:35:36.877496 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869957    2573 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 18:35:36.877496 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869960    2573 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 18:35:36.877496 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869963    2573 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 18:35:36.877496 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869966    2573 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 18:35:36.877496 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869970    2573 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 18:35:36.877496 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869973    2573 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 18:35:36.877496 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869976    2573 flags.go:64] FLAG: --system-cgroups=""
Oct 30 18:35:36.877496 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869979    2573 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 18:35:36.877496 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869985    2573 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 18:35:36.877496 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869988    2573 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 18:35:36.877496 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.869991    2573 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 18:35:36.877496 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.870019    2573 flags.go:64] FLAG: --tls-min-version=""
Oct 30 18:35:36.877496 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.870022    2573 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 18:35:36.877496 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.870026    2573 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 18:35:36.877496 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.870030    2573 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 18:35:36.877496 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.870033    2573 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 18:35:36.877496 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.870036    2573 flags.go:64] FLAG: --v="2"
Oct 30 18:35:36.877496 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.870041    2573 flags.go:64] FLAG: --version="false"
Oct 30 18:35:36.877496 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.870045    2573 flags.go:64] FLAG: --vmodule=""
Oct 30 18:35:36.877496 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.870050    2573 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 18:35:36.877496 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.870053    2573 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 18:35:36.877496 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870219    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:35:36.878104 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870225    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:35:36.878104 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870229    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:35:36.878104 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870232    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:35:36.878104 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870235    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:35:36.878104 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870238    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:35:36.878104 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870241    2573 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:35:36.878104 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870244    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:35:36.878104 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870247    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:35:36.878104 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870249    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:35:36.878104 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870253    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:35:36.878104 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870256    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:35:36.878104 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870258    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:35:36.878104 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870261    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:35:36.878104 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870264    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:35:36.878104 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870266    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:35:36.878104 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870269    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:35:36.878104 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870271    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:35:36.878104 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870274    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:35:36.878104 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870277    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:35:36.878104 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870280    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:35:36.878584 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870282    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:35:36.878584 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870285    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:35:36.878584 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870288    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:35:36.878584 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870295    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:35:36.878584 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870299    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:35:36.878584 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870302    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:35:36.878584 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870304    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:35:36.878584 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870307    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:35:36.878584 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870310    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:35:36.878584 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870312    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:35:36.878584 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870315    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:35:36.878584 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870317    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:35:36.878584 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870322    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:35:36.878584 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870325    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:35:36.878584 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870327    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:35:36.878584 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870330    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:35:36.878584 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870333    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:35:36.878584 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870335    2573 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:35:36.878584 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870338    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:35:36.878584 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870342    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:35:36.879081 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870345    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:35:36.879081 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870348    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:35:36.879081 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870350    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:35:36.879081 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870353    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:35:36.879081 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870356    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:35:36.879081 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870358    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:35:36.879081 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870361    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:35:36.879081 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870364    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:35:36.879081 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870366    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:35:36.879081 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870369    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:35:36.879081 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870371    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:35:36.879081 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870374    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:35:36.879081 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870376    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:35:36.879081 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870379    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:35:36.879081 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870382    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:35:36.879081 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870384    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:35:36.879081 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870387    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:35:36.879081 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870391    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:35:36.879081 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870394    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:35:36.879081 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870396    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:35:36.879559 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870399    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:35:36.879559 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870401    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:35:36.879559 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870404    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:35:36.879559 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870406    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:35:36.879559 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870411    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:35:36.879559 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870414    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:35:36.879559 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870417    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:35:36.879559 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870419    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:35:36.879559 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870422    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:35:36.879559 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870425    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:35:36.879559 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870427    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:35:36.879559 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870433    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:35:36.879559 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870435    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:35:36.879559 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870438    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:35:36.879559 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870441    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:35:36.879559 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870444    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:35:36.879559 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870448    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:35:36.879559 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870451    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:35:36.879559 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870454    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:35:36.880053 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870457    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:35:36.880053 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870460    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:35:36.880053 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870462    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:35:36.880053 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870465    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:35:36.880053 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870468    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:35:36.880053 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870470    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:35:36.880053 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870473    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:35:36.880053 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870475    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:35:36.880053 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870479    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:35:36.880053 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870481    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:35:36.880053 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870484    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:35:36.880053 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870487    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:35:36.880053 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870490    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:35:36.880053 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870493    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:35:36.880053 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870495    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:35:36.880053 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870498    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:35:36.880053 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870500    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:35:36.880053 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870504    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:35:36.880053 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870507    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:35:36.880499 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870510    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:35:36.880499 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870512    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:35:36.880499 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870516    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:35:36.880499 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870520    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:35:36.880499 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870523    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:35:36.880499 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870527    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:35:36.880499 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870530    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:35:36.880499 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.870557    2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 18:35:36.880499 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870600    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:35:36.880499 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870604    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:35:36.880499 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870607    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:35:36.880499 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870610    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:35:36.880499 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870612    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:35:36.880499 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870615    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:35:36.880499 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870618    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:35:36.880874 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870621    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:35:36.880874 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870623    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:35:36.880874 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870626    2573 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:35:36.880874 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870629    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:35:36.880874 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870632    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:35:36.880874 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870634    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:35:36.880874 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870637    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:35:36.880874 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870640    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:35:36.880874 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870643    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:35:36.880874 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870646    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:35:36.880874 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870649    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:35:36.880874 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870652    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:35:36.880874 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870655    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:35:36.880874 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870657    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:35:36.880874 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870660    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:35:36.880874 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870663    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:35:36.880874 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870666    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:35:36.880874 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870669    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:35:36.880874 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870671    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:35:36.880874 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870674    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:35:36.881330 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870699    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:35:36.881330 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870702    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:35:36.881330 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870705    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:35:36.881330 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870709    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:35:36.881330 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870712    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:35:36.881330 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870715    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:35:36.881330 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870717    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:35:36.881330 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870720    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:35:36.881330 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870723    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:35:36.881330 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870725    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:35:36.881330 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870728    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:35:36.881330 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870731    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:35:36.881330 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870733    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:35:36.881330 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870736    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:35:36.881330 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870739    2573 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:35:36.881330 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870741    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:35:36.881330 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870744    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:35:36.881330 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870747    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:35:36.881330 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870749    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:35:36.881330 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870752    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:35:36.881808 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870755    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:35:36.881808 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870757    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:35:36.881808 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870760    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:35:36.881808 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870763    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:35:36.881808 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870766    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:35:36.881808 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870769    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:35:36.881808 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870772    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:35:36.881808 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870774    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:35:36.881808 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870777    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:35:36.881808 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870779    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:35:36.881808 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870782    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:35:36.881808 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870785    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:35:36.881808 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870787    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:35:36.881808 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870790    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:35:36.881808 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870792    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:35:36.881808 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870796    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:35:36.881808 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870799    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:35:36.881808 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870802    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:35:36.881808 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870804    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:35:36.881808 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870807    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:35:36.882273 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870809    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:35:36.882273 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870812    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:35:36.882273 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870815    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:35:36.882273 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870817    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:35:36.882273 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870821    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:35:36.882273 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870823    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:35:36.882273 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870826    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:35:36.882273 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870828    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:35:36.882273 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870831    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:35:36.882273 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870833    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:35:36.882273 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870836    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:35:36.882273 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870839    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:35:36.882273 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870841    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:35:36.882273 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870845    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:35:36.882273 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870849    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:35:36.882273 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870852    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:35:36.882273 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870856    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:35:36.882273 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870859    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:35:36.882273 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870862    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:35:36.882730 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870865    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:35:36.882730 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870867    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:35:36.882730 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870870    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:35:36.882730 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870873    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:35:36.882730 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870875    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:35:36.882730 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870878    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:35:36.882730 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870881    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:35:36.882730 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870883    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:35:36.882730 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870887    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:35:36.882730 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870890    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:35:36.882730 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870892    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:35:36.882730 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870895    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:35:36.882730 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870897    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:35:36.882730 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870900    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:35:36.882730 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870903    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:35:36.882730 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870905    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:35:36.882730 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870908    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:35:36.882730 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870910    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:35:36.882730 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870915    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:35:36.883178 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.870919    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:35:36.883178 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.878772    2573 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 18:35:36.883178 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.878786    2573 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 18:35:36.883178 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.878797    2573 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 18:35:36.883178 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.878802    2573 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 18:35:36.883178 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.878857    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:35:36.883178 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.878861    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:35:36.883178 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.878865    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:35:36.883178 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.878868    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:35:36.883178 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.878871    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:35:36.883178 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.878875    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:35:36.883178 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.878878    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:35:36.883178 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.878881    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:35:36.883178 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.878883    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:35:36.883178 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.878886    2573 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:35:36.883178 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.878889    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:35:36.883178 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.878891    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:35:36.883178 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.878894    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:35:36.883178 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.878896    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:35:36.883178 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.878899    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:35:36.883640 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.878902    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:35:36.883640 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.878904    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:35:36.883640 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.878907    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:35:36.883640 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.878909    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:35:36.883640 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.878912    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:35:36.883640 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.878914    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:35:36.883640 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.878917    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:35:36.883640 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.878920    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:35:36.883640 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.878923    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:35:36.883640 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.878925    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:35:36.883640 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.878928    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:35:36.883640 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.878931    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:35:36.883640 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.878933    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:35:36.883640 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.878936    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:35:36.883640 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.878939    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:35:36.883640 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.878943    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:35:36.883640 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.878945    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:35:36.883640 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.878948    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:35:36.883640 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.878951    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:35:36.883640 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.878953    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:35:36.884115 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.878957    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:35:36.884115 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.878962    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:35:36.884115 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.878965    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:35:36.884115 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.878969    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:35:36.884115 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.878972    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:35:36.884115 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.878975    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:35:36.884115 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.878980    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:35:36.884115 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.878983    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:35:36.884115 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.878986    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:35:36.884115 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.878989    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:35:36.884115 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.878992    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:35:36.884115 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.878994    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:35:36.884115 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.878997    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:35:36.884115 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.878999    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:35:36.884115 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879002    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:35:36.884115 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879004    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:35:36.884115 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879007    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:35:36.884115 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879010    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:35:36.884115 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879012    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:35:36.884553 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879015    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:35:36.884553 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879017    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:35:36.884553 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879020    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:35:36.884553 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879023    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:35:36.884553 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879025    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:35:36.884553 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879029    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:35:36.884553 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879032    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:35:36.884553 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879035    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:35:36.884553 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879037    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:35:36.884553 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879040    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:35:36.884553 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879043    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:35:36.884553 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879045    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:35:36.884553 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879048    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:35:36.884553 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879051    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:35:36.884553 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879053    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:35:36.884553 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879056    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:35:36.884553 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879058    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:35:36.884553 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879061    2573 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:35:36.884553 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879064    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:35:36.885166 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879066    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:35:36.885166 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879069    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:35:36.885166 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879072    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:35:36.885166 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879074    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:35:36.885166 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879077    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:35:36.885166 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879079    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:35:36.885166 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879082    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:35:36.885166 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879085    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:35:36.885166 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879088    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:35:36.885166 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879091    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:35:36.885166 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879093    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:35:36.885166 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879096    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:35:36.885166 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879099    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:35:36.885166 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879102    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:35:36.885166 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879104    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:35:36.885166 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879107    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:35:36.885166 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879109    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:35:36.885166 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879112    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:35:36.885166 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879114    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:35:36.885600 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879117    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:35:36.885600 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879120    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:35:36.885600 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879122    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:35:36.885600 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879125    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:35:36.885600 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879127    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:35:36.885600 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879130    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:35:36.885600 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879133    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:35:36.885600 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879135    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:35:36.885600 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879138    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:35:36.885600 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879141    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:35:36.885600 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879144    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:35:36.885600 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879146    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:35:36.885600 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879150    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:35:36.885600 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879152    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:35:36.885600 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.879180    2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 18:35:36.885600 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879338    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:35:36.885987 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879344    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:35:36.885987 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879347    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:35:36.885987 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879351    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:35:36.885987 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879354    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:35:36.885987 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879357    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:35:36.885987 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879359    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:35:36.885987 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879362    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:35:36.885987 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879365    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:35:36.885987 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879367    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:35:36.885987 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879370    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:35:36.885987 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879373    2573 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:35:36.885987 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879375    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:35:36.885987 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879378    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:35:36.885987 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879381    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:35:36.885987 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879383    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:35:36.885987 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879386    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:35:36.885987 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879389    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:35:36.885987 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879391    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:35:36.885987 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879394    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:35:36.885987 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879396    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:35:36.886445 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879399    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:35:36.886445 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879401    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:35:36.886445 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879404    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:35:36.886445 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879406    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:35:36.886445 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879409    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:35:36.886445 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879411    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:35:36.886445 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879414    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:35:36.886445 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879417    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:35:36.886445 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879419    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:35:36.886445 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879422    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:35:36.886445 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879425    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:35:36.886445 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879427    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:35:36.886445 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879430    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:35:36.886445 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879433    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:35:36.886445 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879435    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:35:36.886445 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879438    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:35:36.886445 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879440    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:35:36.886445 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879443    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:35:36.886445 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879445    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:35:36.886445 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879448    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:35:36.886938 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879450    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:35:36.886938 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879453    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:35:36.886938 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879456    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:35:36.886938 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879458    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:35:36.886938 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879461    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:35:36.886938 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879463    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:35:36.886938 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879466    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:35:36.886938 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879469    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:35:36.886938 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879471    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:35:36.886938 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879474    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:35:36.886938 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879477    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:35:36.886938 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879479    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:35:36.886938 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879482    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:35:36.886938 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879484    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:35:36.886938 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879487    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:35:36.886938 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879489    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:35:36.886938 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879492    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:35:36.886938 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879494    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:35:36.886938 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879497    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:35:36.886938 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879499    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:35:36.887398 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879502    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:35:36.887398 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879504    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:35:36.887398 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879508    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:35:36.887398 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879510    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:35:36.887398 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879513    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:35:36.887398 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879515    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:35:36.887398 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879518    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:35:36.887398 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879520    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:35:36.887398 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879523    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:35:36.887398 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879525    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:35:36.887398 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879528    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:35:36.887398 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879531    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:35:36.887398 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879535    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:35:36.887398 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879538    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:35:36.887398 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879542    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:35:36.887398 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879545    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:35:36.887398 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879548    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:35:36.887398 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879550    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:35:36.887398 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879553    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:35:36.887855 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879556    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:35:36.887855 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879558    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:35:36.887855 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879561    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:35:36.887855 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879563    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:35:36.887855 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879566    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:35:36.887855 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879569    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:35:36.887855 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879572    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:35:36.887855 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879574    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:35:36.887855 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879577    2573 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:35:36.887855 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879580    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:35:36.887855 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879582    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:35:36.887855 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879584    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:35:36.887855 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879587    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:35:36.887855 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879591    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:35:36.887855 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879595    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:35:36.887855 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879598    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:35:36.887855 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879605    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:35:36.887855 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879609    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:35:36.887855 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879611    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:35:36.888325 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879614    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:35:36.888325 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879616    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:35:36.888325 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879619    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:35:36.888325 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879621    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:35:36.888325 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879624    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:35:36.888325 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879626    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:35:36.888325 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879629    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:35:36.888325 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.879654    2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 18:35:36.888325 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879713    2573 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:35:36.888325 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879719    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:35:36.888325 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879722    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:35:36.888325 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879725    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:35:36.888325 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879728    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:35:36.888325 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879730    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:35:36.888325 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879733    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:35:36.888325 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879736    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:35:36.888724 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879738    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:35:36.888724 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879741    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:35:36.888724 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879743    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:35:36.888724 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879746    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:35:36.888724 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879749    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:35:36.888724 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879751    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:35:36.888724 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879753    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:35:36.888724 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879756    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:35:36.888724 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879758    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:35:36.888724 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879761    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:35:36.888724 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879764    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:35:36.888724 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879766    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:35:36.888724 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879769    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:35:36.888724 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879771    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:35:36.888724 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879774    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:35:36.888724 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879777    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:35:36.888724 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879779    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:35:36.888724 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879782    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:35:36.888724 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879784    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:35:36.888724 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879787    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:35:36.889179 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879789    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:35:36.889179 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879792    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:35:36.889179 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879795    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:35:36.889179 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879798    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:35:36.889179 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879800    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:35:36.889179 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879803    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:35:36.889179 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879805    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:35:36.889179 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879808    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:35:36.889179 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879811    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:35:36.889179 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879813    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:35:36.889179 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879816    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:35:36.889179 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879819    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:35:36.889179 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879822    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:35:36.889179 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879824    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:35:36.889179 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879827    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:35:36.889179 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879829    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:35:36.889179 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879832    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:35:36.889179 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879834    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:35:36.889179 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879837    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:35:36.889179 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879839    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:35:36.889632 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879842    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:35:36.889632 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879844    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:35:36.889632 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879846    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:35:36.889632 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879849    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:35:36.889632 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879851    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:35:36.889632 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879854    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:35:36.889632 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879856    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:35:36.889632 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879859    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:35:36.889632 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879862    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:35:36.889632 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879864    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:35:36.889632 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879867    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:35:36.889632 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879869    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:35:36.889632 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879872    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:35:36.889632 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879875    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:35:36.889632 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879879    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:35:36.889632 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879883    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:35:36.889632 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879887    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:35:36.889632 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879890    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:35:36.889632 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879892    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:35:36.890237 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879894    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:35:36.890237 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879897    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:35:36.890237 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879900    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:35:36.890237 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879902    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:35:36.890237 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879905    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:35:36.890237 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879907    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:35:36.890237 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879910    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:35:36.890237 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879913    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:35:36.890237 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879915    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:35:36.890237 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879918    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:35:36.890237 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879921    2573 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:35:36.890237 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879924    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:35:36.890237 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879926    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:35:36.890237 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879929    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:35:36.890237 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879931    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:35:36.890237 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879935    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:35:36.890237 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879938    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:35:36.890237 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879941    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:35:36.890237 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879944    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:35:36.890701 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879947    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:35:36.890701 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879949    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:35:36.890701 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879955    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:35:36.890701 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879958    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:35:36.890701 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879961    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:35:36.890701 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879963    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:35:36.890701 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879966    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:35:36.890701 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879968    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:35:36.890701 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879971    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:35:36.890701 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879974    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:35:36.890701 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879976    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:35:36.890701 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879979    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:35:36.890701 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879981    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:35:36.890701 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879984    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:35:36.890701 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879988    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:35:36.890701 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879990    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:35:36.890701 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879993    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:35:36.890701 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879995    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:35:36.890701 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.879998    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:35:36.891147 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:36.880000    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:35:36.891147 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.880053    2573 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 18:35:36.891147 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.882474    2573 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 18:35:36.891147 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.883576    2573 server.go:1014] "Starting client certificate rotation"
Oct 30 18:35:36.891147 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.883675    2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 18:35:36.891147 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.884544    2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 18:35:36.891147 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.885922    2573 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 18:35:36.891147 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.887043    2573 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 18:35:36.901035 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.901016    2573 log.go:25] "Validated CRI v1 runtime API"
Oct 30 18:35:36.906132 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.906113    2573 log.go:25] "Validated CRI v1 image API"
Oct 30 18:35:36.906949 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.906914    2573 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 18:35:36.910473 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.910451    2573 fs.go:135] Filesystem UUIDs: map[1a093d95-fcdf-4895-a53f-68dfb0bce584:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 8fe4a33a-1be2-41cb-aecd-197927ea84cf:/dev/nvme0n1p4]
Oct 30 18:35:36.910549 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.910472    2573 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 18:35:36.917285 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.917120    2573 manager.go:217] Machine: {Timestamp:2026-10-30 18:35:36.915544532 +0000 UTC m=+0.380482811 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100390 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec276576394ce06bb5ead00377771a31 SystemUUID:ec276576-394c-e06b-b5ea-d00377771a31 BootID:12c44ce9-f5c7-478e-a77c-65ed7915990b Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d9:61:9b:59 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d9:61:9b:59 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:f6:1a:d0:ea:82:48 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 18:35:36.917389 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.917288    2573 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 18:35:36.917473 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.917456    2573 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 18:35:36.921444 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.921421    2573 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 18:35:36.921576 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.921446    2573 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-135.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 18:35:36.921620 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.921579    2573 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 18:35:36.921620 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.921587    2573 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 18:35:36.921668 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.921632    2573 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 18:35:36.921668 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.921649    2573 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 18:35:36.922889 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.922878    2573 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 18:35:36.927556 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.927544    2573 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 18:35:36.927673 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.927663    2573 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 18:35:36.927720 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.927711    2573 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 18:35:36.927749 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.927721    2573 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 18:35:36.927749 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.927730    2573 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 18:35:36.927749 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.927741    2573 kubelet.go:405] "Adding apiserver pod source"
Oct 30 18:35:36.927749 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.927750    2573 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 18:35:36.928102 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.928091    2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 18:35:36.930622 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.930605    2573 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 18:35:36.937633 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.937619    2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 18:35:36.938834 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.938820    2573 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 18:35:36.938880 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.938852    2573 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 18:35:36.939008 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.938997    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 18:35:36.939041 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.939014    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 18:35:36.939041 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.939021    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 18:35:36.939041 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.939028    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 18:35:36.939041 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.939034    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 18:35:36.939041 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.939039    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 18:35:36.939724 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.939715    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 18:35:36.939759 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.939724    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 18:35:36.939759 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.939731    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 18:35:36.939759 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.939737    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 18:35:36.939759 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.939752    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 18:35:36.939759 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.939761    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 18:35:36.941243 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.941219    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 18:35:36.941243 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.941236    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 18:35:36.943716 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:36.943690    2573 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 30 18:35:36.943764 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:36.943718    2573 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-135.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 30 18:35:36.943764 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.943724    2573 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-135.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 18:35:36.945250 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.945238    2573 server.go:1263] "Started kubelet"
Oct 30 18:35:36.945396 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.945372    2573 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 18:35:36.945471 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.945354    2573 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 18:35:36.945527 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.945496    2573 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 18:35:36.946101 ip-10-0-137-135 systemd[1]: Started Kubernetes Kubelet.
Oct 30 18:35:36.947201 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.947068    2573 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 18:35:36.949072 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.949056    2573 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 18:35:36.951945 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:36.950830    2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-135.us-east-2.compute.internal.18e3627652035609  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-135.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-135.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 18:35:36.945202697 +0000 UTC m=+0.410140975,LastTimestamp:2026-10-30 18:35:36.945202697 +0000 UTC m=+0.410140975,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-135.us-east-2.compute.internal,}"
Oct 30 18:35:36.953430 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:36.953399    2573 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 18:35:36.956572 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.956553    2573 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 18:35:36.956572 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.956555    2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 18:35:36.956724 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:36.956655    2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-135.us-east-2.compute.internal\" not found"
Oct 30 18:35:36.957304 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.957286    2573 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 18:35:36.957378 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.957324    2573 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 18:35:36.957378 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.957342    2573 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 18:35:36.957489 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.957474    2573 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 18:35:36.957489 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.957481    2573 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 18:35:36.959910 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.959442    2573 factory.go:153] Registering CRI-O factory
Oct 30 18:35:36.960018 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.959998    2573 factory.go:223] Registration of the crio container factory successfully
Oct 30 18:35:36.960116 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.960062    2573 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 18:35:36.960116 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.960073    2573 factory.go:55] Registering systemd factory
Oct 30 18:35:36.960116 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.960083    2573 factory.go:223] Registration of the systemd container factory successfully
Oct 30 18:35:36.960116 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.960102    2573 factory.go:103] Registering Raw factory
Oct 30 18:35:36.960116 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.960115    2573 manager.go:1199] Started watching for new ooms in manager
Oct 30 18:35:36.961351 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.961336    2573 manager.go:319] Starting recovery of all containers
Oct 30 18:35:36.969779 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.969765    2573 manager.go:324] Recovery completed
Oct 30 18:35:36.971369 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:36.971348    2573 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Oct 30 18:35:36.974337 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.974324    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:35:36.975560 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:36.975539    2573 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-135.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 18:35:36.975560 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:36.975562    2573 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 30 18:35:36.976498 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.976483    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-135.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:35:36.976552 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.976516    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-135.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:35:36.976552 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.976527    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-135.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:35:36.977148 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.977134    2573 cpu_manager.go:226] "Starting" policy="none"
Oct 30 18:35:36.977148 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.977147    2573 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 18:35:36.977275 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.977166    2573 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 18:35:36.977275 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.977139    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:35:36.977853 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.977837    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-135.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:35:36.977910 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.977864    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-135.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:35:36.977910 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.977873    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-135.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:35:36.979533 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.979521    2573 policy_none.go:50] "Start"
Oct 30 18:35:36.979587 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.979538    2573 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 18:35:36.979587 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.979548    2573 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 18:35:36.981748 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.981733    2573 policy_none.go:44] "Start"
Oct 30 18:35:36.987165 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.987151    2573 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 18:35:36.987229 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.987209    2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dqnbx"
Oct 30 18:35:36.988508 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:36.988437    2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-135.us-east-2.compute.internal.18e3627653e0e8cb  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-135.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-135.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-135.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 18:35:36.976500939 +0000 UTC m=+0.441439222,LastTimestamp:2026-10-30 18:35:36.976500939 +0000 UTC m=+0.441439222,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-135.us-east-2.compute.internal,}"
Oct 30 18:35:36.996087 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:36.996067    2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dqnbx"
Oct 30 18:35:36.997888 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:36.997801    2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-135.us-east-2.compute.internal.18e3627653e137c5  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-135.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-137-135.us-east-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-137-135.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 18:35:36.976521157 +0000 UTC m=+0.441459435,LastTimestamp:2026-10-30 18:35:36.976521157 +0000 UTC m=+0.441459435,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-135.us-east-2.compute.internal,}"
Oct 30 18:35:37.015698 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:37.015254    2573 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 18:35:37.038635 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:37.021841    2573 manager.go:341] "Starting Device Plugin manager"
Oct 30 18:35:37.038635 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:37.022052    2573 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 18:35:37.038635 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:37.022066    2573 server.go:87] "Starting device plugin registration server"
Oct 30 18:35:37.038635 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:37.022335    2573 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 18:35:37.038635 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:37.022346    2573 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 18:35:37.038635 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:37.022432    2573 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 18:35:37.038635 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:37.022513    2573 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 18:35:37.038635 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:37.022521    2573 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 18:35:37.038635 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:37.023116    2573 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 18:35:37.038635 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:37.023159    2573 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-135.us-east-2.compute.internal\" not found"
Oct 30 18:35:37.038635 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:37.036059    2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 18:35:37.038635 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:37.037349    2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 18:35:37.038635 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:37.037369    2573 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 18:35:37.038635 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:37.037386    2573 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 18:35:37.038635 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:37.037415    2573 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 30 18:35:37.041289 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:37.041276    2573 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:35:37.122801 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:37.122699    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:35:37.123722 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:37.123703    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-135.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:35:37.123821 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:37.123739    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-135.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:35:37.123821 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:37.123753    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-135.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:35:37.123821 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:37.123779    2573 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-137-135.us-east-2.compute.internal"
Oct 30 18:35:37.132852 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:37.132837    2573 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-137-135.us-east-2.compute.internal"
Oct 30 18:35:37.132905 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:37.132861    2573 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-135.us-east-2.compute.internal\": node \"ip-10-0-137-135.us-east-2.compute.internal\" not found"
Oct 30 18:35:37.137824 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:37.137807    2573 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-135.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-135.us-east-2.compute.internal"]
Oct 30 18:35:37.137903 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:37.137869    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:35:37.138700 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:37.138658    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-135.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:35:37.138799 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:37.138711    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-135.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:35:37.138799 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:37.138726    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-135.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:35:37.141012 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:37.141000    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:35:37.141160 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:37.141134    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-135.us-east-2.compute.internal"
Oct 30 18:35:37.141194 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:37.141179    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:35:37.141724 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:37.141702    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-135.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:35:37.141724 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:37.141726    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-135.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:35:37.141855 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:37.141737    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-135.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:35:37.141855 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:37.141709    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-135.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:35:37.141855 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:37.141804    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-135.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:35:37.141855 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:37.141814    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-135.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:35:37.143879 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:37.143862    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-135.us-east-2.compute.internal"
Oct 30 18:35:37.143950 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:37.143890    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:35:37.144600 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:37.144585    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-135.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:35:37.144708 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:37.144611    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-135.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:35:37.144708 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:37.144621    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-135.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:35:37.145984 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:37.145968    2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-135.us-east-2.compute.internal\" not found"
Oct 30 18:35:37.159293 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:37.159270    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/09a9d0b6ec5615a60675fe6f5b7fa6e4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-135.us-east-2.compute.internal\" (UID: \"09a9d0b6ec5615a60675fe6f5b7fa6e4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-135.us-east-2.compute.internal"
Oct 30 18:35:37.159385 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:37.159302    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/09a9d0b6ec5615a60675fe6f5b7fa6e4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-135.us-east-2.compute.internal\" (UID: \"09a9d0b6ec5615a60675fe6f5b7fa6e4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-135.us-east-2.compute.internal"
Oct 30 18:35:37.159385 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:37.159332    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/826c9de932b6871da80993c953a89ebf-config\") pod \"kube-apiserver-proxy-ip-10-0-137-135.us-east-2.compute.internal\" (UID: \"826c9de932b6871da80993c953a89ebf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-135.us-east-2.compute.internal"
Oct 30 18:35:37.162171 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:37.162154    2573 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-135.us-east-2.compute.internal\" not found" node="ip-10-0-137-135.us-east-2.compute.internal"
Oct 30 18:35:37.164134 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:37.164117    2573 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-135.us-east-2.compute.internal\" not found" node="ip-10-0-137-135.us-east-2.compute.internal"
Oct 30 18:35:37.246242 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:37.246218    2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-135.us-east-2.compute.internal\" not found"
Oct 30 18:35:37.259497 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:37.259474    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/09a9d0b6ec5615a60675fe6f5b7fa6e4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-135.us-east-2.compute.internal\" (UID: \"09a9d0b6ec5615a60675fe6f5b7fa6e4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-135.us-east-2.compute.internal"
Oct 30 18:35:37.259560 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:37.259501    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/09a9d0b6ec5615a60675fe6f5b7fa6e4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-135.us-east-2.compute.internal\" (UID: \"09a9d0b6ec5615a60675fe6f5b7fa6e4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-135.us-east-2.compute.internal"
Oct 30 18:35:37.259560 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:37.259518    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/826c9de932b6871da80993c953a89ebf-config\") pod \"kube-apiserver-proxy-ip-10-0-137-135.us-east-2.compute.internal\" (UID: \"826c9de932b6871da80993c953a89ebf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-135.us-east-2.compute.internal"
Oct 30 18:35:37.259638 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:37.259579    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/09a9d0b6ec5615a60675fe6f5b7fa6e4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-135.us-east-2.compute.internal\" (UID: \"09a9d0b6ec5615a60675fe6f5b7fa6e4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-135.us-east-2.compute.internal"
Oct 30 18:35:37.259638 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:37.259579    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/826c9de932b6871da80993c953a89ebf-config\") pod \"kube-apiserver-proxy-ip-10-0-137-135.us-east-2.compute.internal\" (UID: \"826c9de932b6871da80993c953a89ebf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-135.us-east-2.compute.internal"
Oct 30 18:35:37.259638 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:37.259606    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/09a9d0b6ec5615a60675fe6f5b7fa6e4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-135.us-east-2.compute.internal\" (UID: \"09a9d0b6ec5615a60675fe6f5b7fa6e4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-135.us-east-2.compute.internal"
Oct 30 18:35:37.346719 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:37.346691    2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-135.us-east-2.compute.internal\" not found"
Oct 30 18:35:37.447509 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:37.447431    2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-135.us-east-2.compute.internal\" not found"
Oct 30 18:35:37.466895 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:37.466870    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-135.us-east-2.compute.internal"
Oct 30 18:35:37.469118 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:37.469103    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-135.us-east-2.compute.internal"
Oct 30 18:35:37.548173 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:37.548143    2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-135.us-east-2.compute.internal\" not found"
Oct 30 18:35:37.648750 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:37.648711    2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-135.us-east-2.compute.internal\" not found"
Oct 30 18:35:37.749348 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:37.749276    2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-135.us-east-2.compute.internal\" not found"
Oct 30 18:35:37.837435 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:37.837407    2573 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:35:37.849508 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:37.849484    2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-135.us-east-2.compute.internal\" not found"
Oct 30 18:35:37.882984 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:37.882954    2573 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 18:35:37.883581 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:37.883082    2573 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 18:35:37.883581 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:37.883113    2573 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 18:35:37.950174 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:37.950147    2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-135.us-east-2.compute.internal\" not found"
Oct 30 18:35:38.002486 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:38.002343    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 18:30:37 +0000 UTC" deadline="2028-04-26 15:02:01.470035486 +0000 UTC"
Oct 30 18:35:38.002486 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:38.002428    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13052h26m23.467612152s"
Oct 30 18:35:38.028665 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:38.028632    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:35:38.030017 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:38.029991    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-135.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:35:38.030154 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:38.030036    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-135.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:35:38.030154 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:38.030051    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-135.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:35:38.050604 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:38.050579    2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-135.us-east-2.compute.internal\" not found"
Oct 30 18:35:38.106582 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:38.106543    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod826c9de932b6871da80993c953a89ebf.slice/crio-4613d69f1e8a92f7b0ccf60ce4f2bf0a1ffbe0c8347c420867cde38f8e36fdf4 WatchSource:0}: Error finding container 4613d69f1e8a92f7b0ccf60ce4f2bf0a1ffbe0c8347c420867cde38f8e36fdf4: Status 404 returned error can't find the container with id 4613d69f1e8a92f7b0ccf60ce4f2bf0a1ffbe0c8347c420867cde38f8e36fdf4
Oct 30 18:35:38.107084 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:38.107065    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09a9d0b6ec5615a60675fe6f5b7fa6e4.slice/crio-4a064f4b59347185376167aae61af87390c30621b87547a175af427d35b976fc WatchSource:0}: Error finding container 4a064f4b59347185376167aae61af87390c30621b87547a175af427d35b976fc: Status 404 returned error can't find the container with id 4a064f4b59347185376167aae61af87390c30621b87547a175af427d35b976fc
Oct 30 18:35:38.112153 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:38.112139    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 18:35:38.151532 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:38.151509    2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-135.us-east-2.compute.internal\" not found"
Oct 30 18:35:38.252028 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:38.251981    2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-135.us-east-2.compute.internal\" not found"
Oct 30 18:35:38.300896 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:38.300817    2573 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:35:38.302987 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:38.302971    2573 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:35:38.357213 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:38.357188    2573 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-135.us-east-2.compute.internal"
Oct 30 18:35:38.367638 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:38.367621    2573 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 18:35:38.368775 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:38.368762    2573 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-135.us-east-2.compute.internal"
Oct 30 18:35:38.379227 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:38.379208    2573 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 18:35:38.929556 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:38.929468    2573 apiserver.go:52] "Watching apiserver"
Oct 30 18:35:38.937813 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:38.937782    2573 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 18:35:38.938214 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:38.938179    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-7z9gn","openshift-cluster-node-tuning-operator/tuned-f77jk","openshift-image-registry/node-ca-mhtvp","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-135.us-east-2.compute.internal","openshift-multus/multus-4lmrw","openshift-multus/multus-additional-cni-plugins-tjtmv","openshift-multus/network-metrics-daemon-9p48q","kube-system/kube-apiserver-proxy-ip-10-0-137-135.us-east-2.compute.internal","openshift-network-diagnostics/network-check-target-mcc4d","openshift-network-operator/iptables-alerter-rxbc9","openshift-ovn-kubernetes/ovnkube-node-kwd4p","kube-system/global-pull-secret-syncer-vhtmb","kube-system/konnectivity-agent-dbbbq","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4pz"]
Oct 30 18:35:38.941387 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:38.941261    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rxbc9"
Oct 30 18:35:38.943891 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:38.943866    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dbbbq"
Oct 30 18:35:38.945794 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:38.945770    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-gjz7h\""
Oct 30 18:35:38.945910 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:38.945890    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:35:38.946061 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:38.946045    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 18:35:38.946137 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:38.946124    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 18:35:38.947621 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:38.947600    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mhtvp"
Oct 30 18:35:38.950090 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:38.950072    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4pz"
Oct 30 18:35:38.952617 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:38.952597    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4lmrw"
Oct 30 18:35:38.957310 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:38.957035    2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 18:35:38.957866 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:38.957848    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-tjtmv"
Oct 30 18:35:38.960660 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:38.960425    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mcc4d"
Oct 30 18:35:38.960660 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:38.960516    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mcc4d" podUID="4b27cd9e-69b2-4480-a84f-5b463cec1a0e"
Oct 30 18:35:38.962824 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:38.962790    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9p48q"
Oct 30 18:35:38.962908 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:38.962848    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9p48q" podUID="089fdd4f-daa0-4b3e-b81e-26a764a526bb"
Oct 30 18:35:38.965538 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:38.965516    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-f77jk"
Oct 30 18:35:38.968047 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:38.968023    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7z9gn"
Oct 30 18:35:38.970146 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:38.970125    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/093a372e-74ea-4820-82c8-0dff7c201f10-serviceca\") pod \"node-ca-mhtvp\" (UID: \"093a372e-74ea-4820-82c8-0dff7c201f10\") " pod="openshift-image-registry/node-ca-mhtvp"
Oct 30 18:35:38.970267 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:38.970153    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2ca560f4-9aa8-4bc6-9db1-6992a1fa7c1f-device-dir\") pod \"aws-ebs-csi-driver-node-zb4pz\" (UID: \"2ca560f4-9aa8-4bc6-9db1-6992a1fa7c1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4pz"
Oct 30 18:35:38.970267 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:38.970174    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2ca560f4-9aa8-4bc6-9db1-6992a1fa7c1f-sys-fs\") pod \"aws-ebs-csi-driver-node-zb4pz\" (UID: \"2ca560f4-9aa8-4bc6-9db1-6992a1fa7c1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4pz"
Oct 30 18:35:38.970267 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:38.970203    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d8f39e97-7f68-47da-a247-adf78b523afd-host-var-lib-cni-bin\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:35:38.970267 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:38.970224    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d8f39e97-7f68-47da-a247-adf78b523afd-multus-daemon-config\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:35:38.970267 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:38.970241    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lbr9\" (UniqueName: \"kubernetes.io/projected/0dc90efb-2709-4894-b457-6cadb8abd801-kube-api-access-7lbr9\") pod \"multus-additional-cni-plugins-tjtmv\" (UID: \"0dc90efb-2709-4894-b457-6cadb8abd801\") " pod="openshift-multus/multus-additional-cni-plugins-tjtmv"
Oct 30 18:35:38.970267 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:38.970260    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/093a372e-74ea-4820-82c8-0dff7c201f10-host\") pod \"node-ca-mhtvp\" (UID: \"093a372e-74ea-4820-82c8-0dff7c201f10\") " pod="openshift-image-registry/node-ca-mhtvp"
Oct 30 18:35:38.970625 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:38.970275    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d8f39e97-7f68-47da-a247-adf78b523afd-cni-binary-copy\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:35:38.970625 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:38.970289    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d8f39e97-7f68-47da-a247-adf78b523afd-multus-conf-dir\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:35:38.970625 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:38.970305    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/77c2eb22-4b45-486f-8c62-bd14086942f5-iptables-alerter-script\") pod \"iptables-alerter-rxbc9\" (UID: \"77c2eb22-4b45-486f-8c62-bd14086942f5\") " pod="openshift-network-operator/iptables-alerter-rxbc9"
Oct 30 18:35:38.970625 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:38.970341    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2ca560f4-9aa8-4bc6-9db1-6992a1fa7c1f-socket-dir\") pod \"aws-ebs-csi-driver-node-zb4pz\" (UID: \"2ca560f4-9aa8-4bc6-9db1-6992a1fa7c1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4pz"
Oct 30 18:35:38.970625 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:38.970359    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d8f39e97-7f68-47da-a247-adf78b523afd-system-cni-dir\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:35:38.970625 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:38.970374    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d8f39e97-7f68-47da-a247-adf78b523afd-multus-cni-dir\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:35:38.970625 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:38.970454    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d8f39e97-7f68-47da-a247-adf78b523afd-host-run-netns\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:35:38.970625 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:38.970483    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0dc90efb-2709-4894-b457-6cadb8abd801-cni-binary-copy\") pod \"multus-additional-cni-plugins-tjtmv\" (UID: \"0dc90efb-2709-4894-b457-6cadb8abd801\") " pod="openshift-multus/multus-additional-cni-plugins-tjtmv"
Oct 30 18:35:38.970625 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:38.970510    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0dc90efb-2709-4894-b457-6cadb8abd801-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tjtmv\" (UID: \"0dc90efb-2709-4894-b457-6cadb8abd801\") " pod="openshift-multus/multus-additional-cni-plugins-tjtmv"
Oct 30 18:35:38.970625 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:38.970530    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0dc90efb-2709-4894-b457-6cadb8abd801-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tjtmv\" (UID: \"0dc90efb-2709-4894-b457-6cadb8abd801\") " pod="openshift-multus/multus-additional-cni-plugins-tjtmv"
Oct 30 18:35:38.970625 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:38.970556    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d8f39e97-7f68-47da-a247-adf78b523afd-cnibin\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:35:38.970625 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:38.970574    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plzcb\" (UniqueName: \"kubernetes.io/projected/093a372e-74ea-4820-82c8-0dff7c201f10-kube-api-access-plzcb\") pod \"node-ca-mhtvp\" (UID: \"093a372e-74ea-4820-82c8-0dff7c201f10\") " pod="openshift-image-registry/node-ca-mhtvp"
Oct 30 18:35:38.970625 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:38.970589    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d8f39e97-7f68-47da-a247-adf78b523afd-host-var-lib-cni-multus\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:35:38.970625 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:38.970620    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d8f39e97-7f68-47da-a247-adf78b523afd-host-run-multus-certs\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:35:38.971390 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:38.970648    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:35:38.971390 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:38.970636    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0dc90efb-2709-4894-b457-6cadb8abd801-system-cni-dir\") pod \"multus-additional-cni-plugins-tjtmv\" (UID: \"0dc90efb-2709-4894-b457-6cadb8abd801\") " pod="openshift-multus/multus-additional-cni-plugins-tjtmv"
Oct 30 18:35:38.971390 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:38.970910    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2phv\" (UniqueName: \"kubernetes.io/projected/77c2eb22-4b45-486f-8c62-bd14086942f5-kube-api-access-d2phv\") pod \"iptables-alerter-rxbc9\" (UID: \"77c2eb22-4b45-486f-8c62-bd14086942f5\") " pod="openshift-network-operator/iptables-alerter-rxbc9"
Oct 30 18:35:38.971390 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:38.970928    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2ca560f4-9aa8-4bc6-9db1-6992a1fa7c1f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zb4pz\" (UID: \"2ca560f4-9aa8-4bc6-9db1-6992a1fa7c1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4pz"
Oct 30 18:35:38.971390 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:38.970944    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2ca560f4-9aa8-4bc6-9db1-6992a1fa7c1f-registration-dir\") pod \"aws-ebs-csi-driver-node-zb4pz\" (UID: \"2ca560f4-9aa8-4bc6-9db1-6992a1fa7c1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4pz"
Oct 30 18:35:38.971390 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:38.970972    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kczq\" (UniqueName: \"kubernetes.io/projected/2ca560f4-9aa8-4bc6-9db1-6992a1fa7c1f-kube-api-access-7kczq\") pod \"aws-ebs-csi-driver-node-zb4pz\" (UID: \"2ca560f4-9aa8-4bc6-9db1-6992a1fa7c1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4pz"
Oct 30 18:35:38.971390 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:38.970987    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d8f39e97-7f68-47da-a247-adf78b523afd-multus-socket-dir-parent\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:35:38.971390 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:38.971002    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d8f39e97-7f68-47da-a247-adf78b523afd-hostroot\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:35:38.971390 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:38.971016    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/bd08a0fa-46bc-4250-985c-d2580a2225b8-agent-certs\") pod \"konnectivity-agent-dbbbq\" (UID: \"bd08a0fa-46bc-4250-985c-d2580a2225b8\") " pod="kube-system/konnectivity-agent-dbbbq"
Oct 30 18:35:38.971390 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:38.971031    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2ca560f4-9aa8-4bc6-9db1-6992a1fa7c1f-etc-selinux\") pod \"aws-ebs-csi-driver-node-zb4pz\" (UID: \"2ca560f4-9aa8-4bc6-9db1-6992a1fa7c1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4pz"
Oct 30 18:35:38.971390 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:38.971045    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d8f39e97-7f68-47da-a247-adf78b523afd-host-run-k8s-cni-cncf-io\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:35:38.971390 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:38.971060    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d8f39e97-7f68-47da-a247-adf78b523afd-etc-kubernetes\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:35:38.971390 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:38.971074    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0dc90efb-2709-4894-b457-6cadb8abd801-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tjtmv\" (UID: \"0dc90efb-2709-4894-b457-6cadb8abd801\") " pod="openshift-multus/multus-additional-cni-plugins-tjtmv"
Oct 30 18:35:38.971390 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:38.971092    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/77c2eb22-4b45-486f-8c62-bd14086942f5-host-slash\") pod \"iptables-alerter-rxbc9\" (UID: \"77c2eb22-4b45-486f-8c62-bd14086942f5\") " pod="openshift-network-operator/iptables-alerter-rxbc9"
Oct 30 18:35:38.971390 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:38.971107    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/bd08a0fa-46bc-4250-985c-d2580a2225b8-konnectivity-ca\") pod \"konnectivity-agent-dbbbq\" (UID: \"bd08a0fa-46bc-4250-985c-d2580a2225b8\") " pod="kube-system/konnectivity-agent-dbbbq"
Oct 30 18:35:38.971390 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:38.971122    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d8f39e97-7f68-47da-a247-adf78b523afd-host-var-lib-kubelet\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:35:38.971390 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:38.971139    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq9k2\" (UniqueName: \"kubernetes.io/projected/d8f39e97-7f68-47da-a247-adf78b523afd-kube-api-access-fq9k2\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:35:38.972090 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:38.971157    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0dc90efb-2709-4894-b457-6cadb8abd801-cnibin\") pod \"multus-additional-cni-plugins-tjtmv\" (UID: \"0dc90efb-2709-4894-b457-6cadb8abd801\") " pod="openshift-multus/multus-additional-cni-plugins-tjtmv"
Oct 30 18:35:38.973435 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:38.973197    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vhtmb"
Oct 30 18:35:38.973435 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:38.973263    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vhtmb" podUID="7c3ae3e0-ddf1-414f-bb79-e4afc0527b26"
Oct 30 18:35:38.987631 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:38.987326    2573 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 18:35:38.987631 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:38.987415    2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-jrb2f"
Oct 30 18:35:38.995268 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:38.995229    2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-jrb2f"
Oct 30 18:35:39.043404 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.043346    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-135.us-east-2.compute.internal" event={"ID":"826c9de932b6871da80993c953a89ebf","Type":"ContainerStarted","Data":"4613d69f1e8a92f7b0ccf60ce4f2bf0a1ffbe0c8347c420867cde38f8e36fdf4"}
Oct 30 18:35:39.044994 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.044953    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-135.us-east-2.compute.internal" event={"ID":"09a9d0b6ec5615a60675fe6f5b7fa6e4","Type":"ContainerStarted","Data":"4a064f4b59347185376167aae61af87390c30621b87547a175af427d35b976fc"}
Oct 30 18:35:39.054872 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.054814    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 18:35:39.055025 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.054814    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:35:39.055107 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.055042    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 18:35:39.055107 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.055055    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 18:35:39.055107 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.055101    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 18:35:39.055278 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.055142    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 18:35:39.055278 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.055170    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 18:35:39.055278 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.055181    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 18:35:39.055278 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.055238    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 18:35:39.055408 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.055310    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 18:35:39.055408 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.055318    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 18:35:39.055408 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.055327    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 18:35:39.056452 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.056426    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 18:35:39.056576 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.056469    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 18:35:39.056576 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.056518    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 18:35:39.056576 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.056563    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 18:35:39.056797 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.056613    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 18:35:39.056797 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.056654    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 18:35:39.056797 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.056663    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 18:35:39.056797 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.056519    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 18:35:39.056797 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.056564    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 18:35:39.056797 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.056769    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-xr42l\""
Oct 30 18:35:39.057028 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.056854    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 18:35:39.057702 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.057658    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 18:35:39.057986 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.057970    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-89nks\""
Oct 30 18:35:39.058089 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.058048    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-8njxf\""
Oct 30 18:35:39.058155 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.058139    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lfvmh\""
Oct 30 18:35:39.058215 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.058198    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-sfbzk\""
Oct 30 18:35:39.059873 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.059848    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-ldjxj\""
Oct 30 18:35:39.059994 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.059949    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-8j87p\""
Oct 30 18:35:39.060858 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.060837    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-s4hwp\""
Oct 30 18:35:39.062443 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.062410    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 18:35:39.063915 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.063900    2573 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 18:35:39.071875 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.071848    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d8f39e97-7f68-47da-a247-adf78b523afd-hostroot\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:35:39.071985 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.071890    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/bd08a0fa-46bc-4250-985c-d2580a2225b8-agent-certs\") pod \"konnectivity-agent-dbbbq\" (UID: \"bd08a0fa-46bc-4250-985c-d2580a2225b8\") " pod="kube-system/konnectivity-agent-dbbbq"
Oct 30 18:35:39.071985 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.071921    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/545dd3b9-01a9-40be-8762-5476a7a9ede4-etc-modprobe-d\") pod \"tuned-f77jk\" (UID: \"545dd3b9-01a9-40be-8762-5476a7a9ede4\") " pod="openshift-cluster-node-tuning-operator/tuned-f77jk"
Oct 30 18:35:39.071985 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.071950    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/545dd3b9-01a9-40be-8762-5476a7a9ede4-etc-kubernetes\") pod \"tuned-f77jk\" (UID: \"545dd3b9-01a9-40be-8762-5476a7a9ede4\") " pod="openshift-cluster-node-tuning-operator/tuned-f77jk"
Oct 30 18:35:39.071985 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.071975    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/545dd3b9-01a9-40be-8762-5476a7a9ede4-etc-sysctl-d\") pod \"tuned-f77jk\" (UID: \"545dd3b9-01a9-40be-8762-5476a7a9ede4\") " pod="openshift-cluster-node-tuning-operator/tuned-f77jk"
Oct 30 18:35:39.072167 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.071982    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d8f39e97-7f68-47da-a247-adf78b523afd-hostroot\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:35:39.072167 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.072000    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/545dd3b9-01a9-40be-8762-5476a7a9ede4-var-lib-kubelet\") pod \"tuned-f77jk\" (UID: \"545dd3b9-01a9-40be-8762-5476a7a9ede4\") " pod="openshift-cluster-node-tuning-operator/tuned-f77jk"
Oct 30 18:35:39.072167 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.072064    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2ca560f4-9aa8-4bc6-9db1-6992a1fa7c1f-etc-selinux\") pod \"aws-ebs-csi-driver-node-zb4pz\" (UID: \"2ca560f4-9aa8-4bc6-9db1-6992a1fa7c1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4pz"
Oct 30 18:35:39.072167 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.072088    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d8f39e97-7f68-47da-a247-adf78b523afd-etc-kubernetes\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:35:39.072167 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.072112    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8bcb59a4-381f-49ba-b732-f5e4658707de-ovnkube-config\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:35:39.072167 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.072139    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8bcb59a4-381f-49ba-b732-f5e4658707de-ovnkube-script-lib\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:35:39.072167 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.072147    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d8f39e97-7f68-47da-a247-adf78b523afd-etc-kubernetes\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:35:39.072167 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.072166    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7c3ae3e0-ddf1-414f-bb79-e4afc0527b26-kubelet-config\") pod \"global-pull-secret-syncer-vhtmb\" (UID: \"7c3ae3e0-ddf1-414f-bb79-e4afc0527b26\") " pod="kube-system/global-pull-secret-syncer-vhtmb"
Oct 30 18:35:39.072586 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.072202    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq9k2\" (UniqueName: \"kubernetes.io/projected/d8f39e97-7f68-47da-a247-adf78b523afd-kube-api-access-fq9k2\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:35:39.072586 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.072237    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/089fdd4f-daa0-4b3e-b81e-26a764a526bb-metrics-certs\") pod \"network-metrics-daemon-9p48q\" (UID: \"089fdd4f-daa0-4b3e-b81e-26a764a526bb\") " pod="openshift-multus/network-metrics-daemon-9p48q"
Oct 30 18:35:39.072586 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.072243    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2ca560f4-9aa8-4bc6-9db1-6992a1fa7c1f-etc-selinux\") pod \"aws-ebs-csi-driver-node-zb4pz\" (UID: \"2ca560f4-9aa8-4bc6-9db1-6992a1fa7c1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4pz"
Oct 30 18:35:39.072586 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.072306    2573 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 18:35:39.072586 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.072319    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8bcb59a4-381f-49ba-b732-f5e4658707de-systemd-units\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:35:39.072586 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.072346    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8bcb59a4-381f-49ba-b732-f5e4658707de-host-slash\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:35:39.072586 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.072372    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/093a372e-74ea-4820-82c8-0dff7c201f10-serviceca\") pod \"node-ca-mhtvp\" (UID: \"093a372e-74ea-4820-82c8-0dff7c201f10\") " pod="openshift-image-registry/node-ca-mhtvp"
Oct 30 18:35:39.072586 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.072397    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2ca560f4-9aa8-4bc6-9db1-6992a1fa7c1f-device-dir\") pod \"aws-ebs-csi-driver-node-zb4pz\" (UID: \"2ca560f4-9aa8-4bc6-9db1-6992a1fa7c1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4pz"
Oct 30 18:35:39.072586 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.072421    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2ca560f4-9aa8-4bc6-9db1-6992a1fa7c1f-sys-fs\") pod \"aws-ebs-csi-driver-node-zb4pz\" (UID: \"2ca560f4-9aa8-4bc6-9db1-6992a1fa7c1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4pz"
Oct 30 18:35:39.072586 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.072446    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d8f39e97-7f68-47da-a247-adf78b523afd-multus-daemon-config\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:35:39.072586 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.072503    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d8f39e97-7f68-47da-a247-adf78b523afd-cni-binary-copy\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:35:39.072586 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.072556    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2ca560f4-9aa8-4bc6-9db1-6992a1fa7c1f-device-dir\") pod \"aws-ebs-csi-driver-node-zb4pz\" (UID: \"2ca560f4-9aa8-4bc6-9db1-6992a1fa7c1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4pz"
Oct 30 18:35:39.072586 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.072575    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2ca560f4-9aa8-4bc6-9db1-6992a1fa7c1f-sys-fs\") pod \"aws-ebs-csi-driver-node-zb4pz\" (UID: \"2ca560f4-9aa8-4bc6-9db1-6992a1fa7c1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4pz"
Oct 30 18:35:39.073298 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.072621    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d8f39e97-7f68-47da-a247-adf78b523afd-multus-conf-dir\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:35:39.073298 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.072654    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/545dd3b9-01a9-40be-8762-5476a7a9ede4-run\") pod \"tuned-f77jk\" (UID: \"545dd3b9-01a9-40be-8762-5476a7a9ede4\") " pod="openshift-cluster-node-tuning-operator/tuned-f77jk"
Oct 30 18:35:39.073298 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.072681    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq2h8\" (UniqueName: \"kubernetes.io/projected/545dd3b9-01a9-40be-8762-5476a7a9ede4-kube-api-access-cq2h8\") pod \"tuned-f77jk\" (UID: \"545dd3b9-01a9-40be-8762-5476a7a9ede4\") " pod="openshift-cluster-node-tuning-operator/tuned-f77jk"
Oct 30 18:35:39.073298 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.072728    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8bcb59a4-381f-49ba-b732-f5e4658707de-host-run-netns\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:35:39.073298 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.072754    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8bcb59a4-381f-49ba-b732-f5e4658707de-run-ovn\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:35:39.073298 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.072782    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2ca560f4-9aa8-4bc6-9db1-6992a1fa7c1f-socket-dir\") pod \"aws-ebs-csi-driver-node-zb4pz\" (UID: \"2ca560f4-9aa8-4bc6-9db1-6992a1fa7c1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4pz"
Oct 30 18:35:39.073298 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.072820    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d8f39e97-7f68-47da-a247-adf78b523afd-system-cni-dir\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:35:39.073298 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.072846    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d8f39e97-7f68-47da-a247-adf78b523afd-multus-cni-dir\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:35:39.073298 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.072914    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0dc90efb-2709-4894-b457-6cadb8abd801-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tjtmv\" (UID: \"0dc90efb-2709-4894-b457-6cadb8abd801\") " pod="openshift-multus/multus-additional-cni-plugins-tjtmv"
Oct 30 18:35:39.073298 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.072944    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/545dd3b9-01a9-40be-8762-5476a7a9ede4-host\") pod \"tuned-f77jk\" (UID: \"545dd3b9-01a9-40be-8762-5476a7a9ede4\") " pod="openshift-cluster-node-tuning-operator/tuned-f77jk"
Oct 30 18:35:39.073298 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.072945    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/093a372e-74ea-4820-82c8-0dff7c201f10-serviceca\") pod \"node-ca-mhtvp\" (UID: \"093a372e-74ea-4820-82c8-0dff7c201f10\") " pod="openshift-image-registry/node-ca-mhtvp"
Oct 30 18:35:39.073298 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.072970    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/545dd3b9-01a9-40be-8762-5476a7a9ede4-tmp\") pod \"tuned-f77jk\" (UID: \"545dd3b9-01a9-40be-8762-5476a7a9ede4\") " pod="openshift-cluster-node-tuning-operator/tuned-f77jk"
Oct 30 18:35:39.073298 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.072996    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8bcb59a4-381f-49ba-b732-f5e4658707de-host-kubelet\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:35:39.073298 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.073024    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8bcb59a4-381f-49ba-b732-f5e4658707de-ovn-node-metrics-cert\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:35:39.073298 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.073056    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfrpn\" (UniqueName: \"kubernetes.io/projected/4b27cd9e-69b2-4480-a84f-5b463cec1a0e-kube-api-access-rfrpn\") pod \"network-check-target-mcc4d\" (UID: \"4b27cd9e-69b2-4480-a84f-5b463cec1a0e\") " pod="openshift-network-diagnostics/network-check-target-mcc4d"
Oct 30 18:35:39.073298 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.073095    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8bcb59a4-381f-49ba-b732-f5e4658707de-run-openvswitch\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:35:39.073298 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.073113    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d8f39e97-7f68-47da-a247-adf78b523afd-multus-daemon-config\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:35:39.074222 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.073120    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d8f39e97-7f68-47da-a247-adf78b523afd-cni-binary-copy\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:35:39.074222 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.073121    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8bcb59a4-381f-49ba-b732-f5e4658707de-host-run-ovn-kubernetes\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:35:39.074222 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.073169    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d8f39e97-7f68-47da-a247-adf78b523afd-multus-conf-dir\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:35:39.074222 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.073178    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7c3ae3e0-ddf1-414f-bb79-e4afc0527b26-dbus\") pod \"global-pull-secret-syncer-vhtmb\" (UID: \"7c3ae3e0-ddf1-414f-bb79-e4afc0527b26\") " pod="kube-system/global-pull-secret-syncer-vhtmb"
Oct 30 18:35:39.074222 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.073301    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plzcb\" (UniqueName: \"kubernetes.io/projected/093a372e-74ea-4820-82c8-0dff7c201f10-kube-api-access-plzcb\") pod \"node-ca-mhtvp\" (UID: \"093a372e-74ea-4820-82c8-0dff7c201f10\") " pod="openshift-image-registry/node-ca-mhtvp"
Oct 30 18:35:39.074222 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.073355    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2ca560f4-9aa8-4bc6-9db1-6992a1fa7c1f-socket-dir\") pod \"aws-ebs-csi-driver-node-zb4pz\" (UID: \"2ca560f4-9aa8-4bc6-9db1-6992a1fa7c1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4pz"
Oct 30 18:35:39.074222 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.073366    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d8f39e97-7f68-47da-a247-adf78b523afd-system-cni-dir\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:35:39.074222 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.073399    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d8f39e97-7f68-47da-a247-adf78b523afd-multus-cni-dir\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:35:39.074222 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.073412    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0dc90efb-2709-4894-b457-6cadb8abd801-system-cni-dir\") pod \"multus-additional-cni-plugins-tjtmv\" (UID: \"0dc90efb-2709-4894-b457-6cadb8abd801\") " pod="openshift-multus/multus-additional-cni-plugins-tjtmv"
Oct 30 18:35:39.074222 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.073468    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0dc90efb-2709-4894-b457-6cadb8abd801-system-cni-dir\") pod \"multus-additional-cni-plugins-tjtmv\" (UID: \"0dc90efb-2709-4894-b457-6cadb8abd801\") " pod="openshift-multus/multus-additional-cni-plugins-tjtmv"
Oct 30 18:35:39.074222 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.073513    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/545dd3b9-01a9-40be-8762-5476a7a9ede4-etc-sysconfig\") pod \"tuned-f77jk\" (UID: \"545dd3b9-01a9-40be-8762-5476a7a9ede4\") " pod="openshift-cluster-node-tuning-operator/tuned-f77jk"
Oct 30 18:35:39.074222 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.073551    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2ca560f4-9aa8-4bc6-9db1-6992a1fa7c1f-registration-dir\") pod \"aws-ebs-csi-driver-node-zb4pz\" (UID: \"2ca560f4-9aa8-4bc6-9db1-6992a1fa7c1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4pz"
Oct 30 18:35:39.074222 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.073601    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d8f39e97-7f68-47da-a247-adf78b523afd-multus-socket-dir-parent\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:35:39.074222 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.073612    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2ca560f4-9aa8-4bc6-9db1-6992a1fa7c1f-registration-dir\") pod \"aws-ebs-csi-driver-node-zb4pz\" (UID: \"2ca560f4-9aa8-4bc6-9db1-6992a1fa7c1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4pz"
Oct 30 18:35:39.074222 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.073649    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d8f39e97-7f68-47da-a247-adf78b523afd-host-run-k8s-cni-cncf-io\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:35:39.074222 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.073674    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d8f39e97-7f68-47da-a247-adf78b523afd-multus-socket-dir-parent\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:35:39.074222 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.073743    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0dc90efb-2709-4894-b457-6cadb8abd801-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tjtmv\" (UID: \"0dc90efb-2709-4894-b457-6cadb8abd801\") " pod="openshift-multus/multus-additional-cni-plugins-tjtmv"
Oct 30 18:35:39.075095 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.073755    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d8f39e97-7f68-47da-a247-adf78b523afd-host-run-k8s-cni-cncf-io\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:35:39.075095 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.073806    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/77c2eb22-4b45-486f-8c62-bd14086942f5-host-slash\") pod \"iptables-alerter-rxbc9\" (UID: \"77c2eb22-4b45-486f-8c62-bd14086942f5\") " pod="openshift-network-operator/iptables-alerter-rxbc9"
Oct 30 18:35:39.075095 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.073846    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/bd08a0fa-46bc-4250-985c-d2580a2225b8-konnectivity-ca\") pod \"konnectivity-agent-dbbbq\" (UID: \"bd08a0fa-46bc-4250-985c-d2580a2225b8\") " pod="kube-system/konnectivity-agent-dbbbq"
Oct 30 18:35:39.075095 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.073873    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/545dd3b9-01a9-40be-8762-5476a7a9ede4-sys\") pod \"tuned-f77jk\" (UID: \"545dd3b9-01a9-40be-8762-5476a7a9ede4\") " pod="openshift-cluster-node-tuning-operator/tuned-f77jk"
Oct 30 18:35:39.075095 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.073920    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0dc90efb-2709-4894-b457-6cadb8abd801-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tjtmv\" (UID: \"0dc90efb-2709-4894-b457-6cadb8abd801\") " pod="openshift-multus/multus-additional-cni-plugins-tjtmv"
Oct 30 18:35:39.075095 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.073925    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8bcb59a4-381f-49ba-b732-f5e4658707de-log-socket\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:35:39.075095 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.073965    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d8f39e97-7f68-47da-a247-adf78b523afd-host-var-lib-kubelet\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:35:39.075095 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.074013    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0dc90efb-2709-4894-b457-6cadb8abd801-cnibin\") pod \"multus-additional-cni-plugins-tjtmv\" (UID: \"0dc90efb-2709-4894-b457-6cadb8abd801\") " pod="openshift-multus/multus-additional-cni-plugins-tjtmv"
Oct 30 18:35:39.075095 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.074043    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx6qn\" (UniqueName: \"kubernetes.io/projected/089fdd4f-daa0-4b3e-b81e-26a764a526bb-kube-api-access-sx6qn\") pod \"network-metrics-daemon-9p48q\" (UID: \"089fdd4f-daa0-4b3e-b81e-26a764a526bb\") " pod="openshift-multus/network-metrics-daemon-9p48q"
Oct 30 18:35:39.075095 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.074095    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8bcb59a4-381f-49ba-b732-f5e4658707de-node-log\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:35:39.075095 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.074124    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d8f39e97-7f68-47da-a247-adf78b523afd-host-var-lib-cni-bin\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:35:39.075095 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.074201    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lbr9\" (UniqueName: \"kubernetes.io/projected/0dc90efb-2709-4894-b457-6cadb8abd801-kube-api-access-7lbr9\") pod \"multus-additional-cni-plugins-tjtmv\" (UID: \"0dc90efb-2709-4894-b457-6cadb8abd801\") " pod="openshift-multus/multus-additional-cni-plugins-tjtmv"
Oct 30 18:35:39.075095 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.074230    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/545dd3b9-01a9-40be-8762-5476a7a9ede4-etc-tuned\") pod \"tuned-f77jk\" (UID: \"545dd3b9-01a9-40be-8762-5476a7a9ede4\") " pod="openshift-cluster-node-tuning-operator/tuned-f77jk"
Oct 30 18:35:39.075095 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.074288    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8bcb59a4-381f-49ba-b732-f5e4658707de-env-overrides\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:35:39.075095 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.074316    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/093a372e-74ea-4820-82c8-0dff7c201f10-host\") pod \"node-ca-mhtvp\" (UID: \"093a372e-74ea-4820-82c8-0dff7c201f10\") " pod="openshift-image-registry/node-ca-mhtvp"
Oct 30 18:35:39.075095 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.074344    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/77c2eb22-4b45-486f-8c62-bd14086942f5-iptables-alerter-script\") pod \"iptables-alerter-rxbc9\" (UID: \"77c2eb22-4b45-486f-8c62-bd14086942f5\") " pod="openshift-network-operator/iptables-alerter-rxbc9"
Oct 30 18:35:39.075095 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.074370    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/42b71e2d-a8f2-43cf-8487-0e5da2338c54-hosts-file\") pod \"node-resolver-7z9gn\" (UID: \"42b71e2d-a8f2-43cf-8487-0e5da2338c54\") " pod="openshift-dns/node-resolver-7z9gn"
Oct 30 18:35:39.075888 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.074395    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8bcb59a4-381f-49ba-b732-f5e4658707de-host-cni-netd\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:35:39.075888 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.074424    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8bcb59a4-381f-49ba-b732-f5e4658707de-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:35:39.075888 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.074453    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d8f39e97-7f68-47da-a247-adf78b523afd-host-run-netns\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:35:39.075888 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.074514    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/bd08a0fa-46bc-4250-985c-d2580a2225b8-konnectivity-ca\") pod \"konnectivity-agent-dbbbq\" (UID: \"bd08a0fa-46bc-4250-985c-d2580a2225b8\") " pod="kube-system/konnectivity-agent-dbbbq"
Oct 30 18:35:39.075888 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.074545    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0dc90efb-2709-4894-b457-6cadb8abd801-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tjtmv\" (UID: \"0dc90efb-2709-4894-b457-6cadb8abd801\") " pod="openshift-multus/multus-additional-cni-plugins-tjtmv"
Oct 30 18:35:39.075888 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.074585    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/77c2eb22-4b45-486f-8c62-bd14086942f5-host-slash\") pod \"iptables-alerter-rxbc9\" (UID: \"77c2eb22-4b45-486f-8c62-bd14086942f5\") " pod="openshift-network-operator/iptables-alerter-rxbc9"
Oct 30 18:35:39.075888 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.074631    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d8f39e97-7f68-47da-a247-adf78b523afd-host-var-lib-kubelet\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:35:39.075888 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.074637    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d8f39e97-7f68-47da-a247-adf78b523afd-host-var-lib-cni-bin\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:35:39.075888 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.074633    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0dc90efb-2709-4894-b457-6cadb8abd801-cni-binary-copy\") pod \"multus-additional-cni-plugins-tjtmv\" (UID: \"0dc90efb-2709-4894-b457-6cadb8abd801\") " pod="openshift-multus/multus-additional-cni-plugins-tjtmv"
Oct 30 18:35:39.075888 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.074652    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0dc90efb-2709-4894-b457-6cadb8abd801-cnibin\") pod \"multus-additional-cni-plugins-tjtmv\" (UID: \"0dc90efb-2709-4894-b457-6cadb8abd801\") " pod="openshift-multus/multus-additional-cni-plugins-tjtmv"
Oct 30 18:35:39.075888 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.074690    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d8f39e97-7f68-47da-a247-adf78b523afd-host-run-netns\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:35:39.075888 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.074695    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0dc90efb-2709-4894-b457-6cadb8abd801-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tjtmv\" (UID: \"0dc90efb-2709-4894-b457-6cadb8abd801\") " pod="openshift-multus/multus-additional-cni-plugins-tjtmv"
Oct 30 18:35:39.075888 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.074740    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/545dd3b9-01a9-40be-8762-5476a7a9ede4-etc-sysctl-conf\") pod \"tuned-f77jk\" (UID: \"545dd3b9-01a9-40be-8762-5476a7a9ede4\") " pod="openshift-cluster-node-tuning-operator/tuned-f77jk"
Oct 30 18:35:39.075888 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.074817    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/545dd3b9-01a9-40be-8762-5476a7a9ede4-etc-systemd\") pod \"tuned-f77jk\" (UID: \"545dd3b9-01a9-40be-8762-5476a7a9ede4\") " pod="openshift-cluster-node-tuning-operator/tuned-f77jk"
Oct 30 18:35:39.075888 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.074881    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/545dd3b9-01a9-40be-8762-5476a7a9ede4-lib-modules\") pod \"tuned-f77jk\" (UID: \"545dd3b9-01a9-40be-8762-5476a7a9ede4\") " pod="openshift-cluster-node-tuning-operator/tuned-f77jk"
Oct 30 18:35:39.075888 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.074922    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8bcb59a4-381f-49ba-b732-f5e4658707de-etc-openvswitch\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:35:39.075888 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.074950    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/093a372e-74ea-4820-82c8-0dff7c201f10-host\") pod \"node-ca-mhtvp\" (UID: \"093a372e-74ea-4820-82c8-0dff7c201f10\") " pod="openshift-image-registry/node-ca-mhtvp"
Oct 30 18:35:39.076685 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.074995    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8bcb59a4-381f-49ba-b732-f5e4658707de-host-cni-bin\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:35:39.076685 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.075038    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d8f39e97-7f68-47da-a247-adf78b523afd-cnibin\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:35:39.076685 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.075053    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0dc90efb-2709-4894-b457-6cadb8abd801-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tjtmv\" (UID: \"0dc90efb-2709-4894-b457-6cadb8abd801\") " pod="openshift-multus/multus-additional-cni-plugins-tjtmv"
Oct 30 18:35:39.076685 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.075085    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42b71e2d-a8f2-43cf-8487-0e5da2338c54-tmp-dir\") pod \"node-resolver-7z9gn\" (UID: \"42b71e2d-a8f2-43cf-8487-0e5da2338c54\") " pod="openshift-dns/node-resolver-7z9gn"
Oct 30 18:35:39.076685 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.075131    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d8f39e97-7f68-47da-a247-adf78b523afd-cnibin\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:35:39.076685 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.075140    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0dc90efb-2709-4894-b457-6cadb8abd801-cni-binary-copy\") pod \"multus-additional-cni-plugins-tjtmv\" (UID: \"0dc90efb-2709-4894-b457-6cadb8abd801\") " pod="openshift-multus/multus-additional-cni-plugins-tjtmv"
Oct 30 18:35:39.076685 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.075175    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gm9cn\" (UniqueName: \"kubernetes.io/projected/42b71e2d-a8f2-43cf-8487-0e5da2338c54-kube-api-access-gm9cn\") pod \"node-resolver-7z9gn\" (UID: \"42b71e2d-a8f2-43cf-8487-0e5da2338c54\") " pod="openshift-dns/node-resolver-7z9gn"
Oct 30 18:35:39.076685 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.075225    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8bcb59a4-381f-49ba-b732-f5e4658707de-var-lib-openvswitch\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:35:39.076685 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.075233    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/77c2eb22-4b45-486f-8c62-bd14086942f5-iptables-alerter-script\") pod \"iptables-alerter-rxbc9\" (UID: \"77c2eb22-4b45-486f-8c62-bd14086942f5\") " pod="openshift-network-operator/iptables-alerter-rxbc9"
Oct 30 18:35:39.076685 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.075307    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dltbj\" (UniqueName: \"kubernetes.io/projected/8bcb59a4-381f-49ba-b732-f5e4658707de-kube-api-access-dltbj\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:35:39.076685 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.075359    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7c3ae3e0-ddf1-414f-bb79-e4afc0527b26-original-pull-secret\") pod \"global-pull-secret-syncer-vhtmb\" (UID: \"7c3ae3e0-ddf1-414f-bb79-e4afc0527b26\") " pod="kube-system/global-pull-secret-syncer-vhtmb"
Oct 30 18:35:39.076685 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.075409    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d8f39e97-7f68-47da-a247-adf78b523afd-host-var-lib-cni-multus\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:35:39.076685 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.075453    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d8f39e97-7f68-47da-a247-adf78b523afd-host-run-multus-certs\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:35:39.076685 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.075492    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d8f39e97-7f68-47da-a247-adf78b523afd-host-var-lib-cni-multus\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:35:39.076685 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.075518    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2phv\" (UniqueName: \"kubernetes.io/projected/77c2eb22-4b45-486f-8c62-bd14086942f5-kube-api-access-d2phv\") pod \"iptables-alerter-rxbc9\" (UID: \"77c2eb22-4b45-486f-8c62-bd14086942f5\") " pod="openshift-network-operator/iptables-alerter-rxbc9"
Oct 30 18:35:39.076685 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.075539    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d8f39e97-7f68-47da-a247-adf78b523afd-host-run-multus-certs\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:35:39.076685 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.075593    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8bcb59a4-381f-49ba-b732-f5e4658707de-run-systemd\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:35:39.077479 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.075784    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2ca560f4-9aa8-4bc6-9db1-6992a1fa7c1f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zb4pz\" (UID: \"2ca560f4-9aa8-4bc6-9db1-6992a1fa7c1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4pz"
Oct 30 18:35:39.077479 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.075833    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kczq\" (UniqueName: \"kubernetes.io/projected/2ca560f4-9aa8-4bc6-9db1-6992a1fa7c1f-kube-api-access-7kczq\") pod \"aws-ebs-csi-driver-node-zb4pz\" (UID: \"2ca560f4-9aa8-4bc6-9db1-6992a1fa7c1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4pz"
Oct 30 18:35:39.077479 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.075844    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2ca560f4-9aa8-4bc6-9db1-6992a1fa7c1f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zb4pz\" (UID: \"2ca560f4-9aa8-4bc6-9db1-6992a1fa7c1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4pz"
Oct 30 18:35:39.077479 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.077025    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/bd08a0fa-46bc-4250-985c-d2580a2225b8-agent-certs\") pod \"konnectivity-agent-dbbbq\" (UID: \"bd08a0fa-46bc-4250-985c-d2580a2225b8\") " pod="kube-system/konnectivity-agent-dbbbq"
Oct 30 18:35:39.082647 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.082536    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plzcb\" (UniqueName: \"kubernetes.io/projected/093a372e-74ea-4820-82c8-0dff7c201f10-kube-api-access-plzcb\") pod \"node-ca-mhtvp\" (UID: \"093a372e-74ea-4820-82c8-0dff7c201f10\") " pod="openshift-image-registry/node-ca-mhtvp"
Oct 30 18:35:39.082647 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.082562    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq9k2\" (UniqueName: \"kubernetes.io/projected/d8f39e97-7f68-47da-a247-adf78b523afd-kube-api-access-fq9k2\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:35:39.085194 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.085171    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lbr9\" (UniqueName: \"kubernetes.io/projected/0dc90efb-2709-4894-b457-6cadb8abd801-kube-api-access-7lbr9\") pod \"multus-additional-cni-plugins-tjtmv\" (UID: \"0dc90efb-2709-4894-b457-6cadb8abd801\") " pod="openshift-multus/multus-additional-cni-plugins-tjtmv"
Oct 30 18:35:39.085513 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.085491    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2phv\" (UniqueName: \"kubernetes.io/projected/77c2eb22-4b45-486f-8c62-bd14086942f5-kube-api-access-d2phv\") pod \"iptables-alerter-rxbc9\" (UID: \"77c2eb22-4b45-486f-8c62-bd14086942f5\") " pod="openshift-network-operator/iptables-alerter-rxbc9"
Oct 30 18:35:39.085596 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.085567    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kczq\" (UniqueName: \"kubernetes.io/projected/2ca560f4-9aa8-4bc6-9db1-6992a1fa7c1f-kube-api-access-7kczq\") pod \"aws-ebs-csi-driver-node-zb4pz\" (UID: \"2ca560f4-9aa8-4bc6-9db1-6992a1fa7c1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4pz"
Oct 30 18:35:39.174619 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.174583    2573 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:35:39.176972 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.176935    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8bcb59a4-381f-49ba-b732-f5e4658707de-ovn-node-metrics-cert\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:35:39.177113 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.176985    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfrpn\" (UniqueName: \"kubernetes.io/projected/4b27cd9e-69b2-4480-a84f-5b463cec1a0e-kube-api-access-rfrpn\") pod \"network-check-target-mcc4d\" (UID: \"4b27cd9e-69b2-4480-a84f-5b463cec1a0e\") " pod="openshift-network-diagnostics/network-check-target-mcc4d"
Oct 30 18:35:39.177113 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.177007    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8bcb59a4-381f-49ba-b732-f5e4658707de-run-openvswitch\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:35:39.177113 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.177030    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8bcb59a4-381f-49ba-b732-f5e4658707de-host-run-ovn-kubernetes\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:35:39.177113 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.177090    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8bcb59a4-381f-49ba-b732-f5e4658707de-run-openvswitch\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:35:39.177113 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.177090    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8bcb59a4-381f-49ba-b732-f5e4658707de-host-run-ovn-kubernetes\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:35:39.177390 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.177139    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7c3ae3e0-ddf1-414f-bb79-e4afc0527b26-dbus\") pod \"global-pull-secret-syncer-vhtmb\" (UID: \"7c3ae3e0-ddf1-414f-bb79-e4afc0527b26\") " pod="kube-system/global-pull-secret-syncer-vhtmb"
Oct 30 18:35:39.177390 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.177159    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/545dd3b9-01a9-40be-8762-5476a7a9ede4-etc-sysconfig\") pod \"tuned-f77jk\" (UID: \"545dd3b9-01a9-40be-8762-5476a7a9ede4\") " pod="openshift-cluster-node-tuning-operator/tuned-f77jk"
Oct 30 18:35:39.177390 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.177178    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/545dd3b9-01a9-40be-8762-5476a7a9ede4-sys\") pod \"tuned-f77jk\" (UID: \"545dd3b9-01a9-40be-8762-5476a7a9ede4\") " pod="openshift-cluster-node-tuning-operator/tuned-f77jk"
Oct 30 18:35:39.177390 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.177192    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8bcb59a4-381f-49ba-b732-f5e4658707de-log-socket\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:35:39.177390 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.177218    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx6qn\" (UniqueName: \"kubernetes.io/projected/089fdd4f-daa0-4b3e-b81e-26a764a526bb-kube-api-access-sx6qn\") pod \"network-metrics-daemon-9p48q\" (UID: \"089fdd4f-daa0-4b3e-b81e-26a764a526bb\") " pod="openshift-multus/network-metrics-daemon-9p48q"
Oct 30 18:35:39.177390 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.177233    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8bcb59a4-381f-49ba-b732-f5e4658707de-node-log\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:35:39.177390 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.177252    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/545dd3b9-01a9-40be-8762-5476a7a9ede4-etc-tuned\") pod \"tuned-f77jk\" (UID: \"545dd3b9-01a9-40be-8762-5476a7a9ede4\") " pod="openshift-cluster-node-tuning-operator/tuned-f77jk"
Oct 30 18:35:39.177390 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.177266    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8bcb59a4-381f-49ba-b732-f5e4658707de-env-overrides\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:35:39.177390 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.177283    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/42b71e2d-a8f2-43cf-8487-0e5da2338c54-hosts-file\") pod \"node-resolver-7z9gn\" (UID: \"42b71e2d-a8f2-43cf-8487-0e5da2338c54\") " pod="openshift-dns/node-resolver-7z9gn"
Oct 30 18:35:39.177390 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.177298    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8bcb59a4-381f-49ba-b732-f5e4658707de-host-cni-netd\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:35:39.177390 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.177317    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8bcb59a4-381f-49ba-b732-f5e4658707de-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:35:39.177390 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.177337    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/545dd3b9-01a9-40be-8762-5476a7a9ede4-etc-sysctl-conf\") pod \"tuned-f77jk\" (UID: \"545dd3b9-01a9-40be-8762-5476a7a9ede4\") " pod="openshift-cluster-node-tuning-operator/tuned-f77jk"
Oct 30 18:35:39.177390 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.177362    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/545dd3b9-01a9-40be-8762-5476a7a9ede4-etc-systemd\") pod \"tuned-f77jk\" (UID: \"545dd3b9-01a9-40be-8762-5476a7a9ede4\") " pod="openshift-cluster-node-tuning-operator/tuned-f77jk"
Oct 30 18:35:39.177390 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.177379    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/545dd3b9-01a9-40be-8762-5476a7a9ede4-lib-modules\") pod \"tuned-f77jk\" (UID: \"545dd3b9-01a9-40be-8762-5476a7a9ede4\") " pod="openshift-cluster-node-tuning-operator/tuned-f77jk"
Oct 30 18:35:39.177390 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.177391    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8bcb59a4-381f-49ba-b732-f5e4658707de-node-log\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:35:39.178155 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.177417    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8bcb59a4-381f-49ba-b732-f5e4658707de-etc-openvswitch\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:35:39.178155 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.177437    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8bcb59a4-381f-49ba-b732-f5e4658707de-host-cni-bin\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:35:39.178155 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.177456    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42b71e2d-a8f2-43cf-8487-0e5da2338c54-tmp-dir\") pod \"node-resolver-7z9gn\" (UID: \"42b71e2d-a8f2-43cf-8487-0e5da2338c54\") " pod="openshift-dns/node-resolver-7z9gn"
Oct 30 18:35:39.178155 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.177472    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm9cn\" (UniqueName: \"kubernetes.io/projected/42b71e2d-a8f2-43cf-8487-0e5da2338c54-kube-api-access-gm9cn\") pod \"node-resolver-7z9gn\" (UID: \"42b71e2d-a8f2-43cf-8487-0e5da2338c54\") " pod="openshift-dns/node-resolver-7z9gn"
Oct 30 18:35:39.178155 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.177497    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8bcb59a4-381f-49ba-b732-f5e4658707de-var-lib-openvswitch\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:35:39.178155 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.177518    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dltbj\" (UniqueName: \"kubernetes.io/projected/8bcb59a4-381f-49ba-b732-f5e4658707de-kube-api-access-dltbj\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:35:39.178155 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.177536    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7c3ae3e0-ddf1-414f-bb79-e4afc0527b26-original-pull-secret\") pod \"global-pull-secret-syncer-vhtmb\" (UID: \"7c3ae3e0-ddf1-414f-bb79-e4afc0527b26\") " pod="kube-system/global-pull-secret-syncer-vhtmb"
Oct 30 18:35:39.178155 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.177557    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8bcb59a4-381f-49ba-b732-f5e4658707de-run-systemd\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:35:39.178155 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.177563    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7c3ae3e0-ddf1-414f-bb79-e4afc0527b26-dbus\") pod \"global-pull-secret-syncer-vhtmb\" (UID: \"7c3ae3e0-ddf1-414f-bb79-e4afc0527b26\") " pod="kube-system/global-pull-secret-syncer-vhtmb"
Oct 30 18:35:39.178155 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.177577    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/545dd3b9-01a9-40be-8762-5476a7a9ede4-etc-modprobe-d\") pod \"tuned-f77jk\" (UID: \"545dd3b9-01a9-40be-8762-5476a7a9ede4\") " pod="openshift-cluster-node-tuning-operator/tuned-f77jk"
Oct 30 18:35:39.178155 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.177594    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/545dd3b9-01a9-40be-8762-5476a7a9ede4-etc-kubernetes\") pod \"tuned-f77jk\" (UID: \"545dd3b9-01a9-40be-8762-5476a7a9ede4\") " pod="openshift-cluster-node-tuning-operator/tuned-f77jk"
Oct 30 18:35:39.178155 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.177862    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/545dd3b9-01a9-40be-8762-5476a7a9ede4-etc-modprobe-d\") pod \"tuned-f77jk\" (UID: \"545dd3b9-01a9-40be-8762-5476a7a9ede4\") " pod="openshift-cluster-node-tuning-operator/tuned-f77jk"
Oct 30 18:35:39.178155 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.177903    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/545dd3b9-01a9-40be-8762-5476a7a9ede4-etc-kubernetes\") pod \"tuned-f77jk\" (UID: \"545dd3b9-01a9-40be-8762-5476a7a9ede4\") " pod="openshift-cluster-node-tuning-operator/tuned-f77jk"
Oct 30 18:35:39.178155 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.177909    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/42b71e2d-a8f2-43cf-8487-0e5da2338c54-hosts-file\") pod \"node-resolver-7z9gn\" (UID: \"42b71e2d-a8f2-43cf-8487-0e5da2338c54\") " pod="openshift-dns/node-resolver-7z9gn"
Oct 30 18:35:39.178155 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.177953    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8bcb59a4-381f-49ba-b732-f5e4658707de-log-socket\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:35:39.178155 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.177961    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/545dd3b9-01a9-40be-8762-5476a7a9ede4-sys\") pod \"tuned-f77jk\" (UID: \"545dd3b9-01a9-40be-8762-5476a7a9ede4\") " pod="openshift-cluster-node-tuning-operator/tuned-f77jk"
Oct 30 18:35:39.178155 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.177955    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/545dd3b9-01a9-40be-8762-5476a7a9ede4-etc-sysconfig\") pod \"tuned-f77jk\" (UID: \"545dd3b9-01a9-40be-8762-5476a7a9ede4\") " pod="openshift-cluster-node-tuning-operator/tuned-f77jk"
Oct 30 18:35:39.178155 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.177985    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8bcb59a4-381f-49ba-b732-f5e4658707de-etc-openvswitch\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:35:39.179004 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.177995    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8bcb59a4-381f-49ba-b732-f5e4658707de-host-cni-bin\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:35:39.179004 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.178003    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8bcb59a4-381f-49ba-b732-f5e4658707de-host-cni-netd\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:35:39.179004 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.178038    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8bcb59a4-381f-49ba-b732-f5e4658707de-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:35:39.179004 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.178076    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/545dd3b9-01a9-40be-8762-5476a7a9ede4-etc-sysctl-d\") pod \"tuned-f77jk\" (UID: \"545dd3b9-01a9-40be-8762-5476a7a9ede4\") " pod="openshift-cluster-node-tuning-operator/tuned-f77jk"
Oct 30 18:35:39.179004 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.177910    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/545dd3b9-01a9-40be-8762-5476a7a9ede4-etc-systemd\") pod \"tuned-f77jk\" (UID: \"545dd3b9-01a9-40be-8762-5476a7a9ede4\") " pod="openshift-cluster-node-tuning-operator/tuned-f77jk"
Oct 30 18:35:39.179004 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.178102    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/545dd3b9-01a9-40be-8762-5476a7a9ede4-var-lib-kubelet\") pod \"tuned-f77jk\" (UID: \"545dd3b9-01a9-40be-8762-5476a7a9ede4\") " pod="openshift-cluster-node-tuning-operator/tuned-f77jk"
Oct 30 18:35:39.179004 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.178203    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/545dd3b9-01a9-40be-8762-5476a7a9ede4-etc-sysctl-conf\") pod \"tuned-f77jk\" (UID: \"545dd3b9-01a9-40be-8762-5476a7a9ede4\") " pod="openshift-cluster-node-tuning-operator/tuned-f77jk"
Oct 30 18:35:39.179004 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.178209    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/545dd3b9-01a9-40be-8762-5476a7a9ede4-lib-modules\") pod \"tuned-f77jk\" (UID: \"545dd3b9-01a9-40be-8762-5476a7a9ede4\") " pod="openshift-cluster-node-tuning-operator/tuned-f77jk"
Oct 30 18:35:39.179004 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.178225    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8bcb59a4-381f-49ba-b732-f5e4658707de-run-systemd\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:35:39.179004 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.178256    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/545dd3b9-01a9-40be-8762-5476a7a9ede4-var-lib-kubelet\") pod \"tuned-f77jk\" (UID: \"545dd3b9-01a9-40be-8762-5476a7a9ede4\") " pod="openshift-cluster-node-tuning-operator/tuned-f77jk"
Oct 30 18:35:39.179004 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.178268    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8bcb59a4-381f-49ba-b732-f5e4658707de-var-lib-openvswitch\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:35:39.179004 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.178298    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/545dd3b9-01a9-40be-8762-5476a7a9ede4-etc-sysctl-d\") pod \"tuned-f77jk\" (UID: \"545dd3b9-01a9-40be-8762-5476a7a9ede4\") " pod="openshift-cluster-node-tuning-operator/tuned-f77jk"
Oct 30 18:35:39.179004 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.178316    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8bcb59a4-381f-49ba-b732-f5e4658707de-ovnkube-config\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:35:39.179004 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:39.178344    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 18:35:39.179004 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.178351    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8bcb59a4-381f-49ba-b732-f5e4658707de-ovnkube-script-lib\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:35:39.179004 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.178375    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7c3ae3e0-ddf1-414f-bb79-e4afc0527b26-kubelet-config\") pod \"global-pull-secret-syncer-vhtmb\" (UID: \"7c3ae3e0-ddf1-414f-bb79-e4afc0527b26\") " pod="kube-system/global-pull-secret-syncer-vhtmb"
Oct 30 18:35:39.179004 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.178400    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/089fdd4f-daa0-4b3e-b81e-26a764a526bb-metrics-certs\") pod \"network-metrics-daemon-9p48q\" (UID: \"089fdd4f-daa0-4b3e-b81e-26a764a526bb\") " pod="openshift-multus/network-metrics-daemon-9p48q"
Oct 30 18:35:39.179871 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:39.178423    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c3ae3e0-ddf1-414f-bb79-e4afc0527b26-original-pull-secret podName:7c3ae3e0-ddf1-414f-bb79-e4afc0527b26 nodeName:}" failed. No retries permitted until 2026-10-30 18:35:39.678400882 +0000 UTC m=+3.143339179 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7c3ae3e0-ddf1-414f-bb79-e4afc0527b26-original-pull-secret") pod "global-pull-secret-syncer-vhtmb" (UID: "7c3ae3e0-ddf1-414f-bb79-e4afc0527b26") : object "kube-system"/"original-pull-secret" not registered
Oct 30 18:35:39.179871 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.178455    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8bcb59a4-381f-49ba-b732-f5e4658707de-systemd-units\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:35:39.179871 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:39.178462    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:35:39.179871 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.178481    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8bcb59a4-381f-49ba-b732-f5e4658707de-host-slash\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:35:39.179871 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.178525    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8bcb59a4-381f-49ba-b732-f5e4658707de-host-slash\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:35:39.179871 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.178564    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8bcb59a4-381f-49ba-b732-f5e4658707de-systemd-units\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:35:39.179871 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:39.178572    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/089fdd4f-daa0-4b3e-b81e-26a764a526bb-metrics-certs podName:089fdd4f-daa0-4b3e-b81e-26a764a526bb nodeName:}" failed. No retries permitted until 2026-10-30 18:35:39.678497242 +0000 UTC m=+3.143435510 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/089fdd4f-daa0-4b3e-b81e-26a764a526bb-metrics-certs") pod "network-metrics-daemon-9p48q" (UID: "089fdd4f-daa0-4b3e-b81e-26a764a526bb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:35:39.179871 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.178622    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/545dd3b9-01a9-40be-8762-5476a7a9ede4-run\") pod \"tuned-f77jk\" (UID: \"545dd3b9-01a9-40be-8762-5476a7a9ede4\") " pod="openshift-cluster-node-tuning-operator/tuned-f77jk"
Oct 30 18:35:39.179871 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.178629    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7c3ae3e0-ddf1-414f-bb79-e4afc0527b26-kubelet-config\") pod \"global-pull-secret-syncer-vhtmb\" (UID: \"7c3ae3e0-ddf1-414f-bb79-e4afc0527b26\") " pod="kube-system/global-pull-secret-syncer-vhtmb"
Oct 30 18:35:39.179871 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.178666    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq2h8\" (UniqueName: \"kubernetes.io/projected/545dd3b9-01a9-40be-8762-5476a7a9ede4-kube-api-access-cq2h8\") pod \"tuned-f77jk\" (UID: \"545dd3b9-01a9-40be-8762-5476a7a9ede4\") " pod="openshift-cluster-node-tuning-operator/tuned-f77jk"
Oct 30 18:35:39.179871 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.178677    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/545dd3b9-01a9-40be-8762-5476a7a9ede4-run\") pod \"tuned-f77jk\" (UID: \"545dd3b9-01a9-40be-8762-5476a7a9ede4\") " pod="openshift-cluster-node-tuning-operator/tuned-f77jk"
Oct 30 18:35:39.179871 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.178728    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8bcb59a4-381f-49ba-b732-f5e4658707de-host-run-netns\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:35:39.179871 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.178764    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8bcb59a4-381f-49ba-b732-f5e4658707de-run-ovn\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:35:39.179871 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.178788    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/545dd3b9-01a9-40be-8762-5476a7a9ede4-host\") pod \"tuned-f77jk\" (UID: \"545dd3b9-01a9-40be-8762-5476a7a9ede4\") " pod="openshift-cluster-node-tuning-operator/tuned-f77jk"
Oct 30 18:35:39.179871 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.178812    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/545dd3b9-01a9-40be-8762-5476a7a9ede4-tmp\") pod \"tuned-f77jk\" (UID: \"545dd3b9-01a9-40be-8762-5476a7a9ede4\") " pod="openshift-cluster-node-tuning-operator/tuned-f77jk"
Oct 30 18:35:39.179871 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.178837    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8bcb59a4-381f-49ba-b732-f5e4658707de-host-kubelet\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:35:39.179871 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.178910    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8bcb59a4-381f-49ba-b732-f5e4658707de-host-kubelet\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:35:39.180621 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.178930    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8bcb59a4-381f-49ba-b732-f5e4658707de-env-overrides\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:35:39.180621 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.178938    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8bcb59a4-381f-49ba-b732-f5e4658707de-run-ovn\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:35:39.180621 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.178952    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/545dd3b9-01a9-40be-8762-5476a7a9ede4-host\") pod \"tuned-f77jk\" (UID: \"545dd3b9-01a9-40be-8762-5476a7a9ede4\") " pod="openshift-cluster-node-tuning-operator/tuned-f77jk"
Oct 30 18:35:39.180621 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.178974    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8bcb59a4-381f-49ba-b732-f5e4658707de-host-run-netns\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:35:39.180621 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.179473    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42b71e2d-a8f2-43cf-8487-0e5da2338c54-tmp-dir\") pod \"node-resolver-7z9gn\" (UID: \"42b71e2d-a8f2-43cf-8487-0e5da2338c54\") " pod="openshift-dns/node-resolver-7z9gn"
Oct 30 18:35:39.180621 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.179756    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8bcb59a4-381f-49ba-b732-f5e4658707de-ovnkube-config\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:35:39.180621 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.180229    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8bcb59a4-381f-49ba-b732-f5e4658707de-ovnkube-script-lib\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:35:39.180621 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.180388    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/545dd3b9-01a9-40be-8762-5476a7a9ede4-etc-tuned\") pod \"tuned-f77jk\" (UID: \"545dd3b9-01a9-40be-8762-5476a7a9ede4\") " pod="openshift-cluster-node-tuning-operator/tuned-f77jk"
Oct 30 18:35:39.181688 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.181641    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8bcb59a4-381f-49ba-b732-f5e4658707de-ovn-node-metrics-cert\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:35:39.181806 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.181783    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/545dd3b9-01a9-40be-8762-5476a7a9ede4-tmp\") pod \"tuned-f77jk\" (UID: \"545dd3b9-01a9-40be-8762-5476a7a9ede4\") " pod="openshift-cluster-node-tuning-operator/tuned-f77jk"
Oct 30 18:35:39.183903 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:39.183882    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 18:35:39.184027 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:39.183907    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 18:35:39.184027 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:39.183922    2573 projected.go:196] Error preparing data for projected volume kube-api-access-rfrpn for pod openshift-network-diagnostics/network-check-target-mcc4d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:35:39.184027 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:39.183979    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4b27cd9e-69b2-4480-a84f-5b463cec1a0e-kube-api-access-rfrpn podName:4b27cd9e-69b2-4480-a84f-5b463cec1a0e nodeName:}" failed. No retries permitted until 2026-10-30 18:35:39.683966954 +0000 UTC m=+3.148905235 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-rfrpn" (UniqueName: "kubernetes.io/projected/4b27cd9e-69b2-4480-a84f-5b463cec1a0e-kube-api-access-rfrpn") pod "network-check-target-mcc4d" (UID: "4b27cd9e-69b2-4480-a84f-5b463cec1a0e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:35:39.188285 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.187743    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx6qn\" (UniqueName: \"kubernetes.io/projected/089fdd4f-daa0-4b3e-b81e-26a764a526bb-kube-api-access-sx6qn\") pod \"network-metrics-daemon-9p48q\" (UID: \"089fdd4f-daa0-4b3e-b81e-26a764a526bb\") " pod="openshift-multus/network-metrics-daemon-9p48q"
Oct 30 18:35:39.188285 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.188240    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dltbj\" (UniqueName: \"kubernetes.io/projected/8bcb59a4-381f-49ba-b732-f5e4658707de-kube-api-access-dltbj\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:35:39.188458 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.188292    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq2h8\" (UniqueName: \"kubernetes.io/projected/545dd3b9-01a9-40be-8762-5476a7a9ede4-kube-api-access-cq2h8\") pod \"tuned-f77jk\" (UID: \"545dd3b9-01a9-40be-8762-5476a7a9ede4\") " pod="openshift-cluster-node-tuning-operator/tuned-f77jk"
Oct 30 18:35:39.188663 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.188640    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm9cn\" (UniqueName: \"kubernetes.io/projected/42b71e2d-a8f2-43cf-8487-0e5da2338c54-kube-api-access-gm9cn\") pod \"node-resolver-7z9gn\" (UID: \"42b71e2d-a8f2-43cf-8487-0e5da2338c54\") " pod="openshift-dns/node-resolver-7z9gn"
Oct 30 18:35:39.264598 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.264555    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rxbc9"
Oct 30 18:35:39.272144 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.272116    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dbbbq"
Oct 30 18:35:39.277505 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:39.277470    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77c2eb22_4b45_486f_8c62_bd14086942f5.slice/crio-0400f8430258e09dd25c90c28b8699012a50dcb20ec19647d0dd0c6053101622 WatchSource:0}: Error finding container 0400f8430258e09dd25c90c28b8699012a50dcb20ec19647d0dd0c6053101622: Status 404 returned error can't find the container with id 0400f8430258e09dd25c90c28b8699012a50dcb20ec19647d0dd0c6053101622
Oct 30 18:35:39.283576 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.283411    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mhtvp"
Oct 30 18:35:39.284241 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:39.284215    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd08a0fa_46bc_4250_985c_d2580a2225b8.slice/crio-d73ed64ee847d60fd293abba94e11add3369059a1974a6bea0d29f543c60916a WatchSource:0}: Error finding container d73ed64ee847d60fd293abba94e11add3369059a1974a6bea0d29f543c60916a: Status 404 returned error can't find the container with id d73ed64ee847d60fd293abba94e11add3369059a1974a6bea0d29f543c60916a
Oct 30 18:35:39.293288 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.293262    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4pz"
Oct 30 18:35:39.295558 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:39.295531    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod093a372e_74ea_4820_82c8_0dff7c201f10.slice/crio-dfcdc334a1ec738dcf8b9e141096e23049b36ce9a976f243903ac75b6adfdcf5 WatchSource:0}: Error finding container dfcdc334a1ec738dcf8b9e141096e23049b36ce9a976f243903ac75b6adfdcf5: Status 404 returned error can't find the container with id dfcdc334a1ec738dcf8b9e141096e23049b36ce9a976f243903ac75b6adfdcf5
Oct 30 18:35:39.299660 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.299635    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4lmrw"
Oct 30 18:35:39.304212 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:39.304186    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ca560f4_9aa8_4bc6_9db1_6992a1fa7c1f.slice/crio-376c012613cf5f19aaba03fb9e8f0bcbf03ca640aae6d8e0272d46455bf0a9dd WatchSource:0}: Error finding container 376c012613cf5f19aaba03fb9e8f0bcbf03ca640aae6d8e0272d46455bf0a9dd: Status 404 returned error can't find the container with id 376c012613cf5f19aaba03fb9e8f0bcbf03ca640aae6d8e0272d46455bf0a9dd
Oct 30 18:35:39.310121 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:39.310099    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8f39e97_7f68_47da_a247_adf78b523afd.slice/crio-48f922805b42fae668301dd5c3283d63659b7313d58239bb7c7c369ca9b21b75 WatchSource:0}: Error finding container 48f922805b42fae668301dd5c3283d63659b7313d58239bb7c7c369ca9b21b75: Status 404 returned error can't find the container with id 48f922805b42fae668301dd5c3283d63659b7313d58239bb7c7c369ca9b21b75
Oct 30 18:35:39.321581 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.321549    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-f77jk"
Oct 30 18:35:39.324051 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.324028    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-tjtmv"
Oct 30 18:35:39.326941 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.326916    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7z9gn"
Oct 30 18:35:39.335679 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:39.335634    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod545dd3b9_01a9_40be_8762_5476a7a9ede4.slice/crio-d913a28274a5d2e62987f8b4f8abf7a62f5b5966493a449d6307a7fd8eee202f WatchSource:0}: Error finding container d913a28274a5d2e62987f8b4f8abf7a62f5b5966493a449d6307a7fd8eee202f: Status 404 returned error can't find the container with id d913a28274a5d2e62987f8b4f8abf7a62f5b5966493a449d6307a7fd8eee202f
Oct 30 18:35:39.349348 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.349325    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:35:39.360822 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:35:39.360790    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bcb59a4_381f_49ba_b732_f5e4658707de.slice/crio-4a20442cabc818c37d363f8a47e7e379db232f5f58eb34c3c35d902407158b61 WatchSource:0}: Error finding container 4a20442cabc818c37d363f8a47e7e379db232f5f58eb34c3c35d902407158b61: Status 404 returned error can't find the container with id 4a20442cabc818c37d363f8a47e7e379db232f5f58eb34c3c35d902407158b61
Oct 30 18:35:39.681381 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.681343    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7c3ae3e0-ddf1-414f-bb79-e4afc0527b26-original-pull-secret\") pod \"global-pull-secret-syncer-vhtmb\" (UID: \"7c3ae3e0-ddf1-414f-bb79-e4afc0527b26\") " pod="kube-system/global-pull-secret-syncer-vhtmb"
Oct 30 18:35:39.681604 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.681406    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/089fdd4f-daa0-4b3e-b81e-26a764a526bb-metrics-certs\") pod \"network-metrics-daemon-9p48q\" (UID: \"089fdd4f-daa0-4b3e-b81e-26a764a526bb\") " pod="openshift-multus/network-metrics-daemon-9p48q"
Oct 30 18:35:39.681604 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:39.681455    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 18:35:39.681604 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:39.681522    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c3ae3e0-ddf1-414f-bb79-e4afc0527b26-original-pull-secret podName:7c3ae3e0-ddf1-414f-bb79-e4afc0527b26 nodeName:}" failed. No retries permitted until 2026-10-30 18:35:40.681506943 +0000 UTC m=+4.146445210 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7c3ae3e0-ddf1-414f-bb79-e4afc0527b26-original-pull-secret") pod "global-pull-secret-syncer-vhtmb" (UID: "7c3ae3e0-ddf1-414f-bb79-e4afc0527b26") : object "kube-system"/"original-pull-secret" not registered
Oct 30 18:35:39.681604 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:39.681544    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:35:39.681604 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:39.681595    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/089fdd4f-daa0-4b3e-b81e-26a764a526bb-metrics-certs podName:089fdd4f-daa0-4b3e-b81e-26a764a526bb nodeName:}" failed. No retries permitted until 2026-10-30 18:35:40.681579852 +0000 UTC m=+4.146518119 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/089fdd4f-daa0-4b3e-b81e-26a764a526bb-metrics-certs") pod "network-metrics-daemon-9p48q" (UID: "089fdd4f-daa0-4b3e-b81e-26a764a526bb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:35:39.782743 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.782716    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfrpn\" (UniqueName: \"kubernetes.io/projected/4b27cd9e-69b2-4480-a84f-5b463cec1a0e-kube-api-access-rfrpn\") pod \"network-check-target-mcc4d\" (UID: \"4b27cd9e-69b2-4480-a84f-5b463cec1a0e\") " pod="openshift-network-diagnostics/network-check-target-mcc4d"
Oct 30 18:35:39.782925 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:39.782897    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 18:35:39.782925 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:39.782932    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 18:35:39.782925 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:39.782946    2573 projected.go:196] Error preparing data for projected volume kube-api-access-rfrpn for pod openshift-network-diagnostics/network-check-target-mcc4d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:35:39.783232 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:39.783021    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4b27cd9e-69b2-4480-a84f-5b463cec1a0e-kube-api-access-rfrpn podName:4b27cd9e-69b2-4480-a84f-5b463cec1a0e nodeName:}" failed. No retries permitted until 2026-10-30 18:35:40.782999695 +0000 UTC m=+4.247937977 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-rfrpn" (UniqueName: "kubernetes.io/projected/4b27cd9e-69b2-4480-a84f-5b463cec1a0e-kube-api-access-rfrpn") pod "network-check-target-mcc4d" (UID: "4b27cd9e-69b2-4480-a84f-5b463cec1a0e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:35:39.878416 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.878386    2573 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:35:39.998928 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.998844    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 18:30:39 +0000 UTC" deadline="2028-06-06 22:22:31.771893418 +0000 UTC"
Oct 30 18:35:39.998928 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:39.998907    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14043h46m51.772990831s"
Oct 30 18:35:40.038160 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:40.038130    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vhtmb"
Oct 30 18:35:40.038312 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:40.038273    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vhtmb" podUID="7c3ae3e0-ddf1-414f-bb79-e4afc0527b26"
Oct 30 18:35:40.065967 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:40.065523    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p" event={"ID":"8bcb59a4-381f-49ba-b732-f5e4658707de","Type":"ContainerStarted","Data":"4a20442cabc818c37d363f8a47e7e379db232f5f58eb34c3c35d902407158b61"}
Oct 30 18:35:40.072786 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:40.072677    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tjtmv" event={"ID":"0dc90efb-2709-4894-b457-6cadb8abd801","Type":"ContainerStarted","Data":"8884baec4ce0a848bd5a0ba3a8715596d06504ac637370760df9784b2cb4d109"}
Oct 30 18:35:40.083026 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:40.082984    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4lmrw" event={"ID":"d8f39e97-7f68-47da-a247-adf78b523afd","Type":"ContainerStarted","Data":"48f922805b42fae668301dd5c3283d63659b7313d58239bb7c7c369ca9b21b75"}
Oct 30 18:35:40.093221 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:40.093135    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mhtvp" event={"ID":"093a372e-74ea-4820-82c8-0dff7c201f10","Type":"ContainerStarted","Data":"dfcdc334a1ec738dcf8b9e141096e23049b36ce9a976f243903ac75b6adfdcf5"}
Oct 30 18:35:40.115772 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:40.115613    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dbbbq" event={"ID":"bd08a0fa-46bc-4250-985c-d2580a2225b8","Type":"ContainerStarted","Data":"d73ed64ee847d60fd293abba94e11add3369059a1974a6bea0d29f543c60916a"}
Oct 30 18:35:40.125132 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:40.124251    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-135.us-east-2.compute.internal" event={"ID":"826c9de932b6871da80993c953a89ebf","Type":"ContainerStarted","Data":"231a010045f4737301afbc46f96f56e8e703a0728c9b69fcadfe8745163910c0"}
Oct 30 18:35:40.130972 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:40.130897    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7z9gn" event={"ID":"42b71e2d-a8f2-43cf-8487-0e5da2338c54","Type":"ContainerStarted","Data":"1991b9540f8953c91e896fce71e99b569200da8ee8d7b0b13aeba2742cbc600d"}
Oct 30 18:35:40.135784 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:40.135707    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-f77jk" event={"ID":"545dd3b9-01a9-40be-8762-5476a7a9ede4","Type":"ContainerStarted","Data":"d913a28274a5d2e62987f8b4f8abf7a62f5b5966493a449d6307a7fd8eee202f"}
Oct 30 18:35:40.137346 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:40.137162    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4pz" event={"ID":"2ca560f4-9aa8-4bc6-9db1-6992a1fa7c1f","Type":"ContainerStarted","Data":"376c012613cf5f19aaba03fb9e8f0bcbf03ca640aae6d8e0272d46455bf0a9dd"}
Oct 30 18:35:40.140236 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:40.140210    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rxbc9" event={"ID":"77c2eb22-4b45-486f-8c62-bd14086942f5","Type":"ContainerStarted","Data":"0400f8430258e09dd25c90c28b8699012a50dcb20ec19647d0dd0c6053101622"}
Oct 30 18:35:40.140411 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:40.140313    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-135.us-east-2.compute.internal" podStartSLOduration=1.140295737 podStartE2EDuration="1.140295737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:35:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:35:40.139822761 +0000 UTC m=+3.604761054" watchObservedRunningTime="2026-10-30 18:35:40.140295737 +0000 UTC m=+3.605234029"
Oct 30 18:35:40.691274 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:40.691223    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/089fdd4f-daa0-4b3e-b81e-26a764a526bb-metrics-certs\") pod \"network-metrics-daemon-9p48q\" (UID: \"089fdd4f-daa0-4b3e-b81e-26a764a526bb\") " pod="openshift-multus/network-metrics-daemon-9p48q"
Oct 30 18:35:40.691457 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:40.691330    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7c3ae3e0-ddf1-414f-bb79-e4afc0527b26-original-pull-secret\") pod \"global-pull-secret-syncer-vhtmb\" (UID: \"7c3ae3e0-ddf1-414f-bb79-e4afc0527b26\") " pod="kube-system/global-pull-secret-syncer-vhtmb"
Oct 30 18:35:40.691523 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:40.691466    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 18:35:40.691587 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:40.691532    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c3ae3e0-ddf1-414f-bb79-e4afc0527b26-original-pull-secret podName:7c3ae3e0-ddf1-414f-bb79-e4afc0527b26 nodeName:}" failed. No retries permitted until 2026-10-30 18:35:42.69151347 +0000 UTC m=+6.156451736 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7c3ae3e0-ddf1-414f-bb79-e4afc0527b26-original-pull-secret") pod "global-pull-secret-syncer-vhtmb" (UID: "7c3ae3e0-ddf1-414f-bb79-e4afc0527b26") : object "kube-system"/"original-pull-secret" not registered
Oct 30 18:35:40.691911 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:40.691807    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:35:40.691911 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:40.691889    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/089fdd4f-daa0-4b3e-b81e-26a764a526bb-metrics-certs podName:089fdd4f-daa0-4b3e-b81e-26a764a526bb nodeName:}" failed. No retries permitted until 2026-10-30 18:35:42.691867325 +0000 UTC m=+6.156805606 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/089fdd4f-daa0-4b3e-b81e-26a764a526bb-metrics-certs") pod "network-metrics-daemon-9p48q" (UID: "089fdd4f-daa0-4b3e-b81e-26a764a526bb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:35:40.792668 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:40.792620    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfrpn\" (UniqueName: \"kubernetes.io/projected/4b27cd9e-69b2-4480-a84f-5b463cec1a0e-kube-api-access-rfrpn\") pod \"network-check-target-mcc4d\" (UID: \"4b27cd9e-69b2-4480-a84f-5b463cec1a0e\") " pod="openshift-network-diagnostics/network-check-target-mcc4d"
Oct 30 18:35:40.792886 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:40.792865    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 18:35:40.792958 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:40.792900    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 18:35:40.792958 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:40.792921    2573 projected.go:196] Error preparing data for projected volume kube-api-access-rfrpn for pod openshift-network-diagnostics/network-check-target-mcc4d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:35:40.793071 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:40.792989    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4b27cd9e-69b2-4480-a84f-5b463cec1a0e-kube-api-access-rfrpn podName:4b27cd9e-69b2-4480-a84f-5b463cec1a0e nodeName:}" failed. No retries permitted until 2026-10-30 18:35:42.79296796 +0000 UTC m=+6.257906242 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-rfrpn" (UniqueName: "kubernetes.io/projected/4b27cd9e-69b2-4480-a84f-5b463cec1a0e-kube-api-access-rfrpn") pod "network-check-target-mcc4d" (UID: "4b27cd9e-69b2-4480-a84f-5b463cec1a0e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:35:40.999808 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:40.999676    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 18:30:39 +0000 UTC" deadline="2028-07-01 22:26:00.135273894 +0000 UTC"
Oct 30 18:35:40.999808 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:40.999742    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14643h50m19.135537603s"
Oct 30 18:35:41.040456 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:41.040422    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mcc4d"
Oct 30 18:35:41.040633 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:41.040564    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mcc4d" podUID="4b27cd9e-69b2-4480-a84f-5b463cec1a0e"
Oct 30 18:35:41.041081 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:41.041046    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9p48q"
Oct 30 18:35:41.041191 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:41.041169    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9p48q" podUID="089fdd4f-daa0-4b3e-b81e-26a764a526bb"
Oct 30 18:35:41.154795 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:41.154524    2573 generic.go:356] "Generic (PLEG): container finished" podID="09a9d0b6ec5615a60675fe6f5b7fa6e4" containerID="c0efb16a78c761c15493d05fc695c2c98e1a0d5485a24acdea8e26c6ef70f5b0" exitCode=0
Oct 30 18:35:41.154795 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:41.154636    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-135.us-east-2.compute.internal" event={"ID":"09a9d0b6ec5615a60675fe6f5b7fa6e4","Type":"ContainerDied","Data":"c0efb16a78c761c15493d05fc695c2c98e1a0d5485a24acdea8e26c6ef70f5b0"}
Oct 30 18:35:42.037867 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:42.037834    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vhtmb"
Oct 30 18:35:42.038371 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:42.037996    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vhtmb" podUID="7c3ae3e0-ddf1-414f-bb79-e4afc0527b26"
Oct 30 18:35:42.164868 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:42.164250    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-135.us-east-2.compute.internal" event={"ID":"09a9d0b6ec5615a60675fe6f5b7fa6e4","Type":"ContainerStarted","Data":"83115c77c0240b5b190e823840d18122d63da32b5198af445316537bb28faa8b"}
Oct 30 18:35:42.709083 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:42.709039    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7c3ae3e0-ddf1-414f-bb79-e4afc0527b26-original-pull-secret\") pod \"global-pull-secret-syncer-vhtmb\" (UID: \"7c3ae3e0-ddf1-414f-bb79-e4afc0527b26\") " pod="kube-system/global-pull-secret-syncer-vhtmb"
Oct 30 18:35:42.709271 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:42.709110    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/089fdd4f-daa0-4b3e-b81e-26a764a526bb-metrics-certs\") pod \"network-metrics-daemon-9p48q\" (UID: \"089fdd4f-daa0-4b3e-b81e-26a764a526bb\") " pod="openshift-multus/network-metrics-daemon-9p48q"
Oct 30 18:35:42.709346 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:42.709280    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:35:42.709417 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:42.709385    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/089fdd4f-daa0-4b3e-b81e-26a764a526bb-metrics-certs podName:089fdd4f-daa0-4b3e-b81e-26a764a526bb nodeName:}" failed. No retries permitted until 2026-10-30 18:35:46.709329133 +0000 UTC m=+10.174267415 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/089fdd4f-daa0-4b3e-b81e-26a764a526bb-metrics-certs") pod "network-metrics-daemon-9p48q" (UID: "089fdd4f-daa0-4b3e-b81e-26a764a526bb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:35:42.709902 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:42.709877    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 18:35:42.709991 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:42.709940    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c3ae3e0-ddf1-414f-bb79-e4afc0527b26-original-pull-secret podName:7c3ae3e0-ddf1-414f-bb79-e4afc0527b26 nodeName:}" failed. No retries permitted until 2026-10-30 18:35:46.709924433 +0000 UTC m=+10.174862707 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7c3ae3e0-ddf1-414f-bb79-e4afc0527b26-original-pull-secret") pod "global-pull-secret-syncer-vhtmb" (UID: "7c3ae3e0-ddf1-414f-bb79-e4afc0527b26") : object "kube-system"/"original-pull-secret" not registered
Oct 30 18:35:42.810031 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:42.809980    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfrpn\" (UniqueName: \"kubernetes.io/projected/4b27cd9e-69b2-4480-a84f-5b463cec1a0e-kube-api-access-rfrpn\") pod \"network-check-target-mcc4d\" (UID: \"4b27cd9e-69b2-4480-a84f-5b463cec1a0e\") " pod="openshift-network-diagnostics/network-check-target-mcc4d"
Oct 30 18:35:42.810219 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:42.810165    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 18:35:42.810219 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:42.810189    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 18:35:42.810219 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:42.810202    2573 projected.go:196] Error preparing data for projected volume kube-api-access-rfrpn for pod openshift-network-diagnostics/network-check-target-mcc4d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:35:42.810358 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:42.810267    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4b27cd9e-69b2-4480-a84f-5b463cec1a0e-kube-api-access-rfrpn podName:4b27cd9e-69b2-4480-a84f-5b463cec1a0e nodeName:}" failed. No retries permitted until 2026-10-30 18:35:46.810247685 +0000 UTC m=+10.275185968 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-rfrpn" (UniqueName: "kubernetes.io/projected/4b27cd9e-69b2-4480-a84f-5b463cec1a0e-kube-api-access-rfrpn") pod "network-check-target-mcc4d" (UID: "4b27cd9e-69b2-4480-a84f-5b463cec1a0e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:35:43.045935 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:43.045127    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9p48q"
Oct 30 18:35:43.045935 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:43.045273    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9p48q" podUID="089fdd4f-daa0-4b3e-b81e-26a764a526bb"
Oct 30 18:35:43.045935 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:43.045805    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mcc4d"
Oct 30 18:35:43.045935 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:43.045888    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mcc4d" podUID="4b27cd9e-69b2-4480-a84f-5b463cec1a0e"
Oct 30 18:35:43.178709 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:43.178307    2573 generic.go:356] "Generic (PLEG): container finished" podID="0dc90efb-2709-4894-b457-6cadb8abd801" containerID="e3cd51e8ef968b52b94948d7197ce440bc25aed7c48a7226fc6a55af880bf397" exitCode=0
Oct 30 18:35:43.180431 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:43.179498    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tjtmv" event={"ID":"0dc90efb-2709-4894-b457-6cadb8abd801","Type":"ContainerDied","Data":"e3cd51e8ef968b52b94948d7197ce440bc25aed7c48a7226fc6a55af880bf397"}
Oct 30 18:35:43.211891 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:43.210932    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-135.us-east-2.compute.internal" podStartSLOduration=4.210912369 podStartE2EDuration="4.210912369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:35:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:35:42.181285056 +0000 UTC m=+5.646223348" watchObservedRunningTime="2026-10-30 18:35:43.210912369 +0000 UTC m=+6.675850660"
Oct 30 18:35:44.037699 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:44.037635    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vhtmb"
Oct 30 18:35:44.037888 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:44.037806    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vhtmb" podUID="7c3ae3e0-ddf1-414f-bb79-e4afc0527b26"
Oct 30 18:35:44.192071 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:44.192027    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dbbbq" event={"ID":"bd08a0fa-46bc-4250-985c-d2580a2225b8","Type":"ContainerStarted","Data":"bb4a80a906f379f6f2f39726f2f9378e8f7b294175c252231f95354aa7f1e9e1"}
Oct 30 18:35:45.040801 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:45.040771    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mcc4d"
Oct 30 18:35:45.040967 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:45.040892    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mcc4d" podUID="4b27cd9e-69b2-4480-a84f-5b463cec1a0e"
Oct 30 18:35:45.041345 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:45.041269    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9p48q"
Oct 30 18:35:45.041464 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:45.041390    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9p48q" podUID="089fdd4f-daa0-4b3e-b81e-26a764a526bb"
Oct 30 18:35:45.198550 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:45.198505    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mhtvp" event={"ID":"093a372e-74ea-4820-82c8-0dff7c201f10","Type":"ContainerStarted","Data":"e9ccacb155593ca9c4eee3aa49eba3fff0c5f153e0000f6addb07d4225a45894"}
Oct 30 18:35:45.214794 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:45.214715    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-dbbbq" podStartSLOduration=3.473450031 podStartE2EDuration="7.2146936s" totalImagesPullingTime="3.741243569s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:35:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:35:44.210671925 +0000 UTC m=+7.675610211" watchObservedRunningTime="2026-10-30 18:35:45.2146936 +0000 UTC m=+8.679631893"
Oct 30 18:35:46.037694 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:46.037605    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vhtmb"
Oct 30 18:35:46.037890 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:46.037780    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vhtmb" podUID="7c3ae3e0-ddf1-414f-bb79-e4afc0527b26"
Oct 30 18:35:46.204168 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:46.204129    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4pz" event={"ID":"2ca560f4-9aa8-4bc6-9db1-6992a1fa7c1f","Type":"ContainerStarted","Data":"0aa82debd8264ce64044c7a053216f6d97e33d6cb67445a47aa9ef7ac2739e13"}
Oct 30 18:35:46.746085 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:46.745811    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7c3ae3e0-ddf1-414f-bb79-e4afc0527b26-original-pull-secret\") pod \"global-pull-secret-syncer-vhtmb\" (UID: \"7c3ae3e0-ddf1-414f-bb79-e4afc0527b26\") " pod="kube-system/global-pull-secret-syncer-vhtmb"
Oct 30 18:35:46.746085 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:46.745921    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/089fdd4f-daa0-4b3e-b81e-26a764a526bb-metrics-certs\") pod \"network-metrics-daemon-9p48q\" (UID: \"089fdd4f-daa0-4b3e-b81e-26a764a526bb\") " pod="openshift-multus/network-metrics-daemon-9p48q"
Oct 30 18:35:46.746085 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:46.746002    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 18:35:46.746085 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:46.746038    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:35:46.746085 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:46.746070    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c3ae3e0-ddf1-414f-bb79-e4afc0527b26-original-pull-secret podName:7c3ae3e0-ddf1-414f-bb79-e4afc0527b26 nodeName:}" failed. No retries permitted until 2026-10-30 18:35:54.746051415 +0000 UTC m=+18.210989696 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7c3ae3e0-ddf1-414f-bb79-e4afc0527b26-original-pull-secret") pod "global-pull-secret-syncer-vhtmb" (UID: "7c3ae3e0-ddf1-414f-bb79-e4afc0527b26") : object "kube-system"/"original-pull-secret" not registered
Oct 30 18:35:46.746085 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:46.746091    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/089fdd4f-daa0-4b3e-b81e-26a764a526bb-metrics-certs podName:089fdd4f-daa0-4b3e-b81e-26a764a526bb nodeName:}" failed. No retries permitted until 2026-10-30 18:35:54.746081764 +0000 UTC m=+18.211020036 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/089fdd4f-daa0-4b3e-b81e-26a764a526bb-metrics-certs") pod "network-metrics-daemon-9p48q" (UID: "089fdd4f-daa0-4b3e-b81e-26a764a526bb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:35:46.846358 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:46.846316    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfrpn\" (UniqueName: \"kubernetes.io/projected/4b27cd9e-69b2-4480-a84f-5b463cec1a0e-kube-api-access-rfrpn\") pod \"network-check-target-mcc4d\" (UID: \"4b27cd9e-69b2-4480-a84f-5b463cec1a0e\") " pod="openshift-network-diagnostics/network-check-target-mcc4d"
Oct 30 18:35:46.846538 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:46.846492    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 18:35:46.846538 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:46.846516    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 18:35:46.846538 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:46.846529    2573 projected.go:196] Error preparing data for projected volume kube-api-access-rfrpn for pod openshift-network-diagnostics/network-check-target-mcc4d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:35:46.846701 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:46.846593    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4b27cd9e-69b2-4480-a84f-5b463cec1a0e-kube-api-access-rfrpn podName:4b27cd9e-69b2-4480-a84f-5b463cec1a0e nodeName:}" failed. No retries permitted until 2026-10-30 18:35:54.846572934 +0000 UTC m=+18.311511222 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-rfrpn" (UniqueName: "kubernetes.io/projected/4b27cd9e-69b2-4480-a84f-5b463cec1a0e-kube-api-access-rfrpn") pod "network-check-target-mcc4d" (UID: "4b27cd9e-69b2-4480-a84f-5b463cec1a0e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:35:47.038919 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:47.038848    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mcc4d"
Oct 30 18:35:47.039057 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:47.038973    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mcc4d" podUID="4b27cd9e-69b2-4480-a84f-5b463cec1a0e"
Oct 30 18:35:47.039429 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:47.039406    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9p48q"
Oct 30 18:35:47.039550 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:47.039528    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9p48q" podUID="089fdd4f-daa0-4b3e-b81e-26a764a526bb"
Oct 30 18:35:47.517150 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:47.517105    2573 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 18:35:48.029878 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:48.029781    2573 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T18:35:47.51713591Z","UUID":"0c44c709-88d8-4c29-a9a1-72337274bda6","Handler":null,"Name":"","Endpoint":""}
Oct 30 18:35:48.031379 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:48.031349    2573 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 18:35:48.031379 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:48.031381    2573 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 18:35:48.037938 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:48.037900    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vhtmb"
Oct 30 18:35:48.038066 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:48.038040    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vhtmb" podUID="7c3ae3e0-ddf1-414f-bb79-e4afc0527b26"
Oct 30 18:35:48.211562 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:48.211532    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7z9gn" event={"ID":"42b71e2d-a8f2-43cf-8487-0e5da2338c54","Type":"ContainerStarted","Data":"93c0096d87515bbcc31fea0e73eb64c9013b5b49b1b4af5172aa6f58d5f14f17"}
Oct 30 18:35:48.214856 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:48.214822    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4pz" event={"ID":"2ca560f4-9aa8-4bc6-9db1-6992a1fa7c1f","Type":"ContainerStarted","Data":"c78f6110a36010adc1d69b3fa443e061fcf284bd8cda448e8b638295e3408f5e"}
Oct 30 18:35:48.219168 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:48.219074    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tjtmv" event={"ID":"0dc90efb-2709-4894-b457-6cadb8abd801","Type":"ContainerStarted","Data":"3ca28e215731896dd4a45198e7016f158b7cfdbeef77b5b91d5f57b79d58b3c0"}
Oct 30 18:35:48.223236 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:48.221426    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4lmrw" event={"ID":"d8f39e97-7f68-47da-a247-adf78b523afd","Type":"ContainerStarted","Data":"dad2a651d489ca82bba9e666a557a28a48c0f95fbc7fa0f81256d38dfe98afeb"}
Oct 30 18:35:48.228522 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:48.227016    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-mhtvp" podStartSLOduration=4.968938659 podStartE2EDuration="10.22699912s" totalImagesPullingTime="5.258060461s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:35:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:35:45.215582899 +0000 UTC m=+8.680521193" watchObservedRunningTime="2026-10-30 18:35:48.22699912 +0000 UTC m=+11.691937413"
Oct 30 18:35:48.247135 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:48.247065    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-7z9gn" podStartSLOduration=1.614644279 podStartE2EDuration="10.24704821s" totalImagesPullingTime="8.632403931s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:35:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:35:48.227893786 +0000 UTC m=+11.692832078" watchObservedRunningTime="2026-10-30 18:35:48.24704821 +0000 UTC m=+11.711986501"
Oct 30 18:35:48.261513 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:48.261469    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-4lmrw" podStartSLOduration=2.084035488 podStartE2EDuration="10.261457837s" totalImagesPullingTime="8.177422349s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:35:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:35:48.261193434 +0000 UTC m=+11.726131726" watchObservedRunningTime="2026-10-30 18:35:48.261457837 +0000 UTC m=+11.726396127"
Oct 30 18:35:49.038917 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:49.038583    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9p48q"
Oct 30 18:35:49.039363 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:49.038672    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mcc4d"
Oct 30 18:35:49.039363 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:49.039008    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9p48q" podUID="089fdd4f-daa0-4b3e-b81e-26a764a526bb"
Oct 30 18:35:49.039363 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:49.039073    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mcc4d" podUID="4b27cd9e-69b2-4480-a84f-5b463cec1a0e"
Oct 30 18:35:49.182838 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:49.182804    2573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-dbbbq"
Oct 30 18:35:49.183714 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:49.183692    2573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-dbbbq"
Oct 30 18:35:49.226563 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:49.226466    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4pz" event={"ID":"2ca560f4-9aa8-4bc6-9db1-6992a1fa7c1f","Type":"ContainerStarted","Data":"1ac419321cd0f593c251e51b5bbac03af6d2af5627315414e1557f5710e8c36f"}
Oct 30 18:35:49.228558 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:49.228526    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rxbc9" event={"ID":"77c2eb22-4b45-486f-8c62-bd14086942f5","Type":"ContainerStarted","Data":"9d3ac2eee7dd8ee515a4922efd0bf26e1d29f5b15aa7fabdf3aedf1b3c6de495"}
Oct 30 18:35:49.230986 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:49.230957    2573 generic.go:356] "Generic (PLEG): container finished" podID="0dc90efb-2709-4894-b457-6cadb8abd801" containerID="3ca28e215731896dd4a45198e7016f158b7cfdbeef77b5b91d5f57b79d58b3c0" exitCode=0
Oct 30 18:35:49.231131 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:49.231111    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tjtmv" event={"ID":"0dc90efb-2709-4894-b457-6cadb8abd801","Type":"ContainerDied","Data":"3ca28e215731896dd4a45198e7016f158b7cfdbeef77b5b91d5f57b79d58b3c0"}
Oct 30 18:35:49.243411 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:49.243360    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4pz" podStartSLOduration=2.280464535 podStartE2EDuration="11.243346506s" totalImagesPullingTime="8.962881971s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:35:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:35:49.243030828 +0000 UTC m=+12.707969121" watchObservedRunningTime="2026-10-30 18:35:49.243346506 +0000 UTC m=+12.708284802"
Oct 30 18:35:50.038617 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:50.038573    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vhtmb"
Oct 30 18:35:50.038842 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:50.038731    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vhtmb" podUID="7c3ae3e0-ddf1-414f-bb79-e4afc0527b26"
Oct 30 18:35:51.037732 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:51.037664    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mcc4d"
Oct 30 18:35:51.038720 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:51.037862    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mcc4d" podUID="4b27cd9e-69b2-4480-a84f-5b463cec1a0e"
Oct 30 18:35:51.038720 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:51.037906    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9p48q"
Oct 30 18:35:51.038720 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:51.038015    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9p48q" podUID="089fdd4f-daa0-4b3e-b81e-26a764a526bb"
Oct 30 18:35:51.238571 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:51.238535    2573 generic.go:356] "Generic (PLEG): container finished" podID="0dc90efb-2709-4894-b457-6cadb8abd801" containerID="239187d4ffab1cadbb9bfb61ca3ef1fd67984657e946b634f95668fdbc538d4a" exitCode=0
Oct 30 18:35:51.238766 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:51.238629    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tjtmv" event={"ID":"0dc90efb-2709-4894-b457-6cadb8abd801","Type":"ContainerDied","Data":"239187d4ffab1cadbb9bfb61ca3ef1fd67984657e946b634f95668fdbc538d4a"}
Oct 30 18:35:51.240542 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:51.240513    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-f77jk" event={"ID":"545dd3b9-01a9-40be-8762-5476a7a9ede4","Type":"ContainerStarted","Data":"4d91b11ed4769fdbc60e4fb5ba1ec448cf5ad5bc4d4e3be71081dc3f5240ae81"}
Oct 30 18:35:51.257795 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:51.257742    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-rxbc9" podStartSLOduration=4.667109818 podStartE2EDuration="13.257729527s" totalImagesPullingTime="8.590619709s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:35:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:35:50.247624751 +0000 UTC m=+13.712563041" watchObservedRunningTime="2026-10-30 18:35:51.257729527 +0000 UTC m=+14.722667821"
Oct 30 18:35:51.271008 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:51.270965    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-f77jk" podStartSLOduration=2.269321065 podStartE2EDuration="13.270950627s" totalImagesPullingTime="11.001629562s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:35:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:35:51.27048097 +0000 UTC m=+14.735419257" watchObservedRunningTime="2026-10-30 18:35:51.270950627 +0000 UTC m=+14.735888913"
Oct 30 18:35:52.038582 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:52.038550    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vhtmb"
Oct 30 18:35:52.039057 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:52.038701    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vhtmb" podUID="7c3ae3e0-ddf1-414f-bb79-e4afc0527b26"
Oct 30 18:35:53.037914 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:53.037703    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mcc4d"
Oct 30 18:35:53.038098 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:53.038006    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mcc4d" podUID="4b27cd9e-69b2-4480-a84f-5b463cec1a0e"
Oct 30 18:35:53.038731 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:53.038714    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9p48q"
Oct 30 18:35:53.039055 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:53.038793    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9p48q" podUID="089fdd4f-daa0-4b3e-b81e-26a764a526bb"
Oct 30 18:35:53.246279 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:53.246245    2573 generic.go:356] "Generic (PLEG): container finished" podID="0dc90efb-2709-4894-b457-6cadb8abd801" containerID="773700e87b709b16e985402961ac1cdcd01308643cd3d40fa73b940303311e34" exitCode=0
Oct 30 18:35:53.246438 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:53.246298    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tjtmv" event={"ID":"0dc90efb-2709-4894-b457-6cadb8abd801","Type":"ContainerDied","Data":"773700e87b709b16e985402961ac1cdcd01308643cd3d40fa73b940303311e34"}
Oct 30 18:35:54.038316 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:54.038277    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vhtmb"
Oct 30 18:35:54.038569 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:54.038416    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vhtmb" podUID="7c3ae3e0-ddf1-414f-bb79-e4afc0527b26"
Oct 30 18:35:54.569410 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:54.569376    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-dbbbq"
Oct 30 18:35:54.569955 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:54.569525    2573 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 18:35:54.570077 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:54.570059    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-dbbbq"
Oct 30 18:35:54.810032 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:54.810005    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7c3ae3e0-ddf1-414f-bb79-e4afc0527b26-original-pull-secret\") pod \"global-pull-secret-syncer-vhtmb\" (UID: \"7c3ae3e0-ddf1-414f-bb79-e4afc0527b26\") " pod="kube-system/global-pull-secret-syncer-vhtmb"
Oct 30 18:35:54.810121 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:54.810059    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/089fdd4f-daa0-4b3e-b81e-26a764a526bb-metrics-certs\") pod \"network-metrics-daemon-9p48q\" (UID: \"089fdd4f-daa0-4b3e-b81e-26a764a526bb\") " pod="openshift-multus/network-metrics-daemon-9p48q"
Oct 30 18:35:54.810175 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:54.810149    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 18:35:54.810241 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:54.810219    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c3ae3e0-ddf1-414f-bb79-e4afc0527b26-original-pull-secret podName:7c3ae3e0-ddf1-414f-bb79-e4afc0527b26 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:10.810200984 +0000 UTC m=+34.275139296 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7c3ae3e0-ddf1-414f-bb79-e4afc0527b26-original-pull-secret") pod "global-pull-secret-syncer-vhtmb" (UID: "7c3ae3e0-ddf1-414f-bb79-e4afc0527b26") : object "kube-system"/"original-pull-secret" not registered
Oct 30 18:35:54.810241 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:54.810149    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:35:54.810326 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:54.810267    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/089fdd4f-daa0-4b3e-b81e-26a764a526bb-metrics-certs podName:089fdd4f-daa0-4b3e-b81e-26a764a526bb nodeName:}" failed. No retries permitted until 2026-10-30 18:36:10.810259183 +0000 UTC m=+34.275197448 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/089fdd4f-daa0-4b3e-b81e-26a764a526bb-metrics-certs") pod "network-metrics-daemon-9p48q" (UID: "089fdd4f-daa0-4b3e-b81e-26a764a526bb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:35:54.910858 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:54.910665    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfrpn\" (UniqueName: \"kubernetes.io/projected/4b27cd9e-69b2-4480-a84f-5b463cec1a0e-kube-api-access-rfrpn\") pod \"network-check-target-mcc4d\" (UID: \"4b27cd9e-69b2-4480-a84f-5b463cec1a0e\") " pod="openshift-network-diagnostics/network-check-target-mcc4d"
Oct 30 18:35:54.911049 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:54.911032    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 18:35:54.911101 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:54.911057    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 18:35:54.911101 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:54.911070    2573 projected.go:196] Error preparing data for projected volume kube-api-access-rfrpn for pod openshift-network-diagnostics/network-check-target-mcc4d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:35:54.911194 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:54.911127    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4b27cd9e-69b2-4480-a84f-5b463cec1a0e-kube-api-access-rfrpn podName:4b27cd9e-69b2-4480-a84f-5b463cec1a0e nodeName:}" failed. No retries permitted until 2026-10-30 18:36:10.91110744 +0000 UTC m=+34.376045724 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-rfrpn" (UniqueName: "kubernetes.io/projected/4b27cd9e-69b2-4480-a84f-5b463cec1a0e-kube-api-access-rfrpn") pod "network-check-target-mcc4d" (UID: "4b27cd9e-69b2-4480-a84f-5b463cec1a0e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:35:55.038568 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:55.038546    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mcc4d"
Oct 30 18:35:55.038660 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:55.038587    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9p48q"
Oct 30 18:35:55.038722 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:55.038704    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mcc4d" podUID="4b27cd9e-69b2-4480-a84f-5b463cec1a0e"
Oct 30 18:35:55.038800 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:55.038777    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9p48q" podUID="089fdd4f-daa0-4b3e-b81e-26a764a526bb"
Oct 30 18:35:55.253490 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:55.253453    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p" event={"ID":"8bcb59a4-381f-49ba-b732-f5e4658707de","Type":"ContainerStarted","Data":"b7db0a957a0fc546b3c2affdc9404eba811c80c33ec1be7e2d3fad91100d05b0"}
Oct 30 18:35:55.253668 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:55.253555    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p" event={"ID":"8bcb59a4-381f-49ba-b732-f5e4658707de","Type":"ContainerStarted","Data":"158d7f1964be43a7e22198b7768d2f2ee6ad552a83f4d926cb36395070f2b60d"}
Oct 30 18:35:55.253668 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:55.253595    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p" event={"ID":"8bcb59a4-381f-49ba-b732-f5e4658707de","Type":"ContainerStarted","Data":"c2174b5f4672b3f9f4271fa333aeaa80b02f9f9d7de6cbd4466f789934c3c62c"}
Oct 30 18:35:55.253668 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:55.253623    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p" event={"ID":"8bcb59a4-381f-49ba-b732-f5e4658707de","Type":"ContainerStarted","Data":"1bfdf15597e8704d1466783718765829c6627853606dc191f9d5a27d6d2f2dff"}
Oct 30 18:35:55.253668 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:55.253650    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p" event={"ID":"8bcb59a4-381f-49ba-b732-f5e4658707de","Type":"ContainerStarted","Data":"115904ebf5a5dc90632b1479d61d2721addb735694e7a1f084f7ab268119a3d7"}
Oct 30 18:35:55.253825 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:55.253697    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p" event={"ID":"8bcb59a4-381f-49ba-b732-f5e4658707de","Type":"ContainerStarted","Data":"8c14125473d61e03ad13871029dec364dcab405bb0db40589ffcf1e2f2eeedc1"}
Oct 30 18:35:55.256134 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:55.256101    2573 generic.go:356] "Generic (PLEG): container finished" podID="0dc90efb-2709-4894-b457-6cadb8abd801" containerID="656a4d728f0f368cdd5628a8cbe661835d184fb4ce9b86b15e3943de370f600c" exitCode=0
Oct 30 18:35:55.256234 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:55.256184    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tjtmv" event={"ID":"0dc90efb-2709-4894-b457-6cadb8abd801","Type":"ContainerDied","Data":"656a4d728f0f368cdd5628a8cbe661835d184fb4ce9b86b15e3943de370f600c"}
Oct 30 18:35:56.037819 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:56.037730    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vhtmb"
Oct 30 18:35:56.038279 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:56.037834    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vhtmb" podUID="7c3ae3e0-ddf1-414f-bb79-e4afc0527b26"
Oct 30 18:35:56.261552 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:56.261516    2573 generic.go:356] "Generic (PLEG): container finished" podID="0dc90efb-2709-4894-b457-6cadb8abd801" containerID="8d7936225fddb7a6f691d96456c9eb14b84185ee83ddbe0b3bbc0a2e3e2c4017" exitCode=0
Oct 30 18:35:56.261722 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:56.261556    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tjtmv" event={"ID":"0dc90efb-2709-4894-b457-6cadb8abd801","Type":"ContainerDied","Data":"8d7936225fddb7a6f691d96456c9eb14b84185ee83ddbe0b3bbc0a2e3e2c4017"}
Oct 30 18:35:57.039264 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:57.039233    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mcc4d"
Oct 30 18:35:57.039738 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:57.039322    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mcc4d" podUID="4b27cd9e-69b2-4480-a84f-5b463cec1a0e"
Oct 30 18:35:57.039738 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:57.039373    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9p48q"
Oct 30 18:35:57.039738 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:57.039439    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9p48q" podUID="089fdd4f-daa0-4b3e-b81e-26a764a526bb"
Oct 30 18:35:57.267497 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:57.267461    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p" event={"ID":"8bcb59a4-381f-49ba-b732-f5e4658707de","Type":"ContainerStarted","Data":"4647796e03994fc57f469eca6d523de2e3f3830946c4ed7e9ccab20b1f341b4e"}
Oct 30 18:35:57.270904 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:57.270866    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tjtmv" event={"ID":"0dc90efb-2709-4894-b457-6cadb8abd801","Type":"ContainerStarted","Data":"2c6dff7568c22830c26d6e3fe1e329e4a64f7ea3c3c5bf4419578be127524330"}
Oct 30 18:35:57.289231 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:57.289191    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-tjtmv" podStartSLOduration=7.850062217 podStartE2EDuration="19.289179254s" totalImagesPullingTime="11.439117037s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:35:38 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:35:57.288973033 +0000 UTC m=+20.753911319" watchObservedRunningTime="2026-10-30 18:35:57.289179254 +0000 UTC m=+20.754117541"
Oct 30 18:35:58.037789 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:58.037760    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vhtmb"
Oct 30 18:35:58.037962 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:58.037853    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vhtmb" podUID="7c3ae3e0-ddf1-414f-bb79-e4afc0527b26"
Oct 30 18:35:59.038035 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:59.038003    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mcc4d"
Oct 30 18:35:59.038611 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:35:59.038003    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9p48q"
Oct 30 18:35:59.038611 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:59.038112    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mcc4d" podUID="4b27cd9e-69b2-4480-a84f-5b463cec1a0e"
Oct 30 18:35:59.038611 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:35:59.038191    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9p48q" podUID="089fdd4f-daa0-4b3e-b81e-26a764a526bb"
Oct 30 18:36:00.038279 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:00.038104    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vhtmb"
Oct 30 18:36:00.038812 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:36:00.038345    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vhtmb" podUID="7c3ae3e0-ddf1-414f-bb79-e4afc0527b26"
Oct 30 18:36:00.281024 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:00.280994    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p" event={"ID":"8bcb59a4-381f-49ba-b732-f5e4658707de","Type":"ContainerStarted","Data":"b31cc92be8aacb0c1c1b321fb44dc4efe3b30ffc673dd931cbaf47cb20f83d92"}
Oct 30 18:36:00.281219 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:00.281200    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:36:00.298474 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:00.298395    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:36:00.302634 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:00.302600    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p" podStartSLOduration=6.917643217 podStartE2EDuration="22.302572033s" totalImagesPullingTime="15.384928816s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:35:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:00.300979919 +0000 UTC m=+23.765918206" watchObservedRunningTime="2026-10-30 18:36:00.302572033 +0000 UTC m=+23.767510319"
Oct 30 18:36:01.038503 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:01.038469    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mcc4d"
Oct 30 18:36:01.039018 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:01.038629    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9p48q"
Oct 30 18:36:01.039018 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:36:01.038628    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mcc4d" podUID="4b27cd9e-69b2-4480-a84f-5b463cec1a0e"
Oct 30 18:36:01.039018 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:36:01.038747    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9p48q" podUID="089fdd4f-daa0-4b3e-b81e-26a764a526bb"
Oct 30 18:36:01.144765 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:01.144672    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vhtmb"]
Oct 30 18:36:01.144889 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:01.144823    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vhtmb"
Oct 30 18:36:01.144938 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:36:01.144922    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vhtmb" podUID="7c3ae3e0-ddf1-414f-bb79-e4afc0527b26"
Oct 30 18:36:01.148071 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:01.148043    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mcc4d"]
Oct 30 18:36:01.148210 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:01.148099    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9p48q"]
Oct 30 18:36:01.283115 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:01.283082    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mcc4d"
Oct 30 18:36:01.283115 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:01.283108    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9p48q"
Oct 30 18:36:01.283353 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:01.283174    2573 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 18:36:01.283353 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:36:01.283195    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mcc4d" podUID="4b27cd9e-69b2-4480-a84f-5b463cec1a0e"
Oct 30 18:36:01.283627 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:01.283610    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:36:01.283742 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:36:01.283627    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9p48q" podUID="089fdd4f-daa0-4b3e-b81e-26a764a526bb"
Oct 30 18:36:01.299451 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:01.299418    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:36:02.285088 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:02.285062    2573 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 18:36:02.431193 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:02.431164    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:36:03.038529 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:03.038500    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mcc4d"
Oct 30 18:36:03.038529 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:03.038515    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9p48q"
Oct 30 18:36:03.038796 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:36:03.038590    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mcc4d" podUID="4b27cd9e-69b2-4480-a84f-5b463cec1a0e"
Oct 30 18:36:03.038796 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:36:03.038663    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9p48q" podUID="089fdd4f-daa0-4b3e-b81e-26a764a526bb"
Oct 30 18:36:03.038796 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:03.038716    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vhtmb"
Oct 30 18:36:03.038796 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:36:03.038779    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vhtmb" podUID="7c3ae3e0-ddf1-414f-bb79-e4afc0527b26"
Oct 30 18:36:04.299697 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:04.299642    2573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p" podUID="8bcb59a4-381f-49ba-b732-f5e4658707de" containerName="ovnkube-controller" probeResult="failure" output=""
Oct 30 18:36:05.037901 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:05.037871    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mcc4d"
Oct 30 18:36:05.038134 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:05.037871    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vhtmb"
Oct 30 18:36:05.038134 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:36:05.037980    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mcc4d" podUID="4b27cd9e-69b2-4480-a84f-5b463cec1a0e"
Oct 30 18:36:05.038134 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:36:05.038048    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vhtmb" podUID="7c3ae3e0-ddf1-414f-bb79-e4afc0527b26"
Oct 30 18:36:05.038134 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:05.037872    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9p48q"
Oct 30 18:36:05.038260 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:36:05.038162    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9p48q" podUID="089fdd4f-daa0-4b3e-b81e-26a764a526bb"
Oct 30 18:36:07.039419 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:07.039239    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vhtmb"
Oct 30 18:36:07.039810 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:07.039305    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9p48q"
Oct 30 18:36:07.039810 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:36:07.039490    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vhtmb" podUID="7c3ae3e0-ddf1-414f-bb79-e4afc0527b26"
Oct 30 18:36:07.039810 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:07.039331    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mcc4d"
Oct 30 18:36:07.039810 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:36:07.039571    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9p48q" podUID="089fdd4f-daa0-4b3e-b81e-26a764a526bb"
Oct 30 18:36:07.039810 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:36:07.039651    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mcc4d" podUID="4b27cd9e-69b2-4480-a84f-5b463cec1a0e"
Oct 30 18:36:07.350325 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:07.350257    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-135.us-east-2.compute.internal" event="NodeReady"
Oct 30 18:36:07.350459 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:07.350372    2573 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 18:36:07.395657 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:07.395627    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-lcst7"]
Oct 30 18:36:07.398852 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:07.398833    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-dqxm8"]
Oct 30 18:36:07.399017 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:07.398998    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lcst7"
Oct 30 18:36:07.401658 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:07.401643    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dqxm8"
Oct 30 18:36:07.403499 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:07.403481    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 18:36:07.403585 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:07.403496    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qs4k8\""
Oct 30 18:36:07.403585 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:07.403489    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 18:36:07.407219 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:07.407200    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 18:36:07.409096 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:07.409076    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lcst7"]
Oct 30 18:36:07.411625 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:07.411605    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dqxm8"]
Oct 30 18:36:07.497576 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:07.497550    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f32f2498-2fc8-4bc6-b368-b53d156755c5-cert\") pod \"ingress-canary-dqxm8\" (UID: \"f32f2498-2fc8-4bc6-b368-b53d156755c5\") " pod="openshift-ingress-canary/ingress-canary-dqxm8"
Oct 30 18:36:07.497576 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:07.497582    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a-metrics-tls\") pod \"dns-default-lcst7\" (UID: \"5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a\") " pod="openshift-dns/dns-default-lcst7"
Oct 30 18:36:07.497789 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:07.497600    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a-config-volume\") pod \"dns-default-lcst7\" (UID: \"5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a\") " pod="openshift-dns/dns-default-lcst7"
Oct 30 18:36:07.497789 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:07.497616    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a-tmp-dir\") pod \"dns-default-lcst7\" (UID: \"5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a\") " pod="openshift-dns/dns-default-lcst7"
Oct 30 18:36:07.497789 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:07.497739    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7jl8\" (UniqueName: \"kubernetes.io/projected/f32f2498-2fc8-4bc6-b368-b53d156755c5-kube-api-access-t7jl8\") pod \"ingress-canary-dqxm8\" (UID: \"f32f2498-2fc8-4bc6-b368-b53d156755c5\") " pod="openshift-ingress-canary/ingress-canary-dqxm8"
Oct 30 18:36:07.497789 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:07.497763    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vl5cp\" (UniqueName: \"kubernetes.io/projected/5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a-kube-api-access-vl5cp\") pod \"dns-default-lcst7\" (UID: \"5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a\") " pod="openshift-dns/dns-default-lcst7"
Oct 30 18:36:07.503401 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:07.503380    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 18:36:07.503507 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:07.503492    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-5rxk7\""
Oct 30 18:36:07.503543 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:07.503529    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 18:36:07.598958 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:07.598923    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f32f2498-2fc8-4bc6-b368-b53d156755c5-cert\") pod \"ingress-canary-dqxm8\" (UID: \"f32f2498-2fc8-4bc6-b368-b53d156755c5\") " pod="openshift-ingress-canary/ingress-canary-dqxm8"
Oct 30 18:36:07.599116 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:07.598968    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a-metrics-tls\") pod \"dns-default-lcst7\" (UID: \"5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a\") " pod="openshift-dns/dns-default-lcst7"
Oct 30 18:36:07.599116 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:07.598992    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a-config-volume\") pod \"dns-default-lcst7\" (UID: \"5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a\") " pod="openshift-dns/dns-default-lcst7"
Oct 30 18:36:07.599116 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:07.599010    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a-tmp-dir\") pod \"dns-default-lcst7\" (UID: \"5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a\") " pod="openshift-dns/dns-default-lcst7"
Oct 30 18:36:07.599116 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:07.599042    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7jl8\" (UniqueName: \"kubernetes.io/projected/f32f2498-2fc8-4bc6-b368-b53d156755c5-kube-api-access-t7jl8\") pod \"ingress-canary-dqxm8\" (UID: \"f32f2498-2fc8-4bc6-b368-b53d156755c5\") " pod="openshift-ingress-canary/ingress-canary-dqxm8"
Oct 30 18:36:07.599116 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:07.599077    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl5cp\" (UniqueName: \"kubernetes.io/projected/5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a-kube-api-access-vl5cp\") pod \"dns-default-lcst7\" (UID: \"5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a\") " pod="openshift-dns/dns-default-lcst7"
Oct 30 18:36:07.599116 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:36:07.599090    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 18:36:07.599375 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:36:07.599155    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a-metrics-tls podName:5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a nodeName:}" failed. No retries permitted until 2026-10-30 18:36:08.099136674 +0000 UTC m=+31.564074939 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a-metrics-tls") pod "dns-default-lcst7" (UID: "5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a") : secret "dns-default-metrics-tls" not found
Oct 30 18:36:07.599463 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:07.599444    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a-tmp-dir\") pod \"dns-default-lcst7\" (UID: \"5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a\") " pod="openshift-dns/dns-default-lcst7"
Oct 30 18:36:07.599661 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:07.599639    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a-config-volume\") pod \"dns-default-lcst7\" (UID: \"5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a\") " pod="openshift-dns/dns-default-lcst7"
Oct 30 18:36:07.603283 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:07.603230    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f32f2498-2fc8-4bc6-b368-b53d156755c5-cert\") pod \"ingress-canary-dqxm8\" (UID: \"f32f2498-2fc8-4bc6-b368-b53d156755c5\") " pod="openshift-ingress-canary/ingress-canary-dqxm8"
Oct 30 18:36:07.610865 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:07.610836    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7jl8\" (UniqueName: \"kubernetes.io/projected/f32f2498-2fc8-4bc6-b368-b53d156755c5-kube-api-access-t7jl8\") pod \"ingress-canary-dqxm8\" (UID: \"f32f2498-2fc8-4bc6-b368-b53d156755c5\") " pod="openshift-ingress-canary/ingress-canary-dqxm8"
Oct 30 18:36:07.610998 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:07.610982    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vl5cp\" (UniqueName: \"kubernetes.io/projected/5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a-kube-api-access-vl5cp\") pod \"dns-default-lcst7\" (UID: \"5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a\") " pod="openshift-dns/dns-default-lcst7"
Oct 30 18:36:07.718382 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:07.718345    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dqxm8"
Oct 30 18:36:08.006267 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:08.006239    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dqxm8"]
Oct 30 18:36:08.009919 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:36:08.009884    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf32f2498_2fc8_4bc6_b368_b53d156755c5.slice/crio-693fe2dc18ff556cbd492a9ff21e105b068a26391601a45d32d96e1c677809ea WatchSource:0}: Error finding container 693fe2dc18ff556cbd492a9ff21e105b068a26391601a45d32d96e1c677809ea: Status 404 returned error can't find the container with id 693fe2dc18ff556cbd492a9ff21e105b068a26391601a45d32d96e1c677809ea
Oct 30 18:36:08.103229 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:08.103196    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a-metrics-tls\") pod \"dns-default-lcst7\" (UID: \"5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a\") " pod="openshift-dns/dns-default-lcst7"
Oct 30 18:36:08.103620 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:36:08.103355    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 18:36:08.103620 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:36:08.103415    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a-metrics-tls podName:5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a nodeName:}" failed. No retries permitted until 2026-10-30 18:36:09.103399892 +0000 UTC m=+32.568338157 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a-metrics-tls") pod "dns-default-lcst7" (UID: "5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a") : secret "dns-default-metrics-tls" not found
Oct 30 18:36:08.298530 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:08.298450    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dqxm8" event={"ID":"f32f2498-2fc8-4bc6-b368-b53d156755c5","Type":"ContainerStarted","Data":"693fe2dc18ff556cbd492a9ff21e105b068a26391601a45d32d96e1c677809ea"}
Oct 30 18:36:09.038047 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:09.038007    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9p48q"
Oct 30 18:36:09.038221 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:09.038065    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vhtmb"
Oct 30 18:36:09.038221 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:09.038073    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mcc4d"
Oct 30 18:36:09.042340 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:09.042303    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 18:36:09.042468 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:09.042347    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 18:36:09.043402 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:09.043356    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 18:36:09.043532 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:09.043440    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-phzbf\""
Oct 30 18:36:09.043532 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:09.043448    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vgk4h\""
Oct 30 18:36:09.043532 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:09.043453    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 18:36:09.043532 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:09.043502    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-dnnzq\""
Oct 30 18:36:09.110942 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:09.110914    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a-metrics-tls\") pod \"dns-default-lcst7\" (UID: \"5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a\") " pod="openshift-dns/dns-default-lcst7"
Oct 30 18:36:09.111326 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:36:09.111036    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 18:36:09.111326 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:36:09.111092    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a-metrics-tls podName:5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a nodeName:}" failed. No retries permitted until 2026-10-30 18:36:11.111079537 +0000 UTC m=+34.576017802 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a-metrics-tls") pod "dns-default-lcst7" (UID: "5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a") : secret "dns-default-metrics-tls" not found
Oct 30 18:36:10.826072 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:10.826043    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7c3ae3e0-ddf1-414f-bb79-e4afc0527b26-original-pull-secret\") pod \"global-pull-secret-syncer-vhtmb\" (UID: \"7c3ae3e0-ddf1-414f-bb79-e4afc0527b26\") " pod="kube-system/global-pull-secret-syncer-vhtmb"
Oct 30 18:36:10.826510 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:10.826088    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/089fdd4f-daa0-4b3e-b81e-26a764a526bb-metrics-certs\") pod \"network-metrics-daemon-9p48q\" (UID: \"089fdd4f-daa0-4b3e-b81e-26a764a526bb\") " pod="openshift-multus/network-metrics-daemon-9p48q"
Oct 30 18:36:10.826510 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:36:10.826247    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 18:36:10.826510 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:36:10.826318    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/089fdd4f-daa0-4b3e-b81e-26a764a526bb-metrics-certs podName:089fdd4f-daa0-4b3e-b81e-26a764a526bb nodeName:}" failed. No retries permitted until 2026-10-30 18:36:42.826299125 +0000 UTC m=+66.291237406 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/089fdd4f-daa0-4b3e-b81e-26a764a526bb-metrics-certs") pod "network-metrics-daemon-9p48q" (UID: "089fdd4f-daa0-4b3e-b81e-26a764a526bb") : secret "metrics-daemon-secret" not found
Oct 30 18:36:10.828471 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:10.828447    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7c3ae3e0-ddf1-414f-bb79-e4afc0527b26-original-pull-secret\") pod \"global-pull-secret-syncer-vhtmb\" (UID: \"7c3ae3e0-ddf1-414f-bb79-e4afc0527b26\") " pod="kube-system/global-pull-secret-syncer-vhtmb"
Oct 30 18:36:10.927203 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:10.927161    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfrpn\" (UniqueName: \"kubernetes.io/projected/4b27cd9e-69b2-4480-a84f-5b463cec1a0e-kube-api-access-rfrpn\") pod \"network-check-target-mcc4d\" (UID: \"4b27cd9e-69b2-4480-a84f-5b463cec1a0e\") " pod="openshift-network-diagnostics/network-check-target-mcc4d"
Oct 30 18:36:10.929777 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:10.929754    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfrpn\" (UniqueName: \"kubernetes.io/projected/4b27cd9e-69b2-4480-a84f-5b463cec1a0e-kube-api-access-rfrpn\") pod \"network-check-target-mcc4d\" (UID: \"4b27cd9e-69b2-4480-a84f-5b463cec1a0e\") " pod="openshift-network-diagnostics/network-check-target-mcc4d"
Oct 30 18:36:10.930435 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:10.930415    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vhtmb"
Oct 30 18:36:11.129226 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:11.129192    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a-metrics-tls\") pod \"dns-default-lcst7\" (UID: \"5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a\") " pod="openshift-dns/dns-default-lcst7"
Oct 30 18:36:11.129377 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:36:11.129322    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 18:36:11.129413 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:36:11.129385    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a-metrics-tls podName:5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a nodeName:}" failed. No retries permitted until 2026-10-30 18:36:15.12936976 +0000 UTC m=+38.594308029 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a-metrics-tls") pod "dns-default-lcst7" (UID: "5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a") : secret "dns-default-metrics-tls" not found
Oct 30 18:36:11.207993 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:11.207965    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mcc4d"
Oct 30 18:36:11.231162 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:11.231095    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vhtmb"]
Oct 30 18:36:11.234214 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:36:11.234193    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c3ae3e0_ddf1_414f_bb79_e4afc0527b26.slice/crio-02c85c4602ad9496b82c8c4f0e457cf814e30c624820667e7374a0f7b35e3ded WatchSource:0}: Error finding container 02c85c4602ad9496b82c8c4f0e457cf814e30c624820667e7374a0f7b35e3ded: Status 404 returned error can't find the container with id 02c85c4602ad9496b82c8c4f0e457cf814e30c624820667e7374a0f7b35e3ded
Oct 30 18:36:11.306657 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:11.306620    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dqxm8" event={"ID":"f32f2498-2fc8-4bc6-b368-b53d156755c5","Type":"ContainerStarted","Data":"2a9a92f58abf9c8e594e8eaa018d3c4f1bf604f58c64917e751debfd151e8549"}
Oct 30 18:36:11.307839 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:11.307819    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vhtmb" event={"ID":"7c3ae3e0-ddf1-414f-bb79-e4afc0527b26","Type":"ContainerStarted","Data":"02c85c4602ad9496b82c8c4f0e457cf814e30c624820667e7374a0f7b35e3ded"}
Oct 30 18:36:11.323027 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:11.322988    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-dqxm8" podStartSLOduration=1.628303123 podStartE2EDuration="4.322973126s" totalImagesPullingTime="2.694670003s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:36:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:11.322241976 +0000 UTC m=+34.787180263" watchObservedRunningTime="2026-10-30 18:36:11.322973126 +0000 UTC m=+34.787911416"
Oct 30 18:36:11.498800 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:11.498726    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mcc4d"]
Oct 30 18:36:11.501943 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:36:11.501912    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b27cd9e_69b2_4480_a84f_5b463cec1a0e.slice/crio-2dd07382ff8e72ba2fc1d71510c152d5aa38c29fab5a4d75da82e3b8b9a0770a WatchSource:0}: Error finding container 2dd07382ff8e72ba2fc1d71510c152d5aa38c29fab5a4d75da82e3b8b9a0770a: Status 404 returned error can't find the container with id 2dd07382ff8e72ba2fc1d71510c152d5aa38c29fab5a4d75da82e3b8b9a0770a
Oct 30 18:36:12.312966 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:12.312929    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mcc4d" event={"ID":"4b27cd9e-69b2-4480-a84f-5b463cec1a0e","Type":"ContainerStarted","Data":"2dd07382ff8e72ba2fc1d71510c152d5aa38c29fab5a4d75da82e3b8b9a0770a"}
Oct 30 18:36:13.702172 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:13.700588    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-mhxzk"]
Oct 30 18:36:13.723508 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:13.723472    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-mhxzk"]
Oct 30 18:36:13.723700 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:13.723587    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-mhxzk"
Oct 30 18:36:13.733934 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:13.733904    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 18:36:13.734867 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:13.734814    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 18:36:13.734867 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:13.734832    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-zsxst\""
Oct 30 18:36:13.802910 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:13.802875    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-q4kks-q2hfp"]
Oct 30 18:36:13.823717 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:13.823306    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-4d49l"]
Oct 30 18:36:13.823717 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:13.823510    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-q4kks-q2hfp"
Oct 30 18:36:13.833773 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:13.833752    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 18:36:13.833876 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:13.833811    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 18:36:13.833876 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:13.833813    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 18:36:13.835576 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:13.835415    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-6sn6l\""
Oct 30 18:36:13.837210 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:13.837186    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-q4kks-q2hfp"]
Oct 30 18:36:13.837314 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:13.837297    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4d49l"
Oct 30 18:36:13.838769 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:13.838749    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 18:36:13.840366 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:13.840346    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4d49l"]
Oct 30 18:36:13.845931 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:13.845820    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5t65n\" (UniqueName: \"kubernetes.io/projected/97f514c8-0d31-4103-81c5-45886ff831f0-kube-api-access-5t65n\") pod \"migrator-66df5d496b-mhxzk\" (UID: \"97f514c8-0d31-4103-81c5-45886ff831f0\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-mhxzk"
Oct 30 18:36:13.933783 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:13.933746    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 18:36:13.934139 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:13.934042    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 18:36:13.934350 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:13.934240    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9fb5r\""
Oct 30 18:36:13.946721 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:13.946671    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9ce43bfa-1e19-4fb5-90ff-42751cdd1a50-archives-path\") pod \"periodic-gathering-q4kks-q2hfp\" (UID: \"9ce43bfa-1e19-4fb5-90ff-42751cdd1a50\") " pod="openshift-insights/periodic-gathering-q4kks-q2hfp"
Oct 30 18:36:13.946889 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:13.946773    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ce43bfa-1e19-4fb5-90ff-42751cdd1a50-service-ca-bundle\") pod \"periodic-gathering-q4kks-q2hfp\" (UID: \"9ce43bfa-1e19-4fb5-90ff-42751cdd1a50\") " pod="openshift-insights/periodic-gathering-q4kks-q2hfp"
Oct 30 18:36:13.946889 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:13.946807    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ce43bfa-1e19-4fb5-90ff-42751cdd1a50-trusted-ca-bundle\") pod \"periodic-gathering-q4kks-q2hfp\" (UID: \"9ce43bfa-1e19-4fb5-90ff-42751cdd1a50\") " pod="openshift-insights/periodic-gathering-q4kks-q2hfp"
Oct 30 18:36:13.946889 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:13.946826    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1315feec-3a59-4cae-ad7e-8ada8c5e4788-crio-socket\") pod \"insights-runtime-extractor-4d49l\" (UID: \"1315feec-3a59-4cae-ad7e-8ada8c5e4788\") " pod="openshift-insights/insights-runtime-extractor-4d49l"
Oct 30 18:36:13.946889 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:13.946853    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1315feec-3a59-4cae-ad7e-8ada8c5e4788-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4d49l\" (UID: \"1315feec-3a59-4cae-ad7e-8ada8c5e4788\") " pod="openshift-insights/insights-runtime-extractor-4d49l"
Oct 30 18:36:13.946889 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:13.946888    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1315feec-3a59-4cae-ad7e-8ada8c5e4788-data-volume\") pod \"insights-runtime-extractor-4d49l\" (UID: \"1315feec-3a59-4cae-ad7e-8ada8c5e4788\") " pod="openshift-insights/insights-runtime-extractor-4d49l"
Oct 30 18:36:13.947132 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:13.946922    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5t65n\" (UniqueName: \"kubernetes.io/projected/97f514c8-0d31-4103-81c5-45886ff831f0-kube-api-access-5t65n\") pod \"migrator-66df5d496b-mhxzk\" (UID: \"97f514c8-0d31-4103-81c5-45886ff831f0\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-mhxzk"
Oct 30 18:36:13.947132 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:13.946947    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-862ff\" (UniqueName: \"kubernetes.io/projected/9ce43bfa-1e19-4fb5-90ff-42751cdd1a50-kube-api-access-862ff\") pod \"periodic-gathering-q4kks-q2hfp\" (UID: \"9ce43bfa-1e19-4fb5-90ff-42751cdd1a50\") " pod="openshift-insights/periodic-gathering-q4kks-q2hfp"
Oct 30 18:36:13.947132 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:13.946964    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1315feec-3a59-4cae-ad7e-8ada8c5e4788-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4d49l\" (UID: \"1315feec-3a59-4cae-ad7e-8ada8c5e4788\") " pod="openshift-insights/insights-runtime-extractor-4d49l"
Oct 30 18:36:13.947132 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:13.946989    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tp8kg\" (UniqueName: \"kubernetes.io/projected/1315feec-3a59-4cae-ad7e-8ada8c5e4788-kube-api-access-tp8kg\") pod \"insights-runtime-extractor-4d49l\" (UID: \"1315feec-3a59-4cae-ad7e-8ada8c5e4788\") " pod="openshift-insights/insights-runtime-extractor-4d49l"
Oct 30 18:36:13.965243 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:13.965177    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t65n\" (UniqueName: \"kubernetes.io/projected/97f514c8-0d31-4103-81c5-45886ff831f0-kube-api-access-5t65n\") pod \"migrator-66df5d496b-mhxzk\" (UID: \"97f514c8-0d31-4103-81c5-45886ff831f0\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-mhxzk"
Oct 30 18:36:14.044883 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:14.044853    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-mhxzk"
Oct 30 18:36:14.048124 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:14.048097    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-862ff\" (UniqueName: \"kubernetes.io/projected/9ce43bfa-1e19-4fb5-90ff-42751cdd1a50-kube-api-access-862ff\") pod \"periodic-gathering-q4kks-q2hfp\" (UID: \"9ce43bfa-1e19-4fb5-90ff-42751cdd1a50\") " pod="openshift-insights/periodic-gathering-q4kks-q2hfp"
Oct 30 18:36:14.048124 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:14.048129    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1315feec-3a59-4cae-ad7e-8ada8c5e4788-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4d49l\" (UID: \"1315feec-3a59-4cae-ad7e-8ada8c5e4788\") " pod="openshift-insights/insights-runtime-extractor-4d49l"
Oct 30 18:36:14.048324 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:14.048146    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp8kg\" (UniqueName: \"kubernetes.io/projected/1315feec-3a59-4cae-ad7e-8ada8c5e4788-kube-api-access-tp8kg\") pod \"insights-runtime-extractor-4d49l\" (UID: \"1315feec-3a59-4cae-ad7e-8ada8c5e4788\") " pod="openshift-insights/insights-runtime-extractor-4d49l"
Oct 30 18:36:14.048324 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:14.048206    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9ce43bfa-1e19-4fb5-90ff-42751cdd1a50-archives-path\") pod \"periodic-gathering-q4kks-q2hfp\" (UID: \"9ce43bfa-1e19-4fb5-90ff-42751cdd1a50\") " pod="openshift-insights/periodic-gathering-q4kks-q2hfp"
Oct 30 18:36:14.048324 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:14.048231    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ce43bfa-1e19-4fb5-90ff-42751cdd1a50-service-ca-bundle\") pod \"periodic-gathering-q4kks-q2hfp\" (UID: \"9ce43bfa-1e19-4fb5-90ff-42751cdd1a50\") " pod="openshift-insights/periodic-gathering-q4kks-q2hfp"
Oct 30 18:36:14.048324 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:14.048255    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ce43bfa-1e19-4fb5-90ff-42751cdd1a50-trusted-ca-bundle\") pod \"periodic-gathering-q4kks-q2hfp\" (UID: \"9ce43bfa-1e19-4fb5-90ff-42751cdd1a50\") " pod="openshift-insights/periodic-gathering-q4kks-q2hfp"
Oct 30 18:36:14.048324 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:36:14.048266    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 18:36:14.048324 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:14.048276    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1315feec-3a59-4cae-ad7e-8ada8c5e4788-crio-socket\") pod \"insights-runtime-extractor-4d49l\" (UID: \"1315feec-3a59-4cae-ad7e-8ada8c5e4788\") " pod="openshift-insights/insights-runtime-extractor-4d49l"
Oct 30 18:36:14.048324 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:36:14.048326    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1315feec-3a59-4cae-ad7e-8ada8c5e4788-insights-runtime-extractor-tls podName:1315feec-3a59-4cae-ad7e-8ada8c5e4788 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:14.548306423 +0000 UTC m=+38.013244702 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1315feec-3a59-4cae-ad7e-8ada8c5e4788-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4d49l" (UID: "1315feec-3a59-4cae-ad7e-8ada8c5e4788") : secret "insights-runtime-extractor-tls" not found
Oct 30 18:36:14.048701 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:14.048555    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1315feec-3a59-4cae-ad7e-8ada8c5e4788-crio-socket\") pod \"insights-runtime-extractor-4d49l\" (UID: \"1315feec-3a59-4cae-ad7e-8ada8c5e4788\") " pod="openshift-insights/insights-runtime-extractor-4d49l"
Oct 30 18:36:14.048701 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:14.048558    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1315feec-3a59-4cae-ad7e-8ada8c5e4788-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4d49l\" (UID: \"1315feec-3a59-4cae-ad7e-8ada8c5e4788\") " pod="openshift-insights/insights-runtime-extractor-4d49l"
Oct 30 18:36:14.048701 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:36:14.048621    2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 18:36:14.048701 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:14.048629    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1315feec-3a59-4cae-ad7e-8ada8c5e4788-data-volume\") pod \"insights-runtime-extractor-4d49l\" (UID: \"1315feec-3a59-4cae-ad7e-8ada8c5e4788\") " pod="openshift-insights/insights-runtime-extractor-4d49l"
Oct 30 18:36:14.048701 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:14.048638    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9ce43bfa-1e19-4fb5-90ff-42751cdd1a50-archives-path\") pod \"periodic-gathering-q4kks-q2hfp\" (UID: \"9ce43bfa-1e19-4fb5-90ff-42751cdd1a50\") " pod="openshift-insights/periodic-gathering-q4kks-q2hfp"
Oct 30 18:36:14.048937 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:36:14.048712    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1315feec-3a59-4cae-ad7e-8ada8c5e4788-kube-rbac-proxy-cm podName:1315feec-3a59-4cae-ad7e-8ada8c5e4788 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:14.548670133 +0000 UTC m=+38.013608417 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1315feec-3a59-4cae-ad7e-8ada8c5e4788-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4d49l" (UID: "1315feec-3a59-4cae-ad7e-8ada8c5e4788") : configmap "kube-rbac-proxy" not found
Oct 30 18:36:14.049129 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:14.049104    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1315feec-3a59-4cae-ad7e-8ada8c5e4788-data-volume\") pod \"insights-runtime-extractor-4d49l\" (UID: \"1315feec-3a59-4cae-ad7e-8ada8c5e4788\") " pod="openshift-insights/insights-runtime-extractor-4d49l"
Oct 30 18:36:14.049363 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:14.049335    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ce43bfa-1e19-4fb5-90ff-42751cdd1a50-service-ca-bundle\") pod \"periodic-gathering-q4kks-q2hfp\" (UID: \"9ce43bfa-1e19-4fb5-90ff-42751cdd1a50\") " pod="openshift-insights/periodic-gathering-q4kks-q2hfp"
Oct 30 18:36:14.049588 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:14.049566    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ce43bfa-1e19-4fb5-90ff-42751cdd1a50-trusted-ca-bundle\") pod \"periodic-gathering-q4kks-q2hfp\" (UID: \"9ce43bfa-1e19-4fb5-90ff-42751cdd1a50\") " pod="openshift-insights/periodic-gathering-q4kks-q2hfp"
Oct 30 18:36:14.058094 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:14.058069    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-862ff\" (UniqueName: \"kubernetes.io/projected/9ce43bfa-1e19-4fb5-90ff-42751cdd1a50-kube-api-access-862ff\") pod \"periodic-gathering-q4kks-q2hfp\" (UID: \"9ce43bfa-1e19-4fb5-90ff-42751cdd1a50\") " pod="openshift-insights/periodic-gathering-q4kks-q2hfp"
Oct 30 18:36:14.058367 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:14.058350    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp8kg\" (UniqueName: \"kubernetes.io/projected/1315feec-3a59-4cae-ad7e-8ada8c5e4788-kube-api-access-tp8kg\") pod \"insights-runtime-extractor-4d49l\" (UID: \"1315feec-3a59-4cae-ad7e-8ada8c5e4788\") " pod="openshift-insights/insights-runtime-extractor-4d49l"
Oct 30 18:36:14.152059 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:14.152027    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-q4kks-q2hfp"
Oct 30 18:36:14.209326 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:14.209291    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-5qghx"]
Oct 30 18:36:14.228229 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:14.228142    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-5qghx"]
Oct 30 18:36:14.228326 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:14.228272    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-5qghx"
Oct 30 18:36:14.233205 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:14.233090    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-rqvfh\""
Oct 30 18:36:14.233205 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:14.233114    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 30 18:36:14.233205 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:14.233151    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 30 18:36:14.233205 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:14.233183    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 30 18:36:14.233408 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:14.233254    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 30 18:36:14.339276 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:14.339248    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-mhxzk"]
Oct 30 18:36:14.351363 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:14.351317    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/25255277-4f3b-418c-9e85-2a82e7a13107-signing-cabundle\") pod \"service-ca-5fffc97bdf-5qghx\" (UID: \"25255277-4f3b-418c-9e85-2a82e7a13107\") " pod="openshift-service-ca/service-ca-5fffc97bdf-5qghx"
Oct 30 18:36:14.351468 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:14.351449    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/25255277-4f3b-418c-9e85-2a82e7a13107-signing-key\") pod \"service-ca-5fffc97bdf-5qghx\" (UID: \"25255277-4f3b-418c-9e85-2a82e7a13107\") " pod="openshift-service-ca/service-ca-5fffc97bdf-5qghx"
Oct 30 18:36:14.351520 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:14.351487    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7rn7\" (UniqueName: \"kubernetes.io/projected/25255277-4f3b-418c-9e85-2a82e7a13107-kube-api-access-j7rn7\") pod \"service-ca-5fffc97bdf-5qghx\" (UID: \"25255277-4f3b-418c-9e85-2a82e7a13107\") " pod="openshift-service-ca/service-ca-5fffc97bdf-5qghx"
Oct 30 18:36:14.448113 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:14.448074    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-q4kks-q2hfp"]
Oct 30 18:36:14.451414 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:36:14.451378    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ce43bfa_1e19_4fb5_90ff_42751cdd1a50.slice/crio-1fe07a3980476efe57f370a3f8ffd99d740abe566fde8bf60f8a712e032faac7 WatchSource:0}: Error finding container 1fe07a3980476efe57f370a3f8ffd99d740abe566fde8bf60f8a712e032faac7: Status 404 returned error can't find the container with id 1fe07a3980476efe57f370a3f8ffd99d740abe566fde8bf60f8a712e032faac7
Oct 30 18:36:14.452297 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:14.452273    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/25255277-4f3b-418c-9e85-2a82e7a13107-signing-cabundle\") pod \"service-ca-5fffc97bdf-5qghx\" (UID: \"25255277-4f3b-418c-9e85-2a82e7a13107\") " pod="openshift-service-ca/service-ca-5fffc97bdf-5qghx"
Oct 30 18:36:14.452370 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:14.452326    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/25255277-4f3b-418c-9e85-2a82e7a13107-signing-key\") pod \"service-ca-5fffc97bdf-5qghx\" (UID: \"25255277-4f3b-418c-9e85-2a82e7a13107\") " pod="openshift-service-ca/service-ca-5fffc97bdf-5qghx"
Oct 30 18:36:14.452370 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:14.452357    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7rn7\" (UniqueName: \"kubernetes.io/projected/25255277-4f3b-418c-9e85-2a82e7a13107-kube-api-access-j7rn7\") pod \"service-ca-5fffc97bdf-5qghx\" (UID: \"25255277-4f3b-418c-9e85-2a82e7a13107\") " pod="openshift-service-ca/service-ca-5fffc97bdf-5qghx"
Oct 30 18:36:14.453459 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:14.453436    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/25255277-4f3b-418c-9e85-2a82e7a13107-signing-cabundle\") pod \"service-ca-5fffc97bdf-5qghx\" (UID: \"25255277-4f3b-418c-9e85-2a82e7a13107\") " pod="openshift-service-ca/service-ca-5fffc97bdf-5qghx"
Oct 30 18:36:14.456255 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:14.456237    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/25255277-4f3b-418c-9e85-2a82e7a13107-signing-key\") pod \"service-ca-5fffc97bdf-5qghx\" (UID: \"25255277-4f3b-418c-9e85-2a82e7a13107\") " pod="openshift-service-ca/service-ca-5fffc97bdf-5qghx"
Oct 30 18:36:14.461953 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:14.461917    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7rn7\" (UniqueName: \"kubernetes.io/projected/25255277-4f3b-418c-9e85-2a82e7a13107-kube-api-access-j7rn7\") pod \"service-ca-5fffc97bdf-5qghx\" (UID: \"25255277-4f3b-418c-9e85-2a82e7a13107\") " pod="openshift-service-ca/service-ca-5fffc97bdf-5qghx"
Oct 30 18:36:14.553568 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:14.553480    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1315feec-3a59-4cae-ad7e-8ada8c5e4788-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4d49l\" (UID: \"1315feec-3a59-4cae-ad7e-8ada8c5e4788\") " pod="openshift-insights/insights-runtime-extractor-4d49l"
Oct 30 18:36:14.553568 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:14.553559    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1315feec-3a59-4cae-ad7e-8ada8c5e4788-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4d49l\" (UID: \"1315feec-3a59-4cae-ad7e-8ada8c5e4788\") " pod="openshift-insights/insights-runtime-extractor-4d49l"
Oct 30 18:36:14.553814 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:36:14.553709    2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 18:36:14.553814 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:36:14.553764    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1315feec-3a59-4cae-ad7e-8ada8c5e4788-kube-rbac-proxy-cm podName:1315feec-3a59-4cae-ad7e-8ada8c5e4788 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:15.553748941 +0000 UTC m=+39.018687206 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1315feec-3a59-4cae-ad7e-8ada8c5e4788-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4d49l" (UID: "1315feec-3a59-4cae-ad7e-8ada8c5e4788") : configmap "kube-rbac-proxy" not found
Oct 30 18:36:14.554183 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:36:14.554164    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 18:36:14.554263 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:36:14.554214    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1315feec-3a59-4cae-ad7e-8ada8c5e4788-insights-runtime-extractor-tls podName:1315feec-3a59-4cae-ad7e-8ada8c5e4788 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:15.554198499 +0000 UTC m=+39.019136765 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1315feec-3a59-4cae-ad7e-8ada8c5e4788-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4d49l" (UID: "1315feec-3a59-4cae-ad7e-8ada8c5e4788") : secret "insights-runtime-extractor-tls" not found
Oct 30 18:36:14.554323 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:14.554292    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-5qghx"
Oct 30 18:36:14.868049 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:14.867870    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-5qghx"]
Oct 30 18:36:14.870294 ip-10-0-137-135 kubenswrapper[2573]: W1030 18:36:14.870257    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25255277_4f3b_418c_9e85_2a82e7a13107.slice/crio-da579d74802736a0baaca917640290659ec92fddd742408ef7cd256a6d51bc58 WatchSource:0}: Error finding container da579d74802736a0baaca917640290659ec92fddd742408ef7cd256a6d51bc58: Status 404 returned error can't find the container with id da579d74802736a0baaca917640290659ec92fddd742408ef7cd256a6d51bc58
Oct 30 18:36:15.159494 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:15.159412    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a-metrics-tls\") pod \"dns-default-lcst7\" (UID: \"5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a\") " pod="openshift-dns/dns-default-lcst7"
Oct 30 18:36:15.159647 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:36:15.159582    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 18:36:15.159647 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:36:15.159638    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a-metrics-tls podName:5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a nodeName:}" failed. No retries permitted until 2026-10-30 18:36:23.159623304 +0000 UTC m=+46.624561582 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a-metrics-tls") pod "dns-default-lcst7" (UID: "5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a") : secret "dns-default-metrics-tls" not found
Oct 30 18:36:15.324099 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:15.323365    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mcc4d" event={"ID":"4b27cd9e-69b2-4480-a84f-5b463cec1a0e","Type":"ContainerStarted","Data":"02074b12061294ff540a617d69e8791934692e53870ea64279bfeca7bbb6c791"}
Oct 30 18:36:15.324099 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:15.324038    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-mcc4d"
Oct 30 18:36:15.325545 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:15.325511    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-q4kks-q2hfp" event={"ID":"9ce43bfa-1e19-4fb5-90ff-42751cdd1a50","Type":"ContainerStarted","Data":"1fe07a3980476efe57f370a3f8ffd99d740abe566fde8bf60f8a712e032faac7"}
Oct 30 18:36:15.327780 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:15.327125    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-5qghx" event={"ID":"25255277-4f3b-418c-9e85-2a82e7a13107","Type":"ContainerStarted","Data":"da579d74802736a0baaca917640290659ec92fddd742408ef7cd256a6d51bc58"}
Oct 30 18:36:15.329333 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:15.329311    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-mhxzk" event={"ID":"97f514c8-0d31-4103-81c5-45886ff831f0","Type":"ContainerStarted","Data":"3cdeb882d3478e934ee1baa73a787abbf040e0aa8fc251268cfec63a14e2d499"}
Oct 30 18:36:15.341787 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:15.341744    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-mcc4d" podStartSLOduration=33.674660177 podStartE2EDuration="37.341732862s" totalImagesPullingTime="3.667072685s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:35:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:15.340620507 +0000 UTC m=+38.805558794" watchObservedRunningTime="2026-10-30 18:36:15.341732862 +0000 UTC m=+38.806671146"
Oct 30 18:36:15.562618 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:15.562527    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1315feec-3a59-4cae-ad7e-8ada8c5e4788-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4d49l\" (UID: \"1315feec-3a59-4cae-ad7e-8ada8c5e4788\") " pod="openshift-insights/insights-runtime-extractor-4d49l"
Oct 30 18:36:15.562618 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:15.562607    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1315feec-3a59-4cae-ad7e-8ada8c5e4788-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4d49l\" (UID: \"1315feec-3a59-4cae-ad7e-8ada8c5e4788\") " pod="openshift-insights/insights-runtime-extractor-4d49l"
Oct 30 18:36:15.562854 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:36:15.562792    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 18:36:15.562905 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:36:15.562862    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1315feec-3a59-4cae-ad7e-8ada8c5e4788-insights-runtime-extractor-tls podName:1315feec-3a59-4cae-ad7e-8ada8c5e4788 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:17.562841781 +0000 UTC m=+41.027780070 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1315feec-3a59-4cae-ad7e-8ada8c5e4788-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4d49l" (UID: "1315feec-3a59-4cae-ad7e-8ada8c5e4788") : secret "insights-runtime-extractor-tls" not found
Oct 30 18:36:15.562995 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:36:15.562917    2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 18:36:15.562995 ip-10-0-137-135 kubenswrapper[2573]: E1030 18:36:15.562951    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1315feec-3a59-4cae-ad7e-8ada8c5e4788-kube-rbac-proxy-cm podName:1315feec-3a59-4cae-ad7e-8ada8c5e4788 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:17.562941023 +0000 UTC m=+41.027879302 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1315feec-3a59-4cae-ad7e-8ada8c5e4788-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4d49l" (UID: "1315feec-3a59-4cae-ad7e-8ada8c5e4788") : configmap "kube-rbac-proxy" not found
Oct 30 18:36:16.335207 ip-10-0-137-135 kubenswrapper[2573]: I1030 18:36:16.335135    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-mhxzk" event={"ID":"97f514c8-0d31-4103-81c5-45886ff831f0","Type":"ContainerStarted","Data":"6f610959d030b83437a25cd85f72fc920607b4c340936a8d1d2a88782a036c8c"}
Oct 30 18:36:17.281380 ip-10-0-137-135 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 18:36:17.288399 ip-10-0-137-135 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 18:36:17.288581 ip-10-0-137-135 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 18:36:17.288629 ip-10-0-137-135 systemd[1]: kubelet.service: Consumed 2.394s CPU time.
Oct 30 18:36:17.303310 ip-10-0-137-135 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 18:36:17.428465 ip-10-0-137-135 kubenswrapper[4525]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 18:36:17.428465 ip-10-0-137-135 kubenswrapper[4525]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 18:36:17.428465 ip-10-0-137-135 kubenswrapper[4525]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 18:36:17.428465 ip-10-0-137-135 kubenswrapper[4525]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 18:36:17.431468 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431362    4525 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:36:17.431468 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431467    4525 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:36:17.431468 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431473    4525 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:36:17.431672 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431477    4525 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:36:17.431672 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431482    4525 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:36:17.431672 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431485    4525 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:36:17.431672 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431489    4525 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:36:17.431672 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431494    4525 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:36:17.431672 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431498    4525 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:36:17.431672 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431501    4525 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:36:17.431672 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431505    4525 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:36:17.431672 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431509    4525 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:36:17.431672 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431513    4525 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:36:17.431672 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431519    4525 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:36:17.431672 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431523    4525 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:36:17.431672 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431527    4525 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:36:17.431672 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431531    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:36:17.431672 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431542    4525 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:36:17.431672 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431547    4525 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:36:17.431672 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431552    4525 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:36:17.431672 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431556    4525 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:36:17.431672 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431563    4525 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:36:17.432480 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431569    4525 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:36:17.432480 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431574    4525 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:36:17.432480 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431580    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:36:17.432480 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431584    4525 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:36:17.432480 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431588    4525 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:36:17.432480 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431593    4525 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:36:17.432480 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431598    4525 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:36:17.432480 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431602    4525 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:36:17.432480 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431607    4525 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:36:17.432480 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431611    4525 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:36:17.432480 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431615    4525 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:36:17.432480 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431620    4525 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:36:17.432480 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431624    4525 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:36:17.432480 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431629    4525 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:36:17.432480 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431633    4525 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:36:17.432480 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431637    4525 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:36:17.432480 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431642    4525 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:36:17.432480 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431646    4525 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:36:17.432480 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431651    4525 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:36:17.432480 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431656    4525 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:36:17.433352 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431661    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:36:17.433352 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431665    4525 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:36:17.433352 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431670    4525 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:36:17.433352 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431674    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:36:17.433352 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431696    4525 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:36:17.433352 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431701    4525 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:36:17.433352 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431705    4525 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:36:17.433352 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431710    4525 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:36:17.433352 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431715    4525 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:36:17.433352 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431721    4525 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:36:17.433352 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431726    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:36:17.433352 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431730    4525 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:36:17.433352 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431735    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:36:17.433352 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431739    4525 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:36:17.433352 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431743    4525 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:36:17.433352 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431747    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:36:17.433352 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431752    4525 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:36:17.433352 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431756    4525 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:36:17.433352 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431761    4525 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:36:17.434188 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431764    4525 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:36:17.434188 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431769    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:36:17.434188 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431773    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:36:17.434188 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431777    4525 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:36:17.434188 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431781    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:36:17.434188 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431785    4525 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:36:17.434188 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431791    4525 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:36:17.434188 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431795    4525 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:36:17.434188 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431800    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:36:17.434188 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431804    4525 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:36:17.434188 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431808    4525 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:36:17.434188 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431812    4525 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:36:17.434188 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431816    4525 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:36:17.434188 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431821    4525 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:36:17.434188 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431827    4525 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:36:17.434188 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431834    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:36:17.434188 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431839    4525 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:36:17.434188 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431845    4525 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:36:17.434188 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431850    4525 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:36:17.435024 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431855    4525 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:36:17.435024 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431859    4525 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:36:17.435024 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431865    4525 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:36:17.435024 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431872    4525 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:36:17.435024 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431876    4525 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:36:17.435024 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431881    4525 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:36:17.435024 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431885    4525 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:36:17.435024 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431889    4525 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:36:17.435024 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431893    4525 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:36:17.435024 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431898    4525 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:36:17.435024 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431902    4525 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:36:17.435024 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431906    4525 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:36:17.435024 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431911    4525 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:36:17.435024 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431915    4525 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:36:17.435024 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431920    4525 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:36:17.435024 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431924    4525 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:36:17.435024 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431928    4525 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:36:17.435024 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431932    4525 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:36:17.435024 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431937    4525 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:36:17.435024 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431942    4525 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:36:17.435919 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431947    4525 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:36:17.435919 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431951    4525 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:36:17.435919 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431955    4525 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:36:17.435919 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431960    4525 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:36:17.435919 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431964    4525 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:36:17.435919 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.431970    4525 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:36:17.435919 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433255    4525 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:36:17.435919 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433265    4525 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:36:17.435919 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433271    4525 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:36:17.435919 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433276    4525 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:36:17.435919 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433281    4525 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:36:17.435919 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433285    4525 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:36:17.435919 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433290    4525 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:36:17.435919 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433294    4525 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:36:17.435919 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433298    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:36:17.435919 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433306    4525 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:36:17.435919 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433311    4525 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:36:17.435919 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433316    4525 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:36:17.435919 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433320    4525 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:36:17.435919 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433324    4525 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:36:17.436805 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433328    4525 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:36:17.436805 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433333    4525 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:36:17.436805 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433337    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:36:17.436805 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433341    4525 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:36:17.436805 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433345    4525 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:36:17.436805 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433349    4525 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:36:17.436805 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433353    4525 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:36:17.436805 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433357    4525 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:36:17.436805 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433361    4525 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:36:17.436805 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433365    4525 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:36:17.436805 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433369    4525 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:36:17.436805 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433374    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:36:17.436805 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433378    4525 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:36:17.436805 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433383    4525 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:36:17.436805 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433387    4525 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:36:17.436805 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433392    4525 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:36:17.436805 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433396    4525 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:36:17.436805 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433401    4525 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:36:17.436805 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433405    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:36:17.437605 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433409    4525 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:36:17.437605 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433414    4525 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:36:17.437605 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433418    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:36:17.437605 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433422    4525 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:36:17.437605 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433426    4525 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:36:17.437605 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433431    4525 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:36:17.437605 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433435    4525 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:36:17.437605 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433439    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:36:17.437605 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433443    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:36:17.437605 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433450    4525 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:36:17.437605 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433454    4525 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:36:17.437605 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433458    4525 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:36:17.437605 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433462    4525 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:36:17.437605 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433466    4525 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:36:17.437605 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433471    4525 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:36:17.437605 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433475    4525 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:36:17.437605 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433479    4525 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:36:17.437605 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433483    4525 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:36:17.437605 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433487    4525 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:36:17.437605 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433492    4525 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:36:17.438385 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433496    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:36:17.438385 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433500    4525 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:36:17.438385 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433505    4525 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:36:17.438385 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433509    4525 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:36:17.438385 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433513    4525 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:36:17.438385 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433518    4525 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:36:17.438385 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433522    4525 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:36:17.438385 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433526    4525 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:36:17.438385 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433531    4525 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:36:17.438385 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433535    4525 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:36:17.438385 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433539    4525 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:36:17.438385 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433543    4525 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:36:17.438385 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433547    4525 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:36:17.438385 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433552    4525 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:36:17.438385 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433556    4525 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:36:17.438385 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433561    4525 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:36:17.438385 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433565    4525 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:36:17.438385 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433570    4525 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:36:17.438385 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433574    4525 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:36:17.439084 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433578    4525 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:36:17.439084 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433582    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:36:17.439084 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433587    4525 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:36:17.439084 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433591    4525 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:36:17.439084 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433596    4525 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:36:17.439084 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433601    4525 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:36:17.439084 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433605    4525 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:36:17.439084 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433609    4525 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:36:17.439084 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433613    4525 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:36:17.439084 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433617    4525 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:36:17.439084 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433622    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:36:17.439084 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433629    4525 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:36:17.439084 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433636    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:36:17.439084 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433642    4525 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:36:17.439084 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433647    4525 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:36:17.439084 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433652    4525 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:36:17.439084 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433656    4525 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:36:17.439084 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433661    4525 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:36:17.439084 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433665    4525 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:36:17.439733 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433670    4525 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:36:17.439733 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433674    4525 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:36:17.439733 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433698    4525 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:36:17.439733 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433702    4525 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:36:17.439733 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433706    4525 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:36:17.439733 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433711    4525 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:36:17.439733 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433715    4525 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:36:17.439733 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433721    4525 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:36:17.439733 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433727    4525 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:36:17.439733 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433732    4525 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:36:17.439733 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433736    4525 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:36:17.439733 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433741    4525 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:36:17.439733 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433746    4525 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:36:17.439733 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433750    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:36:17.439733 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.433755    4525 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:36:17.439733 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.434584    4525 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:36:17.439733 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.434594    4525 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:36:17.439733 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.434598    4525 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:36:17.439733 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.434603    4525 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:36:17.440405 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.434607    4525 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:36:17.440405 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.434611    4525 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:36:17.440405 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.434616    4525 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:36:17.440405 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.434620    4525 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:36:17.440405 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.434624    4525 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:36:17.440405 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.434629    4525 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:36:17.440405 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.434634    4525 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:36:17.440405 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.434639    4525 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:36:17.440405 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.434643    4525 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:36:17.440405 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.434647    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:36:17.440405 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.434652    4525 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:36:17.440405 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.434656    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:36:17.440405 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.434660    4525 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:36:17.440405 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.434665    4525 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:36:17.440405 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.434669    4525 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:36:17.440405 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.434674    4525 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:36:17.440405 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.434697    4525 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:36:17.440405 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.434702    4525 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:36:17.440405 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.434707    4525 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:36:17.440405 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.434711    4525 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:36:17.441162 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.434715    4525 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:36:17.441162 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.434719    4525 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:36:17.441162 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.434723    4525 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:36:17.441162 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.434728    4525 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:36:17.441162 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.434732    4525 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:36:17.441162 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.434736    4525 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:36:17.441162 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.434740    4525 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:36:17.441162 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.434744    4525 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:36:17.441162 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.434749    4525 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:36:17.441162 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.434752    4525 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:36:17.441162 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.434758    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:36:17.441162 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.434762    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:36:17.441162 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.434766    4525 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:36:17.441162 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.434770    4525 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:36:17.441162 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.434774    4525 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:36:17.441162 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.434779    4525 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:36:17.441162 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.434784    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:36:17.441162 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.434789    4525 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:36:17.441162 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.434793    4525 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:36:17.441162 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.434798    4525 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:36:17.441813 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.434802    4525 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:36:17.441813 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.434806    4525 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:36:17.441813 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.434813    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:36:17.441813 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.434817    4525 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:36:17.441813 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.434822    4525 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:36:17.441813 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.434827    4525 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:36:17.441813 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.434831    4525 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:36:17.441813 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.434835    4525 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:36:17.441813 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.434840    4525 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:36:17.441813 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.434844    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:36:17.441813 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.434848    4525 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:36:17.441813 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.434852    4525 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:36:17.441813 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.434856    4525 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:36:17.441813 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.434863    4525 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:36:17.441813 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.434871    4525 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:36:17.441813 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.434876    4525 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:36:17.441813 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.434881    4525 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:36:17.441813 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.434885    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:36:17.441813 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.434889    4525 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:36:17.442608 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.434894    4525 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:36:17.442608 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.434899    4525 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:36:17.442608 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.434905    4525 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:36:17.442608 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.434909    4525 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:36:17.442608 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.434915    4525 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:36:17.442608 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.434920    4525 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:36:17.442608 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.434924    4525 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:36:17.442608 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.434928    4525 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:36:17.442608 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.434933    4525 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:36:17.442608 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.434937    4525 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:36:17.442608 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.434941    4525 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:36:17.442608 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.434945    4525 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:36:17.442608 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.434949    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:36:17.442608 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.434956    4525 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:36:17.442608 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.434963    4525 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:36:17.442608 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.434969    4525 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:36:17.442608 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.434974    4525 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:36:17.442608 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.434978    4525 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:36:17.442608 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.434982    4525 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:36:17.443329 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.434986    4525 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:36:17.443329 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.434991    4525 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:36:17.443329 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.434995    4525 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:36:17.443329 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.435000    4525 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:36:17.443329 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.435005    4525 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:36:17.443329 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.435009    4525 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:36:17.443329 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.435013    4525 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:36:17.443329 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.435018    4525 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:36:17.443329 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.435022    4525 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:36:17.443329 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.435026    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:36:17.443329 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.435030    4525 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:36:17.443329 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.435034    4525 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:36:17.443329 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.435039    4525 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:36:17.443329 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.435044    4525 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:36:17.443329 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.435048    4525 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:36:17.443329 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.435052    4525 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:36:17.443329 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.435057    4525 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:36:17.443329 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.435061    4525 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:36:17.443329 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.435066    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:36:17.443329 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.435071    4525 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:36:17.444252 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.435075    4525 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:36:17.444252 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.435079    4525 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:36:17.444252 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.435083    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:36:17.444252 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.435087    4525 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:36:17.444252 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435220    4525 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 18:36:17.444252 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435231    4525 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 18:36:17.444252 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435240    4525 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 18:36:17.444252 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435247    4525 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 18:36:17.444252 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435257    4525 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 18:36:17.444252 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435262    4525 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 18:36:17.444252 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435269    4525 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 18:36:17.444252 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435276    4525 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 18:36:17.444252 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435281    4525 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 18:36:17.444252 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435286    4525 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 18:36:17.444252 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435292    4525 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 18:36:17.444252 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435298    4525 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 18:36:17.444252 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435303    4525 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 18:36:17.444252 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435308    4525 flags.go:64] FLAG: --cgroup-root=""
Oct 30 18:36:17.444252 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435313    4525 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 18:36:17.444252 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435318    4525 flags.go:64] FLAG: --client-ca-file=""
Oct 30 18:36:17.444252 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435322    4525 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 18:36:17.444252 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435327    4525 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 18:36:17.444252 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435334    4525 flags.go:64] FLAG: --cluster-domain=""
Oct 30 18:36:17.445025 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435339    4525 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 18:36:17.445025 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435344    4525 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 18:36:17.445025 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435350    4525 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 18:36:17.445025 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435356    4525 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 18:36:17.445025 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435362    4525 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 18:36:17.445025 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435367    4525 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 18:36:17.445025 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435372    4525 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 18:36:17.445025 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435377    4525 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 18:36:17.445025 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435383    4525 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 18:36:17.445025 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435389    4525 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 18:36:17.445025 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435393    4525 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 18:36:17.445025 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435399    4525 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 18:36:17.445025 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435403    4525 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 18:36:17.445025 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435410    4525 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 18:36:17.445025 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435415    4525 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 18:36:17.445025 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435421    4525 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 18:36:17.445025 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435426    4525 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 18:36:17.445025 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435434    4525 flags.go:64] FLAG: --enable-server="true"
Oct 30 18:36:17.445025 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435440    4525 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 18:36:17.445025 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435447    4525 flags.go:64] FLAG: --event-burst="100"
Oct 30 18:36:17.445025 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435452    4525 flags.go:64] FLAG: --event-qps="50"
Oct 30 18:36:17.445025 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435457    4525 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 18:36:17.445025 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435462    4525 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 18:36:17.445025 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435467    4525 flags.go:64] FLAG: --eviction-hard=""
Oct 30 18:36:17.445905 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435473    4525 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 18:36:17.445905 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435478    4525 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 18:36:17.445905 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435483    4525 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 18:36:17.445905 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435488    4525 flags.go:64] FLAG: --eviction-soft=""
Oct 30 18:36:17.445905 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435493    4525 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 18:36:17.445905 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435497    4525 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 18:36:17.445905 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435502    4525 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 18:36:17.445905 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435507    4525 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 18:36:17.445905 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435512    4525 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 18:36:17.445905 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435517    4525 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 18:36:17.445905 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435522    4525 flags.go:64] FLAG: --feature-gates=""
Oct 30 18:36:17.445905 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435529    4525 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 18:36:17.445905 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435534    4525 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 18:36:17.445905 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435539    4525 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 18:36:17.445905 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435544    4525 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 18:36:17.445905 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435549    4525 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 18:36:17.445905 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435554    4525 flags.go:64] FLAG: --help="false"
Oct 30 18:36:17.445905 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435560    4525 flags.go:64] FLAG: --hostname-override="ip-10-0-137-135.us-east-2.compute.internal"
Oct 30 18:36:17.445905 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435566    4525 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 18:36:17.445905 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435571    4525 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 18:36:17.445905 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435575    4525 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 18:36:17.445905 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435581    4525 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 18:36:17.445905 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435586    4525 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 18:36:17.446698 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435591    4525 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 18:36:17.446698 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435596    4525 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 18:36:17.446698 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435602    4525 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 18:36:17.446698 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435607    4525 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 18:36:17.446698 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435612    4525 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 18:36:17.446698 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435617    4525 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 18:36:17.446698 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435622    4525 flags.go:64] FLAG: --kube-reserved=""
Oct 30 18:36:17.446698 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435626    4525 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 18:36:17.446698 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435631    4525 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 18:36:17.446698 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435636    4525 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 18:36:17.446698 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435641    4525 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 18:36:17.446698 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435646    4525 flags.go:64] FLAG: --lock-file=""
Oct 30 18:36:17.446698 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435650    4525 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 18:36:17.446698 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435655    4525 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 18:36:17.446698 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435660    4525 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 18:36:17.446698 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435669    4525 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 18:36:17.446698 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435673    4525 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 18:36:17.446698 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435697    4525 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 18:36:17.446698 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435702    4525 flags.go:64] FLAG: --logging-format="text"
Oct 30 18:36:17.446698 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435707    4525 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 18:36:17.446698 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435712    4525 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 18:36:17.446698 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435717    4525 flags.go:64] FLAG: --manifest-url=""
Oct 30 18:36:17.446698 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435722    4525 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 18:36:17.446698 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435729    4525 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 18:36:17.446698 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435734    4525 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 18:36:17.447371 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435740    4525 flags.go:64] FLAG: --max-pods="110"
Oct 30 18:36:17.447371 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435745    4525 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 18:36:17.447371 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435752    4525 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 18:36:17.447371 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435756    4525 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 18:36:17.447371 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435761    4525 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 18:36:17.447371 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435766    4525 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 18:36:17.447371 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435771    4525 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 18:36:17.447371 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435776    4525 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 18:36:17.447371 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435787    4525 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 18:36:17.447371 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435793    4525 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 18:36:17.447371 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435799    4525 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 18:36:17.447371 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435804    4525 flags.go:64] FLAG: --pod-cidr=""
Oct 30 18:36:17.447371 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435808    4525 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 18:36:17.447371 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435813    4525 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 18:36:17.447371 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435818    4525 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 18:36:17.447371 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435822    4525 flags.go:64] FLAG: --port="10250"
Oct 30 18:36:17.447371 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435827    4525 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 18:36:17.447371 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435832    4525 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0e74aff2ec6d34e9d"
Oct 30 18:36:17.447371 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435837    4525 flags.go:64] FLAG: --qos-reserved=""
Oct 30 18:36:17.447371 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435842    4525 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 18:36:17.447371 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435847    4525 flags.go:64] FLAG: --register-node="true"
Oct 30 18:36:17.447371 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435852    4525 flags.go:64] FLAG: --register-with-taints=""
Oct 30 18:36:17.447371 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435858    4525 flags.go:64] FLAG: --registry-burst="10"
Oct 30 18:36:17.447371 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435863    4525 flags.go:64] FLAG: --registry-qps="5"
Oct 30 18:36:17.447371 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435868    4525 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 18:36:17.447371 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435873    4525 flags.go:64] FLAG: --reserved-memory=""
Oct 30 18:36:17.448329 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435879    4525 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 18:36:17.448329 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435884    4525 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 18:36:17.448329 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435890    4525 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 18:36:17.448329 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435895    4525 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 18:36:17.448329 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435899    4525 flags.go:64] FLAG: --runonce="false"
Oct 30 18:36:17.448329 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435904    4525 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 18:36:17.448329 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435910    4525 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 18:36:17.448329 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435915    4525 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 18:36:17.448329 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435920    4525 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 18:36:17.448329 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435925    4525 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 18:36:17.448329 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435930    4525 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 18:36:17.448329 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435935    4525 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 18:36:17.448329 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435940    4525 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 18:36:17.448329 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435944    4525 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 18:36:17.448329 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435949    4525 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 18:36:17.448329 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435956    4525 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 18:36:17.448329 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435961    4525 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 18:36:17.448329 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435966    4525 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 18:36:17.448329 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435971    4525 flags.go:64] FLAG: --system-cgroups=""
Oct 30 18:36:17.448329 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435976    4525 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 18:36:17.448329 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435984    4525 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 18:36:17.448329 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.435989    4525 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 18:36:17.448329 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.436001    4525 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 18:36:17.448329 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.436009    4525 flags.go:64] FLAG: --tls-min-version=""
Oct 30 18:36:17.448329 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.436014    4525 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 18:36:17.449234 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.436019    4525 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 18:36:17.449234 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.436024    4525 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 18:36:17.449234 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.436029    4525 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 18:36:17.449234 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.436034    4525 flags.go:64] FLAG: --v="2"
Oct 30 18:36:17.449234 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.436041    4525 flags.go:64] FLAG: --version="false"
Oct 30 18:36:17.449234 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.436048    4525 flags.go:64] FLAG: --vmodule=""
Oct 30 18:36:17.449234 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.436054    4525 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 18:36:17.449234 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.436059    4525 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 18:36:17.449234 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436290    4525 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:36:17.449234 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436298    4525 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:36:17.449234 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436303    4525 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:36:17.449234 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436308    4525 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:36:17.449234 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436313    4525 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:36:17.449234 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436317    4525 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:36:17.449234 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436322    4525 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:36:17.449234 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436326    4525 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:36:17.449234 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436330    4525 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:36:17.449234 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436335    4525 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:36:17.449234 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436340    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:36:17.449234 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436345    4525 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:36:17.449234 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436349    4525 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:36:17.450772 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436353    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:36:17.450772 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436360    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:36:17.450772 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436366    4525 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:36:17.450772 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436370    4525 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:36:17.450772 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436375    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:36:17.450772 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436379    4525 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:36:17.450772 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436383    4525 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:36:17.450772 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436388    4525 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:36:17.450772 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436392    4525 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:36:17.450772 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436397    4525 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:36:17.450772 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436402    4525 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:36:17.450772 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436406    4525 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:36:17.450772 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436411    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:36:17.450772 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436415    4525 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:36:17.450772 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436420    4525 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:36:17.450772 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436424    4525 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:36:17.450772 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436428    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:36:17.450772 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436432    4525 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:36:17.450772 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436436    4525 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:36:17.450772 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436441    4525 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:36:17.451419 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436445    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:36:17.451419 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436449    4525 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:36:17.451419 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436454    4525 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:36:17.451419 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436458    4525 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:36:17.451419 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436462    4525 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:36:17.451419 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436466    4525 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:36:17.451419 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436471    4525 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:36:17.451419 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436475    4525 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:36:17.451419 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436479    4525 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:36:17.451419 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436485    4525 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:36:17.451419 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436489    4525 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:36:17.451419 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436493    4525 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:36:17.451419 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436498    4525 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:36:17.451419 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436506    4525 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:36:17.451419 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436513    4525 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:36:17.451419 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436519    4525 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:36:17.451419 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436524    4525 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:36:17.451419 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436528    4525 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:36:17.451419 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436533    4525 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:36:17.451961 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436537    4525 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:36:17.451961 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436542    4525 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:36:17.451961 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436546    4525 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:36:17.451961 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436550    4525 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:36:17.451961 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436555    4525 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:36:17.451961 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436559    4525 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:36:17.451961 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436563    4525 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:36:17.451961 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436567    4525 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:36:17.451961 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436572    4525 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:36:17.451961 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436576    4525 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:36:17.451961 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436581    4525 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:36:17.451961 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436585    4525 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:36:17.451961 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436589    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:36:17.451961 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436594    4525 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:36:17.451961 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436598    4525 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:36:17.451961 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436603    4525 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:36:17.451961 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436607    4525 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:36:17.451961 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436611    4525 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:36:17.451961 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436615    4525 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:36:17.452486 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436620    4525 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:36:17.452486 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436624    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:36:17.452486 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436628    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:36:17.452486 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436633    4525 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:36:17.452486 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436638    4525 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:36:17.452486 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436642    4525 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:36:17.452486 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436646    4525 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:36:17.452486 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436652    4525 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:36:17.452486 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436657    4525 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:36:17.452486 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436661    4525 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:36:17.452486 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436668    4525 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:36:17.452486 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436672    4525 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:36:17.452486 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436694    4525 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:36:17.452486 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436699    4525 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:36:17.452486 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436703    4525 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:36:17.452486 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436708    4525 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:36:17.452486 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436712    4525 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:36:17.452486 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436716    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:36:17.452486 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436720    4525 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:36:17.452486 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436725    4525 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:36:17.453022 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436729    4525 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:36:17.453022 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436733    4525 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:36:17.453022 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436738    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:36:17.453022 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436742    4525 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:36:17.453022 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436747    4525 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:36:17.453022 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436751    4525 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:36:17.453022 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436755    4525 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:36:17.453022 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436760    4525 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:36:17.453022 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436764    4525 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:36:17.453022 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436769    4525 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:36:17.453022 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436773    4525 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:36:17.453022 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436780    4525 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:36:17.453022 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436786    4525 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:36:17.453022 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436791    4525 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:36:17.453022 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436796    4525 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:36:17.453022 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.436839    4525 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 18:36:17.453703 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436895    4525 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:36:17.453703 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436906    4525 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:36:17.453703 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436911    4525 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:36:17.453703 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436916    4525 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:36:17.453703 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436923    4525 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:36:17.453703 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436928    4525 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:36:17.453703 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436934    4525 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:36:17.453703 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436939    4525 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:36:17.453703 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436943    4525 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:36:17.453703 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436947    4525 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:36:17.453703 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436952    4525 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:36:17.453703 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436956    4525 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:36:17.453703 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436960    4525 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:36:17.453703 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436965    4525 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:36:17.453703 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436969    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:36:17.453703 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436974    4525 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:36:17.453703 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436978    4525 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:36:17.453703 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436982    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:36:17.453703 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436988    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:36:17.454483 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436992    4525 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:36:17.454483 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.436996    4525 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:36:17.454483 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.437001    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:36:17.454483 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.437005    4525 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:36:17.454483 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.437009    4525 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:36:17.454483 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.437014    4525 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:36:17.454483 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.437018    4525 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:36:17.454483 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.437022    4525 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:36:17.454483 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.437027    4525 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:36:17.454483 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.437031    4525 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:36:17.454483 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.437036    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:36:17.454483 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.437040    4525 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:36:17.454483 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.437044    4525 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:36:17.454483 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.437048    4525 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:36:17.454483 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.437053    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:36:17.454483 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.437063    4525 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:36:17.454483 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.437068    4525 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:36:17.454483 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.437074    4525 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:36:17.454483 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.437078    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:36:17.454483 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.437084    4525 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:36:17.455332 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.437088    4525 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:36:17.455332 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.437093    4525 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:36:17.455332 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.437097    4525 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:36:17.455332 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.437101    4525 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:36:17.455332 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.437105    4525 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:36:17.455332 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.437110    4525 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:36:17.455332 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.437114    4525 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:36:17.455332 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.437119    4525 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:36:17.455332 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.437123    4525 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:36:17.455332 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.437127    4525 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:36:17.455332 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.437131    4525 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:36:17.455332 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.437138    4525 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:36:17.455332 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.437146    4525 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:36:17.455332 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.437151    4525 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:36:17.455332 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.437156    4525 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:36:17.455332 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.437160    4525 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:36:17.455332 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.437165    4525 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:36:17.455332 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.437169    4525 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:36:17.455332 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.437173    4525 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:36:17.456026 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.437177    4525 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:36:17.456026 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.437182    4525 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:36:17.456026 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.437186    4525 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:36:17.456026 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.437190    4525 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:36:17.456026 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.437195    4525 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:36:17.456026 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.437199    4525 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:36:17.456026 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.437203    4525 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:36:17.456026 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.437207    4525 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:36:17.456026 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.437212    4525 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:36:17.456026 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.437217    4525 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:36:17.456026 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.437223    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:36:17.456026 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.437227    4525 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:36:17.456026 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.437233    4525 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:36:17.456026 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.437237    4525 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:36:17.456026 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.437241    4525 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:36:17.456026 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.437246    4525 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:36:17.456026 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.437250    4525 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:36:17.456026 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.437254    4525 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:36:17.456026 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.437258    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:36:17.456548 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.437262    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:36:17.456548 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.437267    4525 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:36:17.456548 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.437271    4525 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:36:17.456548 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.437276    4525 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:36:17.456548 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.437280    4525 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:36:17.456548 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.437284    4525 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:36:17.456548 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.437288    4525 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:36:17.456548 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.437293    4525 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:36:17.456548 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.437297    4525 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:36:17.456548 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.437301    4525 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:36:17.456548 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.437306    4525 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:36:17.456548 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.437310    4525 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:36:17.456548 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.437314    4525 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:36:17.456548 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.437319    4525 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:36:17.456548 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.437323    4525 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:36:17.456548 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.437327    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:36:17.456548 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.437331    4525 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:36:17.456548 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.437335    4525 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:36:17.456548 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.437340    4525 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:36:17.456548 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.437344    4525 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:36:17.457348 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.437348    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:36:17.457348 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.437353    4525 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:36:17.457348 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.437357    4525 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:36:17.457348 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.437366    4525 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:36:17.457348 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.437370    4525 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:36:17.457348 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.437376    4525 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:36:17.457348 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.437380    4525 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:36:17.457348 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.437386    4525 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:36:17.457348 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.437391    4525 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:36:17.457348 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.442383    4525 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 18:36:17.457348 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.442399    4525 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 18:36:17.457348 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.442419    4525 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 18:36:17.457348 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.442427    4525 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 18:36:17.457348 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442502    4525 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:36:17.457348 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442508    4525 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:36:17.457348 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442511    4525 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:36:17.457348 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442514    4525 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:36:17.457348 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442517    4525 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:36:17.457348 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442521    4525 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:36:17.458114 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442523    4525 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:36:17.458114 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442526    4525 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:36:17.458114 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442530    4525 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:36:17.458114 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442535    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:36:17.458114 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442539    4525 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:36:17.458114 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442544    4525 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:36:17.458114 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442549    4525 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:36:17.458114 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442554    4525 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:36:17.458114 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442559    4525 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:36:17.458114 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442563    4525 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:36:17.458114 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442567    4525 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:36:17.458114 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442572    4525 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:36:17.458114 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442576    4525 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:36:17.458114 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442585    4525 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:36:17.458114 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442589    4525 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:36:17.458114 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442596    4525 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:36:17.458114 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442601    4525 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:36:17.458114 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442604    4525 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:36:17.458114 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442607    4525 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:36:17.458784 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442610    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:36:17.458784 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442613    4525 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:36:17.458784 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442616    4525 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:36:17.458784 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442619    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:36:17.458784 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442622    4525 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:36:17.458784 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442625    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:36:17.458784 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442628    4525 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:36:17.458784 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442631    4525 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:36:17.458784 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442637    4525 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:36:17.458784 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442641    4525 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:36:17.458784 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442646    4525 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:36:17.458784 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442651    4525 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:36:17.458784 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442656    4525 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:36:17.458784 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442662    4525 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:36:17.458784 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442665    4525 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:36:17.458784 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442668    4525 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:36:17.458784 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442670    4525 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:36:17.458784 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442673    4525 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:36:17.458784 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442694    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:36:17.458784 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442698    4525 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:36:17.459265 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442702    4525 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:36:17.459265 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442706    4525 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:36:17.459265 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442711    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:36:17.459265 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442715    4525 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:36:17.459265 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442720    4525 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:36:17.459265 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442724    4525 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:36:17.459265 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442730    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:36:17.459265 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442739    4525 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:36:17.459265 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442746    4525 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:36:17.459265 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442753    4525 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:36:17.459265 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442759    4525 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:36:17.459265 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442765    4525 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:36:17.459265 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442769    4525 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:36:17.459265 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442774    4525 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:36:17.459265 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442779    4525 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:36:17.459265 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442783    4525 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:36:17.459265 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442787    4525 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:36:17.459265 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442791    4525 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:36:17.459265 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442795    4525 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:36:17.459733 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442800    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:36:17.459733 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442804    4525 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:36:17.459733 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442809    4525 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:36:17.459733 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442812    4525 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:36:17.459733 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442814    4525 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:36:17.459733 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442817    4525 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:36:17.459733 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442820    4525 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:36:17.459733 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442823    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:36:17.459733 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442826    4525 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:36:17.459733 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442828    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:36:17.459733 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442832    4525 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:36:17.459733 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442834    4525 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:36:17.459733 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442837    4525 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:36:17.459733 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442840    4525 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:36:17.459733 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442842    4525 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:36:17.459733 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442845    4525 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:36:17.459733 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442848    4525 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:36:17.459733 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442850    4525 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:36:17.459733 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442853    4525 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:36:17.460180 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442855    4525 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:36:17.460180 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442859    4525 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:36:17.460180 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442865    4525 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:36:17.460180 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442870    4525 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:36:17.460180 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442874    4525 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:36:17.460180 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442878    4525 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:36:17.460180 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442880    4525 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:36:17.460180 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442883    4525 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:36:17.460180 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442886    4525 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:36:17.460180 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442888    4525 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:36:17.460180 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442891    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:36:17.460180 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442893    4525 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:36:17.460180 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442897    4525 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:36:17.460180 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442902    4525 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:36:17.460180 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442907    4525 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:36:17.460180 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442912    4525 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:36:17.460180 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442916    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:36:17.460180 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442921    4525 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:36:17.460180 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442926    4525 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:36:17.460635 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442930    4525 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:36:17.460635 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442935    4525 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:36:17.460635 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442940    4525 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:36:17.460635 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.442944    4525 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:36:17.460635 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.442992    4525 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 18:36:17.460635 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443223    4525 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:36:17.460635 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443234    4525 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:36:17.460635 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443239    4525 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:36:17.460635 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443244    4525 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:36:17.460635 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443249    4525 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:36:17.460635 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443254    4525 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:36:17.460635 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443259    4525 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:36:17.460635 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443262    4525 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:36:17.460635 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443265    4525 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:36:17.460635 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443268    4525 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:36:17.460635 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443272    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:36:17.461066 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443277    4525 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:36:17.461066 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443282    4525 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:36:17.461066 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443286    4525 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:36:17.461066 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443291    4525 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:36:17.461066 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443295    4525 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:36:17.461066 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443299    4525 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:36:17.461066 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443303    4525 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:36:17.461066 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443307    4525 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:36:17.461066 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443312    4525 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:36:17.461066 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443318    4525 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:36:17.461066 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443323    4525 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:36:17.461066 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443327    4525 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:36:17.461066 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443333    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:36:17.461066 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443338    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:36:17.461066 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443341    4525 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:36:17.461066 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443344    4525 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:36:17.461066 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443347    4525 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:36:17.461066 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443350    4525 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:36:17.461066 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443353    4525 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:36:17.461510 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443356    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:36:17.461510 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443359    4525 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:36:17.461510 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443363    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:36:17.461510 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443370    4525 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:36:17.461510 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443376    4525 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:36:17.461510 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443382    4525 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:36:17.461510 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443387    4525 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:36:17.461510 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443392    4525 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:36:17.461510 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443398    4525 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:36:17.461510 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443401    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:36:17.461510 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443404    4525 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:36:17.461510 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443408    4525 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:36:17.461510 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443411    4525 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:36:17.461510 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443414    4525 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:36:17.461510 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443418    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:36:17.461510 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443421    4525 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:36:17.461510 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443425    4525 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:36:17.461510 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443430    4525 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:36:17.461510 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443435    4525 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:36:17.461988 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443439    4525 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:36:17.461988 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443444    4525 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:36:17.461988 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443448    4525 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:36:17.461988 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443455    4525 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:36:17.461988 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443459    4525 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:36:17.461988 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443463    4525 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:36:17.461988 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443466    4525 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:36:17.461988 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443468    4525 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:36:17.461988 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443471    4525 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:36:17.461988 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443474    4525 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:36:17.461988 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443477    4525 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:36:17.461988 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443479    4525 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:36:17.461988 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443483    4525 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:36:17.461988 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443489    4525 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:36:17.461988 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443495    4525 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:36:17.461988 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443500    4525 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:36:17.461988 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443505    4525 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:36:17.461988 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443509    4525 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:36:17.461988 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443514    4525 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:36:17.461988 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443518    4525 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:36:17.462487 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443523    4525 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:36:17.462487 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443527    4525 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:36:17.462487 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443531    4525 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:36:17.462487 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443535    4525 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:36:17.462487 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443539    4525 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:36:17.462487 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443543    4525 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:36:17.462487 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443548    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:36:17.462487 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443552    4525 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:36:17.462487 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443557    4525 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:36:17.462487 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443562    4525 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:36:17.462487 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443566    4525 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:36:17.462487 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443570    4525 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:36:17.462487 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443574    4525 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:36:17.462487 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443579    4525 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:36:17.462487 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443584    4525 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:36:17.462487 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443590    4525 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:36:17.462487 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443595    4525 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:36:17.462487 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443599    4525 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:36:17.462487 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443603    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:36:17.462487 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443608    4525 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:36:17.463054 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443614    4525 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:36:17.463054 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443618    4525 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:36:17.463054 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443623    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:36:17.463054 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443628    4525 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:36:17.463054 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443632    4525 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:36:17.463054 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443638    4525 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:36:17.463054 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443643    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:36:17.463054 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443648    4525 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:36:17.463054 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443652    4525 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:36:17.463054 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443657    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:36:17.463054 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443661    4525 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:36:17.463054 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443665    4525 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:36:17.463054 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443670    4525 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:36:17.463054 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443674    4525 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:36:17.463054 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443695    4525 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:36:17.463054 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443700    4525 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:36:17.463054 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443704    4525 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:36:17.463478 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.443750    4525 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 18:36:17.463478 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443798    4525 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:36:17.463478 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443803    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:36:17.463478 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443808    4525 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:36:17.463478 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443814    4525 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:36:17.463478 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443819    4525 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:36:17.463478 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443823    4525 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:36:17.463478 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443827    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:36:17.463478 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443831    4525 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:36:17.463478 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443836    4525 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:36:17.463478 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443842    4525 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:36:17.463478 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443846    4525 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:36:17.463478 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443850    4525 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:36:17.463478 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443854    4525 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:36:17.463478 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443859    4525 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:36:17.463478 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443862    4525 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:36:17.463911 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443867    4525 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:36:17.463911 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443871    4525 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:36:17.463911 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443875    4525 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:36:17.463911 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443879    4525 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:36:17.463911 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443884    4525 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:36:17.463911 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443889    4525 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:36:17.463911 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443892    4525 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:36:17.463911 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443895    4525 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:36:17.463911 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443898    4525 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:36:17.463911 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443902    4525 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:36:17.463911 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443906    4525 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:36:17.463911 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443909    4525 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:36:17.463911 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443913    4525 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:36:17.463911 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443917    4525 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:36:17.463911 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443921    4525 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:36:17.463911 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443926    4525 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:36:17.463911 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443931    4525 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:36:17.463911 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443937    4525 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:36:17.463911 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443941    4525 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:36:17.463911 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443946    4525 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:36:17.464379 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443953    4525 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:36:17.464379 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443958    4525 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:36:17.464379 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443964    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:36:17.464379 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443969    4525 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:36:17.464379 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443975    4525 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:36:17.464379 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443979    4525 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:36:17.464379 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443987    4525 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:36:17.464379 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443991    4525 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:36:17.464379 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443994    4525 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:36:17.464379 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.443997    4525 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:36:17.464379 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.444000    4525 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:36:17.464379 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.444002    4525 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:36:17.464379 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.444005    4525 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:36:17.464379 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.444008    4525 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:36:17.464379 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.444011    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:36:17.464379 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.444013    4525 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:36:17.464379 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.444018    4525 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:36:17.464379 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.444024    4525 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:36:17.464379 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.444028    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:36:17.464379 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.444033    4525 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:36:17.464879 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.444038    4525 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:36:17.464879 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.444042    4525 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:36:17.464879 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.444047    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:36:17.464879 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.444051    4525 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:36:17.464879 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.444055    4525 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:36:17.464879 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.444059    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:36:17.464879 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.444063    4525 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:36:17.464879 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.444067    4525 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:36:17.464879 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.444071    4525 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:36:17.464879 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.444075    4525 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:36:17.464879 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.444080    4525 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:36:17.464879 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.444084    4525 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:36:17.464879 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.444088    4525 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:36:17.464879 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.444091    4525 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:36:17.464879 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.444095    4525 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:36:17.464879 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.444099    4525 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:36:17.464879 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.444104    4525 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:36:17.464879 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.444108    4525 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:36:17.464879 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.444113    4525 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:36:17.465463 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.444118    4525 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:36:17.465463 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.444122    4525 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:36:17.465463 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.444126    4525 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:36:17.465463 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.444130    4525 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:36:17.465463 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.444134    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:36:17.465463 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.444139    4525 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:36:17.465463 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.444143    4525 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:36:17.465463 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.444146    4525 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:36:17.465463 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.444150    4525 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:36:17.465463 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.444154    4525 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:36:17.465463 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.444159    4525 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:36:17.465463 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.444164    4525 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:36:17.465463 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.444167    4525 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:36:17.465463 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.444170    4525 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:36:17.465463 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.444175    4525 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:36:17.465463 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.444179    4525 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:36:17.465463 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.444183    4525 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:36:17.465463 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.444187    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:36:17.465463 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.444191    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:36:17.466130 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.444195    4525 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:36:17.466130 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.444199    4525 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:36:17.466130 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.444204    4525 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:36:17.466130 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.444208    4525 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:36:17.466130 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.444212    4525 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:36:17.466130 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.444217    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:36:17.466130 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.444224    4525 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:36:17.466130 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.444228    4525 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:36:17.466130 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.444237    4525 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:36:17.466130 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.444242    4525 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:36:17.466130 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.444246    4525 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:36:17.466130 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.444249    4525 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:36:17.466130 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:17.444254    4525 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:36:17.466130 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.444324    4525 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 18:36:17.466130 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.445611    4525 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 18:36:17.466130 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.445701    4525 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 18:36:17.466130 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.445966    4525 server.go:1014] "Starting client certificate rotation"
Oct 30 18:36:17.466130 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.447269    4525 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 18:36:17.466846 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.447323    4525 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 18:36:17.466846 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.447380    4525 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 18:30:37 +0000 UTC" deadline="2028-06-30 21:44:26.287598183 +0000 UTC"
Oct 30 18:36:17.466846 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.447423    4525 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14619h8m8.840179045s"
Oct 30 18:36:17.466846 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.449446    4525 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 18:36:17.466846 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.451588    4525 log.go:25] "Validated CRI v1 runtime API"
Oct 30 18:36:17.466846 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.453147    4525 log.go:25] "Validated CRI v1 image API"
Oct 30 18:36:17.466846 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.453963    4525 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 18:36:17.466846 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.457525    4525 fs.go:135] Filesystem UUIDs: map[1a093d95-fcdf-4895-a53f-68dfb0bce584:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 8fe4a33a-1be2-41cb-aecd-197927ea84cf:/dev/nvme0n1p4]
Oct 30 18:36:17.467187 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.457551    4525 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/02c85c4602ad9496b82c8c4f0e457cf814e30c624820667e7374a0f7b35e3ded/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/02c85c4602ad9496b82c8c4f0e457cf814e30c624820667e7374a0f7b35e3ded/userdata/shm major:0 minor:214 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0400f8430258e09dd25c90c28b8699012a50dcb20ec19647d0dd0c6053101622/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0400f8430258e09dd25c90c28b8699012a50dcb20ec19647d0dd0c6053101622/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1991b9540f8953c91e896fce71e99b569200da8ee8d7b0b13aeba2742cbc600d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1991b9540f8953c91e896fce71e99b569200da8ee8d7b0b13aeba2742cbc600d/userdata/shm major:0 minor:93 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1fe07a3980476efe57f370a3f8ffd99d740abe566fde8bf60f8a712e032faac7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1fe07a3980476efe57f370a3f8ffd99d740abe566fde8bf60f8a712e032faac7/userdata/shm major:0 minor:227 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2dd07382ff8e72ba2fc1d71510c152d5aa38c29fab5a4d75da82e3b8b9a0770a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2dd07382ff8e72ba2fc1d71510c152d5aa38c29fab5a4d75da82e3b8b9a0770a/userdata/shm major:0 minor:218 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/376c012613cf5f19aaba03fb9e8f0bcbf03ca640aae6d8e0272d46455bf0a9dd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/376c012613cf5f19aaba03fb9e8f0bcbf03ca640aae6d8e0272d46455bf0a9dd/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3cdeb882d3478e934ee1baa73a787abbf040e0aa8fc251268cfec63a14e2d499/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3cdeb882d3478e934ee1baa73a787abbf040e0aa8fc251268cfec63a14e2d499/userdata/shm major:0 minor:225 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4613d69f1e8a92f7b0ccf60ce4f2bf0a1ffbe0c8347c420867cde38f8e36fdf4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4613d69f1e8a92f7b0ccf60ce4f2bf0a1ffbe0c8347c420867cde38f8e36fdf4/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/48f922805b42fae668301dd5c3283d63659b7313d58239bb7c7c369ca9b21b75/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/48f922805b42fae668301dd5c3283d63659b7313d58239bb7c7c369ca9b21b75/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4a064f4b59347185376167aae61af87390c30621b87547a175af427d35b976fc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4a064f4b59347185376167aae61af87390c30621b87547a175af427d35b976fc/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4a20442cabc818c37d363f8a47e7e379db232f5f58eb34c3c35d902407158b61/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4a20442cabc818c37d363f8a47e7e379db232f5f58eb34c3c35d902407158b61/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/693fe2dc18ff556cbd492a9ff21e105b068a26391601a45d32d96e1c677809ea/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/693fe2dc18ff556cbd492a9ff21e105b068a26391601a45d32d96e1c677809ea/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8884baec4ce0a848bd5a0ba3a8715596d06504ac637370760df9784b2cb4d109/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8884baec4ce0a848bd5a0ba3a8715596d06504ac637370760df9784b2cb4d109/userdata/shm major:0 minor:89 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d73ed64ee847d60fd293abba94e11add3369059a1974a6bea0d29f543c60916a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d73ed64ee847d60fd293abba94e11add3369059a1974a6bea0d29f543c60916a/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/da579d74802736a0baaca917640290659ec92fddd742408ef7cd256a6d51bc58/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/da579d74802736a0baaca917640290659ec92fddd742408ef7cd256a6d51bc58/userdata/shm major:0 minor:235 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dfcdc334a1ec738dcf8b9e141096e23049b36ce9a976f243903ac75b6adfdcf5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dfcdc334a1ec738dcf8b9e141096e23049b36ce9a976f243903ac75b6adfdcf5/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/089fdd4f-daa0-4b3e-b81e-26a764a526bb/volumes/kubernetes.io~projected/kube-api-access-sx6qn:{mountpoint:/var/lib/kubelet/pods/089fdd4f-daa0-4b3e-b81e-26a764a526bb/volumes/kubernetes.io~projected/kube-api-access-sx6qn major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/093a372e-74ea-4820-82c8-0dff7c201f10/volumes/kubernetes.io~projected/kube-api-access-plzcb:{mountpoint:/var/lib/kubelet/pods/093a372e-74ea-4820-82c8-0dff7c201f10/volumes/kubernetes.io~projected/kube-api-access-plzcb major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0dc90efb-2709-4894-b457-6cadb8abd801/volumes/kubernetes.io~projected/kube-api-access-7lbr9:{mountpoint:/var/lib/kubelet/pods/0dc90efb-2709-4894-b457-6cadb8abd801/volumes/kubernetes.io~projected/kube-api-access-7lbr9 major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1315feec-3a59-4cae-ad7e-8ada8c5e4788/volumes/kubernetes.io~projected/kube-api-access-tp8kg:{mountpoint:/var/lib/kubelet/pods/1315feec-3a59-4cae-ad7e-8ada8c5e4788/volumes/kubernetes.io~projected/kube-api-access-tp8kg major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/25255277-4f3b-418c-9e85-2a82e7a13107/volumes/kubernetes.io~projected/kube-api-access-j7rn7:{mountpoint:/var/lib/kubelet/pods/25255277-4f3b-418c-9e85-2a82e7a13107/volumes/kubernetes.io~projected/kube-api-access-j7rn7 major:0 minor:234 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/25255277-4f3b-418c-9e85-2a82e7a13107/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/25255277-4f3b-418c-9e85-2a82e7a13107/volumes/kubernetes.io~secret/signing-key major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2ca560f4-9aa8-4bc6-9db1-6992a1fa7c1f/volumes/kubernetes.io~projected/kube-api-access-7kczq:{mountpoint:/var/lib/kubelet/pods/2ca560f4-9aa8-4bc6-9db1-6992a1fa7c1f/volumes/kubernetes.io~projected/kube-api-access-7kczq major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/42b71e2d-a8f2-43cf-8487-0e5da2338c54/volumes/kubernetes.io~projected/kube-api-access-gm9cn:{mountpoint:/var/lib/kubelet/pods/42b71e2d-a8f2-43cf-8487-0e5da2338c54/volumes/kubernetes.io~projected/kube-api-access-gm9cn major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4b27cd9e-69b2-4480-a84f-5b463cec1a0e/volumes/kubernetes.io~projected/kube-api-access-rfrpn:{mountpoint:/var/lib/kubelet/pods/4b27cd9e-69b2-4480-a84f-5b463cec1a0e/volumes/kubernetes.io~projected/kube-api-access-rfrpn major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/545dd3b9-01a9-40be-8762-5476a7a9ede4/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/545dd3b9-01a9-40be-8762-5476a7a9ede4/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/545dd3b9-01a9-40be-8762-5476a7a9ede4/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/545dd3b9-01a9-40be-8762-5476a7a9ede4/volumes/kubernetes.io~empty-dir/tmp major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/545dd3b9-01a9-40be-8762-5476a7a9ede4/volumes/kubernetes.io~projected/kube-api-access-cq2h8:{mountpoint:/var/lib/kubelet/pods/545dd3b9-01a9-40be-8762-5476a7a9ede4/volumes/kubernetes.io~projected/kube-api-access-cq2h8 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a/volumes/kubernetes.io~projected/kube-api-access-vl5cp:{mountpoint:/var/lib/kubelet/pods/5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a/volumes/kubernetes.io~projected/kube-api-access-vl5cp major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/77c2eb22-4b45-486f-8c62-bd14086942f5/volumes/kubernetes.io~projected/kube-api-access-d2phv:{mountpoint:/var/lib/kubelet/pods/77c2eb22-4b45-486f-8c62-bd14086942f5/volumes/kubernetes.io~projected/kube-api-access-d2phv major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7c3ae3e0-ddf1-414f-bb79-e4afc0527b26/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/7c3ae3e0-ddf1-414f-bb79-e4afc0527b26/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8bcb59a4-381f-49ba-b732-f5e4658707de/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/8bcb59a4-381f-49ba-b732-f5e4658707de/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8bcb59a4-381f-49ba-b732-f5e4658707de/volumes/kubernetes.io~projected/kube-api-access-dltbj:{mountpoint:/var/lib/kubelet/pods/8bcb59a4-381f-49ba-b732-f5e4658707de/volumes/kubernetes.io~projected/kube-api-access-dltbj major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8bcb59a4-381f-49ba-b732-f5e4658707de/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/8bcb59a4-381f-49ba-b732-f5e4658707de/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/97f514c8-0d31-4103-81c5-45886ff831f0/volumes/kubernetes.io~projected/kube-api-access-5t65n:{mountpoint:/var/lib/kubelet/pods/97f514c8-0d31-4103-81c5-45886ff831f0/volumes/kubernetes.io~projected/kube-api-access-5t65n major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9ce43bfa-1e19-4fb5-90ff-42751cdd1a50/volumes/kubernetes.io~projected/kube-api-access-862ff:{mountpoint:/var/lib/kubelet/pods/9ce43bfa-1e19-4fb5-90ff-42751cdd1a50/volumes/kubernetes.io~projected/kube-api-access-862ff major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bd08a0fa-46bc-4250-985c-d2580a2225b8/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/bd08a0fa-46bc-4250-985c-d2580a2225b8/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d8f39e97-7f68-47da-a247-adf78b523afd/volumes/kubernetes.io~projected/kube-api-access-fq9k2:{mountpoint:/var/lib/kubelet/pods/d8f39e97-7f68-47da-a247-adf78b523afd/volumes/kubernetes.io~projected/kube-api-access-fq9k2 major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f32f2498-2fc8-4bc6-b368-b53d156755c5/volumes/kubernetes.io~projected/kube-api-access-t7jl8:{mountpoint:/var/lib/kubelet/pods/f32f2498-2fc8-4bc6-b368-b53d156755c5/volumes/kubernetes.io~projected/kube-api-access-t7jl8 major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f32f2498-2fc8-4bc6-b368-b53d156755c5/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/f32f2498-2fc8-4bc6-b368-b53d156755c5/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/f1c3f92f1b03e02e2ef539edca7c7d43048be55078a45e7b36d8cedc4bc7b78a/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/fdc5a42ccbf6e37370e6a3d617ffd148478b611fadb807e66b5d38c30b8fc0cd/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/b606d32b36ef6b1a8fefe4b6f269faa0d83da4c810c885bca7a5097d197f21b3/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/131ae770714dc695bb67b1abb2b7f5b3ee8c28a10317bf2b28e06dc150bd1bf9/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/36b76d95181a0adf26d5058deeb3588c8bbe99800c4cc89ab321ce97b2e8edc8/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/76d2ec76bec6a56f16fdce6c4542418ba4e6c7ff7af9d12f24724b813bebf2c3/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/75c225d6b8ca7658027392e82f90b190a534f1a247092e92284495d613cdd44f/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/226690fabf13a2a0238a02e383262ebdb357020133df56fdd0952f3d8ee08f62/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/c2e8b229cf830c8875a5338b95cc6a095c2bc4afa8670778983195780fd3982a/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/37ba5386ac6cb08d39df7199f12373bcd2139be88b3d583ce9b2c6bdd965e623/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/f95b2ec0191314cc02a2ca84d1c7e2f06326d167f62c9c22d66031363b8059c1/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/1bbdbebbc42b23c5734bc2aeb74a8658ea3032e869aae07311aecb698926c37e/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-153:{mountpoint:/var/lib/containers/storage/overlay/82cea04a8a2c2da796fb08097c4d5ebfd0e66e52b4036e2300be386d6499a883/merged major:0 minor:153 fsType:overlay blockSize:0} overlay_0-155:{mountpoint:/var/lib/containers/storage/overlay/f6ff63aa181a33ed487851af51d4bee1c80352d744bbc695b8f20153cb46e4f3/merged major:0 minor:155 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/529e80f0052f692048c648a387166aec1c9e6923a6fe49e65a73ccffaee5058c/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-169:{mountpoint:/var/lib/containers/storage/overlay/43164c481b56c6ac18b5cb714f37c4d1b9a8e22779c32c43094e464e3f57460c/merged major:0 minor:169 fsType:overlay blockSize:0} overlay_0-174:{mountpoint:/var/lib/containers/storage/overlay/7d25410c0a26e8909f217b8d8d2042152af4b699e9256a26e70fefbc38e1f4f2/merged major:0 minor:174 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/2a7b16ae2257bf1af451168fdc6e79b5410940a00f3d5f3400e5f8487f5b5fdb/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/cb8cde93fa74515c105c333bd9367d7ea254ef8806ba36e92a7d798c957d979e/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/fa37d43a149422c298c29c69c372e05ea722fec6508fce90bc7ac95231ab11e2/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/417c8580e3a1310b2a6af94e6c77f5f292da72252de80166dfcd842544ea5825/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/4c849d565b01578c2db5662168a5312b2e4c93114bf7c37e0d4b01c03801320f/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-206:{mountpoint:/var/lib/containers/storage/overlay/e150bec104cef8be8bdb10a1bb98de49e2ffddba1843a2444c9ab863005b65ed/merged major:0 minor:206 fsType:overlay blockSize:0} overlay_0-216:{mountpoint:/var/lib/containers/storage/overlay/9e3ea77b1676e84a240b1ee19d956a0adcc5645be256ecaa810695b101c94f57/merged major:0 minor:216 fsType:overlay blockSize:0} overlay_0-220:{mountpoint:/var/lib/containers/storage/overlay/e3b9601c6cd27fa6cf2d68c85daada80687b7c6e70ec12296dc96ce5d8b46dd7/merged major:0 minor:220 fsType:overlay blockSize:0} overlay_0-229:{mountpoint:/var/lib/containers/storage/overlay/ed86417b3b923922aff6c9ee73f39e703da834cadc562c1e51109edbe0a6073e/merged major:0 minor:229 fsType:overlay blockSize:0} overlay_0-231:{mountpoint:/var/lib/containers/storage/overlay/32a66e5cab6bf4c3a3889c2f9f12762a0645ea3bd5906e054968d7d05754f9e8/merged major:0 minor:231 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/105f77b433060ac409a47a2bb1b73d43e0250bc163b6d077e620ec68f0b2393f/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-239:{mountpoint:/var/lib/containers/storage/overlay/5b112c2de76d293b86140aca5845a1bb9cd9ec709cf209417d3589661aa81400/merged major:0 minor:239 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/693e06a19f365bd054ac93d3af8a9eb4fb5d97eb970de3bf5bcbd0af9206f363/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/6bc1450d070db8483fbd9075b83489a4f94c45595a6025b55dde14eed112987c/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/e3dbb187e470450985853b76fb16bae9eea5025d019c63547168281be7879522/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/cc2c8af1d6be6fe84ce9b2605c3778a30fcaeadcf725e10d96a6bbd4185e3db8/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/9f8ffafa9ea37d4ee02f257760546af55e168e3f496db3f44abe1b180f60ddac/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/0bb151b190791909775b8a84f755b9b2ea66c65676c9fb62fbe43e3aad04c584/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/f760105f313301fdc3db6b15fb1100442910a0ff674206c986c7fb91f68699ac/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/0302712b66841e22f931558fc0986d193b3f000e069153c50fbc2043d5a3b45f/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/f561ae0dc922fdaacb61b0a756be31c172345d01ae725caa86d39dbf295fa431/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/839b6d8bf8c49efda6700262ddab47340064c438d49bbf8c2c706b5aeb9db5a0/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/b29806cb0dca5a77b212f2995a5ce577d5c3aa16ea745bfc2e83424afba609fd/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/d2b43b1d2ae1871336cc9d74c063d1672a8751c2c9db6a3398327024e7c0617f/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 30 18:36:17.467429 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.463857    4525 manager.go:217] Machine: {Timestamp:2026-10-30 18:36:17.463364456 +0000 UTC m=+0.089034431 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101049 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec276576394ce06bb5ead00377771a31 SystemUUID:ec276576-394c-e06b-b5ea-d00377771a31 BootID:12c44ce9-f5c7-478e-a77c-65ed7915990b Filesystems:[{Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f32f2498-2fc8-4bc6-b368-b53d156755c5/volumes/kubernetes.io~projected/kube-api-access-t7jl8 DeviceMajor:0 DeviceMinor:197 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4613d69f1e8a92f7b0ccf60ce4f2bf0a1ffbe0c8347c420867cde38f8e36fdf4/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/77c2eb22-4b45-486f-8c62-bd14086942f5/volumes/kubernetes.io~projected/kube-api-access-d2phv DeviceMajor:0 DeviceMinor:59 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8bcb59a4-381f-49ba-b732-f5e4658707de/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:62 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/545dd3b9-01a9-40be-8762-5476a7a9ede4/volumes/kubernetes.io~projected/kube-api-access-cq2h8 DeviceMajor:0 DeviceMinor:65 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f32f2498-2fc8-4bc6-b368-b53d156755c5/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-174 DeviceMajor:0 DeviceMinor:174 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a/volumes/kubernetes.io~projected/kube-api-access-vl5cp DeviceMajor:0 DeviceMinor:201 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-216 DeviceMajor:0 DeviceMinor:216 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/25255277-4f3b-418c-9e85-2a82e7a13107/volumes/kubernetes.io~projected/kube-api-access-j7rn7 DeviceMajor:0 DeviceMinor:234 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/545dd3b9-01a9-40be-8762-5476a7a9ede4/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:61 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-155 DeviceMajor:0 DeviceMinor:155 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/02c85c4602ad9496b82c8c4f0e457cf814e30c624820667e7374a0f7b35e3ded/userdata/shm DeviceMajor:0 DeviceMinor:214 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2dd07382ff8e72ba2fc1d71510c152d5aa38c29fab5a4d75da82e3b8b9a0770a/userdata/shm DeviceMajor:0 DeviceMinor:218 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/693fe2dc18ff556cbd492a9ff21e105b068a26391601a45d32d96e1c677809ea/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2ca560f4-9aa8-4bc6-9db1-6992a1fa7c1f/volumes/kubernetes.io~projected/kube-api-access-7kczq DeviceMajor:0 DeviceMinor:60 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-169 DeviceMajor:0 DeviceMinor:169 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-220 DeviceMajor:0 DeviceMinor:220 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/97f514c8-0d31-4103-81c5-45886ff831f0/volumes/kubernetes.io~projected/kube-api-access-5t65n DeviceMajor:0 DeviceMinor:222 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/093a372e-74ea-4820-82c8-0dff7c201f10/volumes/kubernetes.io~projected/kube-api-access-plzcb DeviceMajor:0 DeviceMinor:56 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1fe07a3980476efe57f370a3f8ffd99d740abe566fde8bf60f8a712e032faac7/userdata/shm DeviceMajor:0 DeviceMinor:227 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8bcb59a4-381f-49ba-b732-f5e4658707de/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/9ce43bfa-1e19-4fb5-90ff-42751cdd1a50/volumes/kubernetes.io~projected/kube-api-access-862ff DeviceMajor:0 DeviceMinor:224 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/7c3ae3e0-ddf1-414f-bb79-e4afc0527b26/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:208 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3cdeb882d3478e934ee1baa73a787abbf040e0aa8fc251268cfec63a14e2d499/userdata/shm DeviceMajor:0 DeviceMinor:225 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/089fdd4f-daa0-4b3e-b81e-26a764a526bb/volumes/kubernetes.io~projected/kube-api-access-sx6qn DeviceMajor:0 DeviceMinor:64 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bd08a0fa-46bc-4250-985c-d2580a2225b8/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/42b71e2d-a8f2-43cf-8487-0e5da2338c54/volumes/kubernetes.io~projected/kube-api-access-gm9cn DeviceMajor:0 DeviceMinor:67 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d73ed64ee847d60fd293abba94e11add3369059a1974a6bea0d29f543c60916a/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/376c012613cf5f19aaba03fb9e8f0bcbf03ca640aae6d8e0272d46455bf0a9dd/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1991b9540f8953c91e896fce71e99b569200da8ee8d7b0b13aeba2742cbc600d/userdata/shm DeviceMajor:0 DeviceMinor:93 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/4b27cd9e-69b2-4480-a84f-5b463cec1a0e/volumes/kubernetes.io~projected/kube-api-access-rfrpn DeviceMajor:0 DeviceMinor:213 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1315feec-3a59-4cae-ad7e-8ada8c5e4788/volumes/kubernetes.io~projected/kube-api-access-tp8kg DeviceMajor:0 DeviceMinor:223 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-229 DeviceMajor:0 DeviceMinor:229 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/25255277-4f3b-418c-9e85-2a82e7a13107/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:233 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4a064f4b59347185376167aae61af87390c30621b87547a175af427d35b976fc/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0400f8430258e09dd25c90c28b8699012a50dcb20ec19647d0dd0c6053101622/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-231 DeviceMajor:0 DeviceMinor:231 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/da579d74802736a0baaca917640290659ec92fddd742408ef7cd256a6d51bc58/userdata/shm DeviceMajor:0 DeviceMinor:235 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-239 DeviceMajor:0 DeviceMinor:239 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/545dd3b9-01a9-40be-8762-5476a7a9ede4/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:63 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d8f39e97-7f68-47da-a247-adf78b523afd/volumes/kubernetes.io~projected/kube-api-access-fq9k2 DeviceMajor:0 DeviceMinor:57 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dfcdc334a1ec738dcf8b9e141096e23049b36ce9a976f243903ac75b6adfdcf5/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0dc90efb-2709-4894-b457-6cadb8abd801/volumes/kubernetes.io~projected/kube-api-access-7lbr9 DeviceMajor:0 DeviceMinor:58 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8bcb59a4-381f-49ba-b732-f5e4658707de/volumes/kubernetes.io~projected/kube-api-access-dltbj DeviceMajor:0 DeviceMinor:66 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8884baec4ce0a848bd5a0ba3a8715596d06504ac637370760df9784b2cb4d109/userdata/shm DeviceMajor:0 DeviceMinor:89 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-153 DeviceMajor:0 DeviceMinor:153 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-206 DeviceMajor:0 DeviceMinor:206 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/48f922805b42fae668301dd5c3283d63659b7313d58239bb7c7c369ca9b21b75/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4a20442cabc818c37d363f8a47e7e379db232f5f58eb34c3c35d902407158b61/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:02c85c4602ad949 MacAddress:9a:73:e1:18:42:9d Speed:10000 Mtu:8901} {Name:1fe07a3980476ef MacAddress:be:ce:10:d2:ae:9a Speed:10000 Mtu:8901} {Name:2dd07382ff8e72b MacAddress:a2:05:2d:8c:3b:be Speed:10000 Mtu:8901} {Name:3cdeb882d3478e9 MacAddress:e2:d7:8d:14:05:63 Speed:10000 Mtu:8901} {Name:693fe2dc18ff556 MacAddress:b2:37:81:dc:1b:ff Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:d9:61:9b:59 Speed:0 Mtu:9001} {Name:br-int MacAddress:82:5e:00:50:2a:5e Speed:0 Mtu:8901} {Name:da579d74802736a MacAddress:1e:9a:ee:dd:da:f4 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:d9:61:9b:59 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:52:16:b9:5e:c8:f8 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:f6:1a:d0:ea:82:48 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 18:36:17.467429 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.464123    4525 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 18:36:17.467429 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.464166    4525 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 18:36:17.467429 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.464434    4525 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 18:36:17.467429 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.464457    4525 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-135.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 18:36:17.467429 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.464589    4525 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 18:36:17.467429 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.464597    4525 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 18:36:17.467429 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.464607    4525 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 18:36:17.467429 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.464620    4525 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 18:36:17.467429 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.464726    4525 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 18:36:17.467429 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.464883    4525 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 18:36:17.467429 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.464984    4525 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 18:36:17.467429 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.465017    4525 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 18:36:17.467429 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.465031    4525 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 18:36:17.467429 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.465040    4525 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 18:36:17.467429 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.465049    4525 kubelet.go:405] "Adding apiserver pod source"
Oct 30 18:36:17.467429 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.465057    4525 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 18:36:17.467429 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.466079    4525 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 18:36:17.467429 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.466100    4525 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 18:36:17.468125 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.467596    4525 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 18:36:17.482486 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.482462    4525 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:36:17.484099 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.484077    4525 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:36:17.498777 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.498763    4525 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 18:36:17.498777 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.498777    4525 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 18:36:17.498893 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.498883    4525 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 18:36:17.499085 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.499075    4525 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 18:36:17.499123 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.499104    4525 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 18:36:17.499210 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.499201    4525 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 18:36:17.499239 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.499215    4525 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 18:36:17.499239 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.499221    4525 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 18:36:17.499239 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.499227    4525 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 18:36:17.499239 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.499233    4525 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 18:36:17.499239 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.499238    4525 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 18:36:17.499401 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.499245    4525 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 18:36:17.499401 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.499254    4525 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 18:36:17.499401 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.499261    4525 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 18:36:17.499401 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.499267    4525 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 18:36:17.499401 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.499274    4525 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 18:36:17.499401 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.499284    4525 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 18:36:17.499401 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.499312    4525 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 18:36:17.499401 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.499320    4525 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 18:36:17.502805 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.502794    4525 server.go:1263] "Started kubelet"
Oct 30 18:36:17.503403 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.502928    4525 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 18:36:17.503403 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.502921    4525 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 18:36:17.503403 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.503001    4525 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 18:36:17.503403 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.503285    4525 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 18:36:17.503323 ip-10-0-137-135 systemd[1]: Started Kubernetes Kubelet.
Oct 30 18:36:17.504198 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.504096    4525 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 18:36:17.506039 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.506016    4525 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 18:36:17.506163 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.506148    4525 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 18:36:17.506163 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.506162    4525 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 18:36:17.506376 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.506334    4525 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 18:36:17.506512 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.506477    4525 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 18:36:17.507507 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.507459    4525 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 18:30:39 +0000 UTC" deadline="2028-04-08 22:08:18.664850591 +0000 UTC"
Oct 30 18:36:17.507507 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.507506    4525 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12627h32m1.157348047s"
Oct 30 18:36:17.511801 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.509117    4525 factory.go:55] Registering systemd factory
Oct 30 18:36:17.511801 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.509136    4525 factory.go:223] Registration of the systemd container factory successfully
Oct 30 18:36:17.511801 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.509412    4525 factory.go:153] Registering CRI-O factory
Oct 30 18:36:17.511801 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.509425    4525 factory.go:223] Registration of the crio container factory successfully
Oct 30 18:36:17.511801 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.509486    4525 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 18:36:17.511801 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.509503    4525 factory.go:103] Registering Raw factory
Oct 30 18:36:17.511801 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.509519    4525 manager.go:1199] Started watching for new ooms in manager
Oct 30 18:36:17.511801 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.509597    4525 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="093a372e-74ea-4820-82c8-0dff7c201f10" volumeName="kubernetes.io/configmap/093a372e-74ea-4820-82c8-0dff7c201f10-serviceca" seLinuxMountContext=""
Oct 30 18:36:17.511801 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.509672    4525 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0dc90efb-2709-4894-b457-6cadb8abd801" volumeName="kubernetes.io/configmap/0dc90efb-2709-4894-b457-6cadb8abd801-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 18:36:17.511801 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.509712    4525 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8f39e97-7f68-47da-a247-adf78b523afd" volumeName="kubernetes.io/configmap/d8f39e97-7f68-47da-a247-adf78b523afd-cni-binary-copy" seLinuxMountContext=""
Oct 30 18:36:17.511801 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.509732    4525 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8f39e97-7f68-47da-a247-adf78b523afd" volumeName="kubernetes.io/configmap/d8f39e97-7f68-47da-a247-adf78b523afd-multus-daemon-config" seLinuxMountContext=""
Oct 30 18:36:17.511801 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.509745    4525 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8f39e97-7f68-47da-a247-adf78b523afd" volumeName="kubernetes.io/projected/d8f39e97-7f68-47da-a247-adf78b523afd-kube-api-access-fq9k2" seLinuxMountContext=""
Oct 30 18:36:17.511801 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.509764    4525 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1315feec-3a59-4cae-ad7e-8ada8c5e4788" volumeName="kubernetes.io/projected/1315feec-3a59-4cae-ad7e-8ada8c5e4788-kube-api-access-tp8kg" seLinuxMountContext=""
Oct 30 18:36:17.511801 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.509779    4525 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="545dd3b9-01a9-40be-8762-5476a7a9ede4" volumeName="kubernetes.io/projected/545dd3b9-01a9-40be-8762-5476a7a9ede4-kube-api-access-cq2h8" seLinuxMountContext=""
Oct 30 18:36:17.511801 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.509794    4525 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bcb59a4-381f-49ba-b732-f5e4658707de" volumeName="kubernetes.io/configmap/8bcb59a4-381f-49ba-b732-f5e4658707de-env-overrides" seLinuxMountContext=""
Oct 30 18:36:17.511801 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.509814    4525 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bcb59a4-381f-49ba-b732-f5e4658707de" volumeName="kubernetes.io/projected/8bcb59a4-381f-49ba-b732-f5e4658707de-kube-api-access-dltbj" seLinuxMountContext=""
Oct 30 18:36:17.511801 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.509828    4525 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ce43bfa-1e19-4fb5-90ff-42751cdd1a50" volumeName="kubernetes.io/empty-dir/9ce43bfa-1e19-4fb5-90ff-42751cdd1a50-archives-path" seLinuxMountContext=""
Oct 30 18:36:17.511801 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.509846    4525 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f32f2498-2fc8-4bc6-b368-b53d156755c5" volumeName="kubernetes.io/secret/f32f2498-2fc8-4bc6-b368-b53d156755c5-cert" seLinuxMountContext=""
Oct 30 18:36:17.511801 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.509863    4525 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25255277-4f3b-418c-9e85-2a82e7a13107" volumeName="kubernetes.io/secret/25255277-4f3b-418c-9e85-2a82e7a13107-signing-key" seLinuxMountContext=""
Oct 30 18:36:17.511801 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.509882    4525 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ca560f4-9aa8-4bc6-9db1-6992a1fa7c1f" volumeName="kubernetes.io/projected/2ca560f4-9aa8-4bc6-9db1-6992a1fa7c1f-kube-api-access-7kczq" seLinuxMountContext=""
Oct 30 18:36:17.511801 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.509888    4525 manager.go:319] Starting recovery of all containers
Oct 30 18:36:17.511801 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.509895    4525 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a" volumeName="kubernetes.io/empty-dir/5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a-tmp-dir" seLinuxMountContext=""
Oct 30 18:36:17.511801 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.509939    4525 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a" volumeName="kubernetes.io/projected/5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a-kube-api-access-vl5cp" seLinuxMountContext=""
Oct 30 18:36:17.511801 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.509962    4525 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="77c2eb22-4b45-486f-8c62-bd14086942f5" volumeName="kubernetes.io/projected/77c2eb22-4b45-486f-8c62-bd14086942f5-kube-api-access-d2phv" seLinuxMountContext=""
Oct 30 18:36:17.511801 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.509973    4525 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bcb59a4-381f-49ba-b732-f5e4658707de" volumeName="kubernetes.io/secret/8bcb59a4-381f-49ba-b732-f5e4658707de-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 18:36:17.511801 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.509990    4525 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd08a0fa-46bc-4250-985c-d2580a2225b8" volumeName="kubernetes.io/secret/bd08a0fa-46bc-4250-985c-d2580a2225b8-agent-certs" seLinuxMountContext=""
Oct 30 18:36:17.511801 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.510004    4525 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0dc90efb-2709-4894-b457-6cadb8abd801" volumeName="kubernetes.io/configmap/0dc90efb-2709-4894-b457-6cadb8abd801-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 18:36:17.511801 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.510022    4525 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0dc90efb-2709-4894-b457-6cadb8abd801" volumeName="kubernetes.io/projected/0dc90efb-2709-4894-b457-6cadb8abd801-kube-api-access-7lbr9" seLinuxMountContext=""
Oct 30 18:36:17.511801 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.510034    4525 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4b27cd9e-69b2-4480-a84f-5b463cec1a0e" volumeName="kubernetes.io/projected/4b27cd9e-69b2-4480-a84f-5b463cec1a0e-kube-api-access-rfrpn" seLinuxMountContext=""
Oct 30 18:36:17.511801 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.510046    4525 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a" volumeName="kubernetes.io/configmap/5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a-config-volume" seLinuxMountContext=""
Oct 30 18:36:17.511801 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.510063    4525 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ce43bfa-1e19-4fb5-90ff-42751cdd1a50" volumeName="kubernetes.io/configmap/9ce43bfa-1e19-4fb5-90ff-42751cdd1a50-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 18:36:17.511801 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.510091    4525 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="089fdd4f-daa0-4b3e-b81e-26a764a526bb" volumeName="kubernetes.io/projected/089fdd4f-daa0-4b3e-b81e-26a764a526bb-kube-api-access-sx6qn" seLinuxMountContext=""
Oct 30 18:36:17.511801 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.510110    4525 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="545dd3b9-01a9-40be-8762-5476a7a9ede4" volumeName="kubernetes.io/empty-dir/545dd3b9-01a9-40be-8762-5476a7a9ede4-etc-tuned" seLinuxMountContext=""
Oct 30 18:36:17.511801 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.510178    4525 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="545dd3b9-01a9-40be-8762-5476a7a9ede4" volumeName="kubernetes.io/empty-dir/545dd3b9-01a9-40be-8762-5476a7a9ede4-tmp" seLinuxMountContext=""
Oct 30 18:36:17.511801 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.510203    4525 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bcb59a4-381f-49ba-b732-f5e4658707de" volumeName="kubernetes.io/configmap/8bcb59a4-381f-49ba-b732-f5e4658707de-ovnkube-config" seLinuxMountContext=""
Oct 30 18:36:17.511801 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.510234    4525 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ce43bfa-1e19-4fb5-90ff-42751cdd1a50" volumeName="kubernetes.io/configmap/9ce43bfa-1e19-4fb5-90ff-42751cdd1a50-service-ca-bundle" seLinuxMountContext=""
Oct 30 18:36:17.511801 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.510262    4525 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="093a372e-74ea-4820-82c8-0dff7c201f10" volumeName="kubernetes.io/projected/093a372e-74ea-4820-82c8-0dff7c201f10-kube-api-access-plzcb" seLinuxMountContext=""
Oct 30 18:36:17.511801 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.510277    4525 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0dc90efb-2709-4894-b457-6cadb8abd801" volumeName="kubernetes.io/configmap/0dc90efb-2709-4894-b457-6cadb8abd801-cni-binary-copy" seLinuxMountContext=""
Oct 30 18:36:17.511801 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.510287    4525 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bcb59a4-381f-49ba-b732-f5e4658707de" volumeName="kubernetes.io/configmap/8bcb59a4-381f-49ba-b732-f5e4658707de-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 18:36:17.511801 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.510324    4525 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f32f2498-2fc8-4bc6-b368-b53d156755c5" volumeName="kubernetes.io/projected/f32f2498-2fc8-4bc6-b368-b53d156755c5-kube-api-access-t7jl8" seLinuxMountContext=""
Oct 30 18:36:17.511801 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.510335    4525 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25255277-4f3b-418c-9e85-2a82e7a13107" volumeName="kubernetes.io/configmap/25255277-4f3b-418c-9e85-2a82e7a13107-signing-cabundle" seLinuxMountContext=""
Oct 30 18:36:17.511801 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.510349    4525 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25255277-4f3b-418c-9e85-2a82e7a13107" volumeName="kubernetes.io/projected/25255277-4f3b-418c-9e85-2a82e7a13107-kube-api-access-j7rn7" seLinuxMountContext=""
Oct 30 18:36:17.511801 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.510362    4525 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="42b71e2d-a8f2-43cf-8487-0e5da2338c54" volumeName="kubernetes.io/projected/42b71e2d-a8f2-43cf-8487-0e5da2338c54-kube-api-access-gm9cn" seLinuxMountContext=""
Oct 30 18:36:17.511801 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.510372    4525 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:36:17.511801 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.510373    4525 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ce43bfa-1e19-4fb5-90ff-42751cdd1a50" volumeName="kubernetes.io/projected/9ce43bfa-1e19-4fb5-90ff-42751cdd1a50-kube-api-access-862ff" seLinuxMountContext=""
Oct 30 18:36:17.511801 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.510843    4525 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1315feec-3a59-4cae-ad7e-8ada8c5e4788" volumeName="kubernetes.io/empty-dir/1315feec-3a59-4cae-ad7e-8ada8c5e4788-data-volume" seLinuxMountContext=""
Oct 30 18:36:17.511801 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.510866    4525 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="42b71e2d-a8f2-43cf-8487-0e5da2338c54" volumeName="kubernetes.io/empty-dir/42b71e2d-a8f2-43cf-8487-0e5da2338c54-tmp-dir" seLinuxMountContext=""
Oct 30 18:36:17.511801 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.510888    4525 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="77c2eb22-4b45-486f-8c62-bd14086942f5" volumeName="kubernetes.io/configmap/77c2eb22-4b45-486f-8c62-bd14086942f5-iptables-alerter-script" seLinuxMountContext=""
Oct 30 18:36:17.511801 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.510902    4525 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c3ae3e0-ddf1-414f-bb79-e4afc0527b26" volumeName="kubernetes.io/secret/7c3ae3e0-ddf1-414f-bb79-e4afc0527b26-original-pull-secret" seLinuxMountContext=""
Oct 30 18:36:17.511801 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.510916    4525 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="97f514c8-0d31-4103-81c5-45886ff831f0" volumeName="kubernetes.io/projected/97f514c8-0d31-4103-81c5-45886ff831f0-kube-api-access-5t65n" seLinuxMountContext=""
Oct 30 18:36:17.511801 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.510934    4525 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd08a0fa-46bc-4250-985c-d2580a2225b8" volumeName="kubernetes.io/configmap/bd08a0fa-46bc-4250-985c-d2580a2225b8-konnectivity-ca" seLinuxMountContext=""
Oct 30 18:36:17.511801 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.510949    4525 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 18:36:17.511801 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.510957    4525 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 18:36:17.514566 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.514548    4525 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 18:36:17.534703 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.534588    4525 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 18:36:17.536349 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.536323    4525 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 18:36:17.536451 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.536354    4525 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 18:36:17.536451 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.536377    4525 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 18:36:17.536451 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:36:17.536420    4525 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 18:36:17.539873 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.539846    4525 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:36:17.547075 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.547051    4525 generic.go:356] "Generic (PLEG): container finished" podID="0dc90efb-2709-4894-b457-6cadb8abd801" containerID="8d7936225fddb7a6f691d96456c9eb14b84185ee83ddbe0b3bbc0a2e3e2c4017" exitCode=0
Oct 30 18:36:17.547075 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.547075    4525 generic.go:356] "Generic (PLEG): container finished" podID="0dc90efb-2709-4894-b457-6cadb8abd801" containerID="656a4d728f0f368cdd5628a8cbe661835d184fb4ce9b86b15e3943de370f600c" exitCode=0
Oct 30 18:36:17.547179 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.547083    4525 generic.go:356] "Generic (PLEG): container finished" podID="0dc90efb-2709-4894-b457-6cadb8abd801" containerID="773700e87b709b16e985402961ac1cdcd01308643cd3d40fa73b940303311e34" exitCode=0
Oct 30 18:36:17.547179 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.547088    4525 generic.go:356] "Generic (PLEG): container finished" podID="0dc90efb-2709-4894-b457-6cadb8abd801" containerID="239187d4ffab1cadbb9bfb61ca3ef1fd67984657e946b634f95668fdbc538d4a" exitCode=0
Oct 30 18:36:17.547179 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.547093    4525 generic.go:356] "Generic (PLEG): container finished" podID="0dc90efb-2709-4894-b457-6cadb8abd801" containerID="3ca28e215731896dd4a45198e7016f158b7cfdbeef77b5b91d5f57b79d58b3c0" exitCode=0
Oct 30 18:36:17.547179 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.547099    4525 generic.go:356] "Generic (PLEG): container finished" podID="0dc90efb-2709-4894-b457-6cadb8abd801" containerID="e3cd51e8ef968b52b94948d7197ce440bc25aed7c48a7226fc6a55af880bf397" exitCode=0
Oct 30 18:36:17.571100 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.571068    4525 generic.go:356] "Generic (PLEG): container finished" podID="09a9d0b6ec5615a60675fe6f5b7fa6e4" containerID="c0efb16a78c761c15493d05fc695c2c98e1a0d5485a24acdea8e26c6ef70f5b0" exitCode=0
Oct 30 18:36:17.577939 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.577918    4525 manager.go:324] Recovery completed
Oct 30 18:36:17.594766 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.594740    4525 cpu_manager.go:226] "Starting" policy="none"
Oct 30 18:36:17.594766 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.594760    4525 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 18:36:17.594922 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.594781    4525 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 18:36:17.594983 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.594965    4525 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 18:36:17.595033 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.594987    4525 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 18:36:17.595033 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.595011    4525 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 18:36:17.595033 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.595019    4525 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 18:36:17.595033 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.595027    4525 policy_none.go:50] "Start"
Oct 30 18:36:17.595136 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.595038    4525 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 18:36:17.595136 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.595051    4525 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 18:36:17.595186 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.595176    4525 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 18:36:17.595186 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.595182    4525 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 18:36:17.595240 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.595188    4525 policy_none.go:44] "Start"
Oct 30 18:36:17.599714 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.599666    4525 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 18:36:17.603319 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.603303    4525 manager.go:341] "Starting Device Plugin manager"
Oct 30 18:36:17.603417 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:36:17.603331    4525 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 18:36:17.603417 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.603340    4525 server.go:87] "Starting device plugin registration server"
Oct 30 18:36:17.603629 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.603616    4525 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 18:36:17.603713 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.603631    4525 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 18:36:17.603762 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.603730    4525 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 18:36:17.603879 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.603839    4525 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 18:36:17.603915 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.603889    4525 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 18:36:17.603915 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.603896    4525 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 18:36:17.603984 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.603923    4525 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T18:36:17.603877822Z","UUID":"83c9503a-4ecc-4da8-beea-b3779d98cece","Handler":null,"Name":"","Endpoint":""}
Oct 30 18:36:17.605399 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:36:17.605378    4525 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 18:36:17.605520 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.605473    4525 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 18:36:17.605520 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.605514    4525 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 18:36:17.637337 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.637294    4525 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-135.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-135.us-east-2.compute.internal"]
Oct 30 18:36:17.637884 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.637850    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-135.us-east-2.compute.internal" event={"ID":"826c9de932b6871da80993c953a89ebf","Type":"ContainerStarted","Data":"231a010045f4737301afbc46f96f56e8e703a0728c9b69fcadfe8745163910c0"}
Oct 30 18:36:17.637923 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.637898    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-135.us-east-2.compute.internal" event={"ID":"826c9de932b6871da80993c953a89ebf","Type":"ContainerStarted","Data":"4613d69f1e8a92f7b0ccf60ce4f2bf0a1ffbe0c8347c420867cde38f8e36fdf4"}
Oct 30 18:36:17.637977 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.637921    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-135.us-east-2.compute.internal" event={"ID":"09a9d0b6ec5615a60675fe6f5b7fa6e4","Type":"ContainerStarted","Data":"83115c77c0240b5b190e823840d18122d63da32b5198af445316537bb28faa8b"}
Oct 30 18:36:17.637977 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.637936    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-135.us-east-2.compute.internal" event={"ID":"09a9d0b6ec5615a60675fe6f5b7fa6e4","Type":"ContainerDied","Data":"c0efb16a78c761c15493d05fc695c2c98e1a0d5485a24acdea8e26c6ef70f5b0"}
Oct 30 18:36:17.637977 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.637959    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-135.us-east-2.compute.internal" event={"ID":"09a9d0b6ec5615a60675fe6f5b7fa6e4","Type":"ContainerStarted","Data":"4a064f4b59347185376167aae61af87390c30621b87547a175af427d35b976fc"}
Oct 30 18:36:17.637977 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.637972    4525 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-135.us-east-2.compute.internal"
Oct 30 18:36:17.638081 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.638009    4525 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-135.us-east-2.compute.internal"
Oct 30 18:36:17.649508 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.649491    4525 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 18:36:17.649560 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:36:17.649537    4525 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-137-135.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-135.us-east-2.compute.internal"
Oct 30 18:36:17.649909 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.649895    4525 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 18:36:17.649940 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:36:17.649933    4525 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-137-135.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-135.us-east-2.compute.internal"
Oct 30 18:36:17.704096 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.704052    4525 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:36:17.707423 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.707395    4525 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-135.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:36:17.707543 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.707431    4525 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-135.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:36:17.707543 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.707442    4525 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-135.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:36:17.707543 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.707478    4525 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-137-135.us-east-2.compute.internal"
Oct 30 18:36:17.715459 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.715433    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/09a9d0b6ec5615a60675fe6f5b7fa6e4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-135.us-east-2.compute.internal\" (UID: \"09a9d0b6ec5615a60675fe6f5b7fa6e4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-135.us-east-2.compute.internal"
Oct 30 18:36:17.715553 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.715468    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/09a9d0b6ec5615a60675fe6f5b7fa6e4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-135.us-east-2.compute.internal\" (UID: \"09a9d0b6ec5615a60675fe6f5b7fa6e4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-135.us-east-2.compute.internal"
Oct 30 18:36:17.715553 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.715492    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/826c9de932b6871da80993c953a89ebf-config\") pod \"kube-apiserver-proxy-ip-10-0-137-135.us-east-2.compute.internal\" (UID: \"826c9de932b6871da80993c953a89ebf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-135.us-east-2.compute.internal"
Oct 30 18:36:17.718398 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.718384    4525 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-137-135.us-east-2.compute.internal"
Oct 30 18:36:17.718448 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.718439    4525 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-137-135.us-east-2.compute.internal"
Oct 30 18:36:17.816095 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.816012    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/09a9d0b6ec5615a60675fe6f5b7fa6e4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-135.us-east-2.compute.internal\" (UID: \"09a9d0b6ec5615a60675fe6f5b7fa6e4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-135.us-east-2.compute.internal"
Oct 30 18:36:17.816095 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.816052    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/09a9d0b6ec5615a60675fe6f5b7fa6e4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-135.us-east-2.compute.internal\" (UID: \"09a9d0b6ec5615a60675fe6f5b7fa6e4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-135.us-east-2.compute.internal"
Oct 30 18:36:17.816095 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.816068    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/826c9de932b6871da80993c953a89ebf-config\") pod \"kube-apiserver-proxy-ip-10-0-137-135.us-east-2.compute.internal\" (UID: \"826c9de932b6871da80993c953a89ebf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-135.us-east-2.compute.internal"
Oct 30 18:36:17.816317 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.816123    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/09a9d0b6ec5615a60675fe6f5b7fa6e4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-135.us-east-2.compute.internal\" (UID: \"09a9d0b6ec5615a60675fe6f5b7fa6e4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-135.us-east-2.compute.internal"
Oct 30 18:36:17.816317 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.816141    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/09a9d0b6ec5615a60675fe6f5b7fa6e4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-135.us-east-2.compute.internal\" (UID: \"09a9d0b6ec5615a60675fe6f5b7fa6e4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-135.us-east-2.compute.internal"
Oct 30 18:36:17.816317 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:17.816179    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/826c9de932b6871da80993c953a89ebf-config\") pod \"kube-apiserver-proxy-ip-10-0-137-135.us-east-2.compute.internal\" (UID: \"826c9de932b6871da80993c953a89ebf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-135.us-east-2.compute.internal"
Oct 30 18:36:18.465652 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.465615    4525 apiserver.go:52] "Watching apiserver"
Oct 30 18:36:18.476986 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.476957    4525 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 18:36:18.477425 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.477401    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-9p48q","openshift-ovn-kubernetes/ovnkube-node-kwd4p","kube-system/konnectivity-agent-dbbbq","kube-system/kube-apiserver-proxy-ip-10-0-137-135.us-east-2.compute.internal","openshift-kube-storage-version-migrator/migrator-66df5d496b-mhxzk","openshift-insights/insights-runtime-extractor-4d49l","kube-system/global-pull-secret-syncer-vhtmb","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-135.us-east-2.compute.internal","openshift-insights/periodic-gathering-q4kks-q2hfp","openshift-ingress-canary/ingress-canary-dqxm8","openshift-network-operator/iptables-alerter-rxbc9","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4pz","openshift-dns/node-resolver-7z9gn","openshift-service-ca/service-ca-5fffc97bdf-5qghx","openshift-dns/dns-default-lcst7","openshift-image-registry/node-ca-mhtvp","openshift-cluster-node-tuning-operator/tuned-f77jk","openshift-multus/multus-additional-cni-plugins-tjtmv","openshift-network-diagnostics/network-check-target-mcc4d","openshift-multus/multus-4lmrw"]
Oct 30 18:36:18.478608 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.478586    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9p48q"
Oct 30 18:36:18.481770 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.481749    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lcst7"
Oct 30 18:36:18.482195 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.482180    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4d49l"
Oct 30 18:36:18.483672 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.483646    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 18:36:18.483915 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.483897    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 18:36:18.483971 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.483939    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 18:36:18.484018 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.483994    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 18:36:18.484189 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.484171    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-89nks\""
Oct 30 18:36:18.484301 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.484284    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 18:36:18.484430 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.484333    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 18:36:18.484600 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.484581    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-xr42l\""
Oct 30 18:36:18.484650 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.484530    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 18:36:18.484713 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.484647    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 18:36:18.484713 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.484706    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 18:36:18.484804 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.484460    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lfvmh\""
Oct 30 18:36:18.485991 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.485970    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 18:36:18.486099 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.485982    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 18:36:18.486869 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.486843    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 18:36:18.486967 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.486858    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 18:36:18.486967 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.486884    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 18:36:18.486967 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.486933    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 18:36:18.487167 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.486970    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 18:36:18.487167 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.487151    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-dnnzq\""
Oct 30 18:36:18.487167 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.487161    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:36:18.487269 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.487197    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-6sn6l\""
Oct 30 18:36:18.487844 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.487825    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 18:36:18.487922 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.487829    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 18:36:18.488294 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.488280    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 18:36:18.488342 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.488294    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 18:36:18.488342 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.488325    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 18:36:18.488409 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.488351    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-5rxk7\""
Oct 30 18:36:18.488409 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.488356    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-sfbzk\""
Oct 30 18:36:18.488409 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.488404    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vgk4h\""
Oct 30 18:36:18.489142 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.489127    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 18:36:18.489185 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.489149    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qs4k8\""
Oct 30 18:36:18.489331 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.489317    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9fb5r\""
Oct 30 18:36:18.489384 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.489369    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 18:36:18.489503 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.489381    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-s4hwp\""
Oct 30 18:36:18.489503 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.489397    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 18:36:18.489503 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.489419    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 18:36:18.489503 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.489420    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-8njxf\""
Oct 30 18:36:18.489620 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.489516    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 18:36:18.490030 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.490017    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 18:36:18.490073 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.490032    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 18:36:18.490313 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.490299    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 18:36:18.490355 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.490339    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:36:18.490409 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.490342    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 18:36:18.490409 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.490361    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 18:36:18.490409 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.490389    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 18:36:18.490409 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.490393    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 18:36:18.490409 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.490394    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 18:36:18.490542 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.490425    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 18:36:18.490542 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.490439    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 18:36:18.490542 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.490486    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-phzbf\""
Oct 30 18:36:18.491333 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.491321    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 18:36:18.491371 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.491343    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 18:36:18.491371 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.491359    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 18:36:18.491424 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.491363    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-ldjxj\""
Oct 30 18:36:18.495120 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.495097    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 18:36:18.496323 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.496311    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 30 18:36:18.496425 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.496346    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 30 18:36:18.496425 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.496365    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 30 18:36:18.496425 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.496371    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 30 18:36:18.497395 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.497381    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-gjz7h\""
Oct 30 18:36:18.497442 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.497392    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-zsxst\""
Oct 30 18:36:18.497442 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.497385    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-8j87p\""
Oct 30 18:36:18.498376 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.498362    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-rqvfh\""
Oct 30 18:36:18.499856 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.499841    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 18:36:18.499981 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.499969    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 18:36:18.507867 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.507854    4525 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 18:36:18.519237 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.519220    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d8f39e97-7f68-47da-a247-adf78b523afd-multus-daemon-config\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:36:18.519335 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.519246    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d8f39e97-7f68-47da-a247-adf78b523afd-etc-kubernetes\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:36:18.519335 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.519262    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl5cp\" (UniqueName: \"kubernetes.io/projected/5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a-kube-api-access-vl5cp\") pod \"dns-default-lcst7\" (UID: \"5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a\") " pod="openshift-dns/dns-default-lcst7"
Oct 30 18:36:18.519335 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.519278    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0dc90efb-2709-4894-b457-6cadb8abd801-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tjtmv\" (UID: \"0dc90efb-2709-4894-b457-6cadb8abd801\") " pod="openshift-multus/multus-additional-cni-plugins-tjtmv"
Oct 30 18:36:18.519335 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.519295    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8bcb59a4-381f-49ba-b732-f5e4658707de-host-cni-bin\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:36:18.519512 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.519412    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/545dd3b9-01a9-40be-8762-5476a7a9ede4-lib-modules\") pod \"tuned-f77jk\" (UID: \"545dd3b9-01a9-40be-8762-5476a7a9ede4\") " pod="openshift-cluster-node-tuning-operator/tuned-f77jk"
Oct 30 18:36:18.519512 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.519436    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/545dd3b9-01a9-40be-8762-5476a7a9ede4-host\") pod \"tuned-f77jk\" (UID: \"545dd3b9-01a9-40be-8762-5476a7a9ede4\") " pod="openshift-cluster-node-tuning-operator/tuned-f77jk"
Oct 30 18:36:18.519512 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.519453    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d8f39e97-7f68-47da-a247-adf78b523afd-host-run-k8s-cni-cncf-io\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:36:18.519512 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.519476    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq9k2\" (UniqueName: \"kubernetes.io/projected/d8f39e97-7f68-47da-a247-adf78b523afd-kube-api-access-fq9k2\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:36:18.519512 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.519510    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1315feec-3a59-4cae-ad7e-8ada8c5e4788-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4d49l\" (UID: \"1315feec-3a59-4cae-ad7e-8ada8c5e4788\") " pod="openshift-insights/insights-runtime-extractor-4d49l"
Oct 30 18:36:18.519758 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.519530    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0dc90efb-2709-4894-b457-6cadb8abd801-cni-binary-copy\") pod \"multus-additional-cni-plugins-tjtmv\" (UID: \"0dc90efb-2709-4894-b457-6cadb8abd801\") " pod="openshift-multus/multus-additional-cni-plugins-tjtmv"
Oct 30 18:36:18.519758 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.519660    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8bcb59a4-381f-49ba-b732-f5e4658707de-systemd-units\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:36:18.519758 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.519672    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d8f39e97-7f68-47da-a247-adf78b523afd-multus-daemon-config\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:36:18.519758 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.519719    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0dc90efb-2709-4894-b457-6cadb8abd801-cni-binary-copy\") pod \"multus-additional-cni-plugins-tjtmv\" (UID: \"0dc90efb-2709-4894-b457-6cadb8abd801\") " pod="openshift-multus/multus-additional-cni-plugins-tjtmv"
Oct 30 18:36:18.519758 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.519717    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8bcb59a4-381f-49ba-b732-f5e4658707de-host-cni-netd\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:36:18.519758 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.519665    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0dc90efb-2709-4894-b457-6cadb8abd801-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tjtmv\" (UID: \"0dc90efb-2709-4894-b457-6cadb8abd801\") " pod="openshift-multus/multus-additional-cni-plugins-tjtmv"
Oct 30 18:36:18.519758 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.519751    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/545dd3b9-01a9-40be-8762-5476a7a9ede4-tmp\") pod \"tuned-f77jk\" (UID: \"545dd3b9-01a9-40be-8762-5476a7a9ede4\") " pod="openshift-cluster-node-tuning-operator/tuned-f77jk"
Oct 30 18:36:18.519970 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.519783    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2ca560f4-9aa8-4bc6-9db1-6992a1fa7c1f-device-dir\") pod \"aws-ebs-csi-driver-node-zb4pz\" (UID: \"2ca560f4-9aa8-4bc6-9db1-6992a1fa7c1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4pz"
Oct 30 18:36:18.519970 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.519808    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kczq\" (UniqueName: \"kubernetes.io/projected/2ca560f4-9aa8-4bc6-9db1-6992a1fa7c1f-kube-api-access-7kczq\") pod \"aws-ebs-csi-driver-node-zb4pz\" (UID: \"2ca560f4-9aa8-4bc6-9db1-6992a1fa7c1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4pz"
Oct 30 18:36:18.519970 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.519822    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/545dd3b9-01a9-40be-8762-5476a7a9ede4-tmp\") pod \"tuned-f77jk\" (UID: \"545dd3b9-01a9-40be-8762-5476a7a9ede4\") " pod="openshift-cluster-node-tuning-operator/tuned-f77jk"
Oct 30 18:36:18.519970 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.519824    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2phv\" (UniqueName: \"kubernetes.io/projected/77c2eb22-4b45-486f-8c62-bd14086942f5-kube-api-access-d2phv\") pod \"iptables-alerter-rxbc9\" (UID: \"77c2eb22-4b45-486f-8c62-bd14086942f5\") " pod="openshift-network-operator/iptables-alerter-rxbc9"
Oct 30 18:36:18.519970 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.519843    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a-config-volume\") pod \"dns-default-lcst7\" (UID: \"5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a\") " pod="openshift-dns/dns-default-lcst7"
Oct 30 18:36:18.519970 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.519859    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1315feec-3a59-4cae-ad7e-8ada8c5e4788-crio-socket\") pod \"insights-runtime-extractor-4d49l\" (UID: \"1315feec-3a59-4cae-ad7e-8ada8c5e4788\") " pod="openshift-insights/insights-runtime-extractor-4d49l"
Oct 30 18:36:18.519970 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.519873    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8bcb59a4-381f-49ba-b732-f5e4658707de-var-lib-openvswitch\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:36:18.519970 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.519889    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/42b71e2d-a8f2-43cf-8487-0e5da2338c54-hosts-file\") pod \"node-resolver-7z9gn\" (UID: \"42b71e2d-a8f2-43cf-8487-0e5da2338c54\") " pod="openshift-dns/node-resolver-7z9gn"
Oct 30 18:36:18.519970 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.519904    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/545dd3b9-01a9-40be-8762-5476a7a9ede4-etc-systemd\") pod \"tuned-f77jk\" (UID: \"545dd3b9-01a9-40be-8762-5476a7a9ede4\") " pod="openshift-cluster-node-tuning-operator/tuned-f77jk"
Oct 30 18:36:18.520357 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.519973    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7rn7\" (UniqueName: \"kubernetes.io/projected/25255277-4f3b-418c-9e85-2a82e7a13107-kube-api-access-j7rn7\") pod \"service-ca-5fffc97bdf-5qghx\" (UID: \"25255277-4f3b-418c-9e85-2a82e7a13107\") " pod="openshift-service-ca/service-ca-5fffc97bdf-5qghx"
Oct 30 18:36:18.520357 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.520023    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0dc90efb-2709-4894-b457-6cadb8abd801-cnibin\") pod \"multus-additional-cni-plugins-tjtmv\" (UID: \"0dc90efb-2709-4894-b457-6cadb8abd801\") " pod="openshift-multus/multus-additional-cni-plugins-tjtmv"
Oct 30 18:36:18.520357 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.520085    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a-config-volume\") pod \"dns-default-lcst7\" (UID: \"5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a\") " pod="openshift-dns/dns-default-lcst7"
Oct 30 18:36:18.520357 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.520116    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lbr9\" (UniqueName: \"kubernetes.io/projected/0dc90efb-2709-4894-b457-6cadb8abd801-kube-api-access-7lbr9\") pod \"multus-additional-cni-plugins-tjtmv\" (UID: \"0dc90efb-2709-4894-b457-6cadb8abd801\") " pod="openshift-multus/multus-additional-cni-plugins-tjtmv"
Oct 30 18:36:18.520357 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.520145    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8bcb59a4-381f-49ba-b732-f5e4658707de-log-socket\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:36:18.520357 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.520196    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8bcb59a4-381f-49ba-b732-f5e4658707de-ovnkube-config\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:36:18.520357 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.520229    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d8f39e97-7f68-47da-a247-adf78b523afd-hostroot\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:36:18.520357 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.520267    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7c3ae3e0-ddf1-414f-bb79-e4afc0527b26-original-pull-secret\") pod \"global-pull-secret-syncer-vhtmb\" (UID: \"7c3ae3e0-ddf1-414f-bb79-e4afc0527b26\") " pod="kube-system/global-pull-secret-syncer-vhtmb"
Oct 30 18:36:18.520357 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.520316    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/77c2eb22-4b45-486f-8c62-bd14086942f5-iptables-alerter-script\") pod \"iptables-alerter-rxbc9\" (UID: \"77c2eb22-4b45-486f-8c62-bd14086942f5\") " pod="openshift-network-operator/iptables-alerter-rxbc9"
Oct 30 18:36:18.520842 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.520397    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f32f2498-2fc8-4bc6-b368-b53d156755c5-cert\") pod \"ingress-canary-dqxm8\" (UID: \"f32f2498-2fc8-4bc6-b368-b53d156755c5\") " pod="openshift-ingress-canary/ingress-canary-dqxm8"
Oct 30 18:36:18.520842 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.520425    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/25255277-4f3b-418c-9e85-2a82e7a13107-signing-cabundle\") pod \"service-ca-5fffc97bdf-5qghx\" (UID: \"25255277-4f3b-418c-9e85-2a82e7a13107\") " pod="openshift-service-ca/service-ca-5fffc97bdf-5qghx"
Oct 30 18:36:18.520842 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.520425    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8bcb59a4-381f-49ba-b732-f5e4658707de-ovnkube-config\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:36:18.520842 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.520448    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8bcb59a4-381f-49ba-b732-f5e4658707de-host-kubelet\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:36:18.520842 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.520465    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7c3ae3e0-ddf1-414f-bb79-e4afc0527b26-original-pull-secret\") pod \"global-pull-secret-syncer-vhtmb\" (UID: \"7c3ae3e0-ddf1-414f-bb79-e4afc0527b26\") " pod="kube-system/global-pull-secret-syncer-vhtmb"
Oct 30 18:36:18.520842 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.520474    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9ce43bfa-1e19-4fb5-90ff-42751cdd1a50-archives-path\") pod \"periodic-gathering-q4kks-q2hfp\" (UID: \"9ce43bfa-1e19-4fb5-90ff-42751cdd1a50\") " pod="openshift-insights/periodic-gathering-q4kks-q2hfp"
Oct 30 18:36:18.520842 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.520508    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d8f39e97-7f68-47da-a247-adf78b523afd-multus-socket-dir-parent\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:36:18.520842 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.520541    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/77c2eb22-4b45-486f-8c62-bd14086942f5-iptables-alerter-script\") pod \"iptables-alerter-rxbc9\" (UID: \"77c2eb22-4b45-486f-8c62-bd14086942f5\") " pod="openshift-network-operator/iptables-alerter-rxbc9"
Oct 30 18:36:18.520842 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.520565    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7c3ae3e0-ddf1-414f-bb79-e4afc0527b26-kubelet-config\") pod \"global-pull-secret-syncer-vhtmb\" (UID: \"7c3ae3e0-ddf1-414f-bb79-e4afc0527b26\") " pod="kube-system/global-pull-secret-syncer-vhtmb"
Oct 30 18:36:18.520842 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.520579    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9ce43bfa-1e19-4fb5-90ff-42751cdd1a50-archives-path\") pod \"periodic-gathering-q4kks-q2hfp\" (UID: \"9ce43bfa-1e19-4fb5-90ff-42751cdd1a50\") " pod="openshift-insights/periodic-gathering-q4kks-q2hfp"
Oct 30 18:36:18.520842 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.520591    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/bd08a0fa-46bc-4250-985c-d2580a2225b8-agent-certs\") pod \"konnectivity-agent-dbbbq\" (UID: \"bd08a0fa-46bc-4250-985c-d2580a2225b8\") " pod="kube-system/konnectivity-agent-dbbbq"
Oct 30 18:36:18.520842 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.520632    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/25255277-4f3b-418c-9e85-2a82e7a13107-signing-cabundle\") pod \"service-ca-5fffc97bdf-5qghx\" (UID: \"25255277-4f3b-418c-9e85-2a82e7a13107\") " pod="openshift-service-ca/service-ca-5fffc97bdf-5qghx"
Oct 30 18:36:18.520842 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.520638    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f32f2498-2fc8-4bc6-b368-b53d156755c5-cert\") pod \"ingress-canary-dqxm8\" (UID: \"f32f2498-2fc8-4bc6-b368-b53d156755c5\") " pod="openshift-ingress-canary/ingress-canary-dqxm8"
Oct 30 18:36:18.520842 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.520628    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a-metrics-tls\") pod \"dns-default-lcst7\" (UID: \"5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a\") " pod="openshift-dns/dns-default-lcst7"
Oct 30 18:36:18.520842 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.520696    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8bcb59a4-381f-49ba-b732-f5e4658707de-ovn-node-metrics-cert\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:36:18.520842 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.520713    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/545dd3b9-01a9-40be-8762-5476a7a9ede4-etc-sysconfig\") pod \"tuned-f77jk\" (UID: \"545dd3b9-01a9-40be-8762-5476a7a9ede4\") " pod="openshift-cluster-node-tuning-operator/tuned-f77jk"
Oct 30 18:36:18.520842 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.520729    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5t65n\" (UniqueName: \"kubernetes.io/projected/97f514c8-0d31-4103-81c5-45886ff831f0-kube-api-access-5t65n\") pod \"migrator-66df5d496b-mhxzk\" (UID: \"97f514c8-0d31-4103-81c5-45886ff831f0\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-mhxzk"
Oct 30 18:36:18.520842 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.520743    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d8f39e97-7f68-47da-a247-adf78b523afd-host-var-lib-kubelet\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:36:18.520842 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.520758    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d8f39e97-7f68-47da-a247-adf78b523afd-host-run-multus-certs\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:36:18.520842 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.520787    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/093a372e-74ea-4820-82c8-0dff7c201f10-serviceca\") pod \"node-ca-mhtvp\" (UID: \"093a372e-74ea-4820-82c8-0dff7c201f10\") " pod="openshift-image-registry/node-ca-mhtvp"
Oct 30 18:36:18.520842 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.520808    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm9cn\" (UniqueName: \"kubernetes.io/projected/42b71e2d-a8f2-43cf-8487-0e5da2338c54-kube-api-access-gm9cn\") pod \"node-resolver-7z9gn\" (UID: \"42b71e2d-a8f2-43cf-8487-0e5da2338c54\") " pod="openshift-dns/node-resolver-7z9gn"
Oct 30 18:36:18.520842 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.520843    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8bcb59a4-381f-49ba-b732-f5e4658707de-ovn-node-metrics-cert\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:36:18.520842 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.520859    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/bd08a0fa-46bc-4250-985c-d2580a2225b8-agent-certs\") pod \"konnectivity-agent-dbbbq\" (UID: \"bd08a0fa-46bc-4250-985c-d2580a2225b8\") " pod="kube-system/konnectivity-agent-dbbbq"
Oct 30 18:36:18.520842 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.520876    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7c3ae3e0-ddf1-414f-bb79-e4afc0527b26-dbus\") pod \"global-pull-secret-syncer-vhtmb\" (UID: \"7c3ae3e0-ddf1-414f-bb79-e4afc0527b26\") " pod="kube-system/global-pull-secret-syncer-vhtmb"
Oct 30 18:36:18.521565 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.520955    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/77c2eb22-4b45-486f-8c62-bd14086942f5-host-slash\") pod \"iptables-alerter-rxbc9\" (UID: \"77c2eb22-4b45-486f-8c62-bd14086942f5\") " pod="openshift-network-operator/iptables-alerter-rxbc9"
Oct 30 18:36:18.521565 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.520971    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/093a372e-74ea-4820-82c8-0dff7c201f10-serviceca\") pod \"node-ca-mhtvp\" (UID: \"093a372e-74ea-4820-82c8-0dff7c201f10\") " pod="openshift-image-registry/node-ca-mhtvp"
Oct 30 18:36:18.521565 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.520993    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7jl8\" (UniqueName: \"kubernetes.io/projected/f32f2498-2fc8-4bc6-b368-b53d156755c5-kube-api-access-t7jl8\") pod \"ingress-canary-dqxm8\" (UID: \"f32f2498-2fc8-4bc6-b368-b53d156755c5\") " pod="openshift-ingress-canary/ingress-canary-dqxm8"
Oct 30 18:36:18.521565 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.521025    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8bcb59a4-381f-49ba-b732-f5e4658707de-node-log\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:36:18.521565 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.521120    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8bcb59a4-381f-49ba-b732-f5e4658707de-ovnkube-script-lib\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:36:18.521565 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.521157    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/545dd3b9-01a9-40be-8762-5476a7a9ede4-etc-sysctl-d\") pod \"tuned-f77jk\" (UID: \"545dd3b9-01a9-40be-8762-5476a7a9ede4\") " pod="openshift-cluster-node-tuning-operator/tuned-f77jk"
Oct 30 18:36:18.521565 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.521191    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8bcb59a4-381f-49ba-b732-f5e4658707de-etc-openvswitch\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:36:18.521565 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.521217    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8bcb59a4-381f-49ba-b732-f5e4658707de-env-overrides\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:36:18.521565 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.521250    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/545dd3b9-01a9-40be-8762-5476a7a9ede4-etc-kubernetes\") pod \"tuned-f77jk\" (UID: \"545dd3b9-01a9-40be-8762-5476a7a9ede4\") " pod="openshift-cluster-node-tuning-operator/tuned-f77jk"
Oct 30 18:36:18.521565 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.521287    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2ca560f4-9aa8-4bc6-9db1-6992a1fa7c1f-socket-dir\") pod \"aws-ebs-csi-driver-node-zb4pz\" (UID: \"2ca560f4-9aa8-4bc6-9db1-6992a1fa7c1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4pz"
Oct 30 18:36:18.521565 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.521313    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2ca560f4-9aa8-4bc6-9db1-6992a1fa7c1f-registration-dir\") pod \"aws-ebs-csi-driver-node-zb4pz\" (UID: \"2ca560f4-9aa8-4bc6-9db1-6992a1fa7c1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4pz"
Oct 30 18:36:18.521565 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.521314    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8bcb59a4-381f-49ba-b732-f5e4658707de-ovnkube-script-lib\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:36:18.521565 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.521333    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ce43bfa-1e19-4fb5-90ff-42751cdd1a50-service-ca-bundle\") pod \"periodic-gathering-q4kks-q2hfp\" (UID: \"9ce43bfa-1e19-4fb5-90ff-42751cdd1a50\") " pod="openshift-insights/periodic-gathering-q4kks-q2hfp"
Oct 30 18:36:18.521565 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.521353    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-862ff\" (UniqueName: \"kubernetes.io/projected/9ce43bfa-1e19-4fb5-90ff-42751cdd1a50-kube-api-access-862ff\") pod \"periodic-gathering-q4kks-q2hfp\" (UID: \"9ce43bfa-1e19-4fb5-90ff-42751cdd1a50\") " pod="openshift-insights/periodic-gathering-q4kks-q2hfp"
Oct 30 18:36:18.521565 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.521369    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8bcb59a4-381f-49ba-b732-f5e4658707de-env-overrides\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:36:18.521565 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.521435    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ce43bfa-1e19-4fb5-90ff-42751cdd1a50-service-ca-bundle\") pod \"periodic-gathering-q4kks-q2hfp\" (UID: \"9ce43bfa-1e19-4fb5-90ff-42751cdd1a50\") " pod="openshift-insights/periodic-gathering-q4kks-q2hfp"
Oct 30 18:36:18.521565 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.521476    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d8f39e97-7f68-47da-a247-adf78b523afd-host-var-lib-cni-bin\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:36:18.521565 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.521495    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/093a372e-74ea-4820-82c8-0dff7c201f10-host\") pod \"node-ca-mhtvp\" (UID: \"093a372e-74ea-4820-82c8-0dff7c201f10\") " pod="openshift-image-registry/node-ca-mhtvp"
Oct 30 18:36:18.521565 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.521528    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx6qn\" (UniqueName: \"kubernetes.io/projected/089fdd4f-daa0-4b3e-b81e-26a764a526bb-kube-api-access-sx6qn\") pod \"network-metrics-daemon-9p48q\" (UID: \"089fdd4f-daa0-4b3e-b81e-26a764a526bb\") " pod="openshift-multus/network-metrics-daemon-9p48q"
Oct 30 18:36:18.521565 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.521560    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfrpn\" (UniqueName: \"kubernetes.io/projected/4b27cd9e-69b2-4480-a84f-5b463cec1a0e-kube-api-access-rfrpn\") pod \"network-check-target-mcc4d\" (UID: \"4b27cd9e-69b2-4480-a84f-5b463cec1a0e\") " pod="openshift-network-diagnostics/network-check-target-mcc4d"
Oct 30 18:36:18.522223 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.521636    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0dc90efb-2709-4894-b457-6cadb8abd801-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tjtmv\" (UID: \"0dc90efb-2709-4894-b457-6cadb8abd801\") " pod="openshift-multus/multus-additional-cni-plugins-tjtmv"
Oct 30 18:36:18.522223 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.521655    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/545dd3b9-01a9-40be-8762-5476a7a9ede4-run\") pod \"tuned-f77jk\" (UID: \"545dd3b9-01a9-40be-8762-5476a7a9ede4\") " pod="openshift-cluster-node-tuning-operator/tuned-f77jk"
Oct 30 18:36:18.522223 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.521699    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/545dd3b9-01a9-40be-8762-5476a7a9ede4-sys\") pod \"tuned-f77jk\" (UID: \"545dd3b9-01a9-40be-8762-5476a7a9ede4\") " pod="openshift-cluster-node-tuning-operator/tuned-f77jk"
Oct 30 18:36:18.522223 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.521726    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/545dd3b9-01a9-40be-8762-5476a7a9ede4-var-lib-kubelet\") pod \"tuned-f77jk\" (UID: \"545dd3b9-01a9-40be-8762-5476a7a9ede4\") " pod="openshift-cluster-node-tuning-operator/tuned-f77jk"
Oct 30 18:36:18.522223 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.521761    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2ca560f4-9aa8-4bc6-9db1-6992a1fa7c1f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zb4pz\" (UID: \"2ca560f4-9aa8-4bc6-9db1-6992a1fa7c1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4pz"
Oct 30 18:36:18.522223 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.521787    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8bcb59a4-381f-49ba-b732-f5e4658707de-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:36:18.522223 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.521812    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1315feec-3a59-4cae-ad7e-8ada8c5e4788-data-volume\") pod \"insights-runtime-extractor-4d49l\" (UID: \"1315feec-3a59-4cae-ad7e-8ada8c5e4788\") " pod="openshift-insights/insights-runtime-extractor-4d49l"
Oct 30 18:36:18.522223 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.521830    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/bd08a0fa-46bc-4250-985c-d2580a2225b8-konnectivity-ca\") pod \"konnectivity-agent-dbbbq\" (UID: \"bd08a0fa-46bc-4250-985c-d2580a2225b8\") " pod="kube-system/konnectivity-agent-dbbbq"
Oct 30 18:36:18.522223 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.521852    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp8kg\" (UniqueName: \"kubernetes.io/projected/1315feec-3a59-4cae-ad7e-8ada8c5e4788-kube-api-access-tp8kg\") pod \"insights-runtime-extractor-4d49l\" (UID: \"1315feec-3a59-4cae-ad7e-8ada8c5e4788\") " pod="openshift-insights/insights-runtime-extractor-4d49l"
Oct 30 18:36:18.522223 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.521884    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0dc90efb-2709-4894-b457-6cadb8abd801-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tjtmv\" (UID: \"0dc90efb-2709-4894-b457-6cadb8abd801\") " pod="openshift-multus/multus-additional-cni-plugins-tjtmv"
Oct 30 18:36:18.522223 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.521913    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8bcb59a4-381f-49ba-b732-f5e4658707de-run-openvswitch\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:36:18.522223 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.521938    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8bcb59a4-381f-49ba-b732-f5e4658707de-run-ovn\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:36:18.522223 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.521978    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/25255277-4f3b-418c-9e85-2a82e7a13107-signing-key\") pod \"service-ca-5fffc97bdf-5qghx\" (UID: \"25255277-4f3b-418c-9e85-2a82e7a13107\") " pod="openshift-service-ca/service-ca-5fffc97bdf-5qghx"
Oct 30 18:36:18.522223 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.521990    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1315feec-3a59-4cae-ad7e-8ada8c5e4788-data-volume\") pod \"insights-runtime-extractor-4d49l\" (UID: \"1315feec-3a59-4cae-ad7e-8ada8c5e4788\") " pod="openshift-insights/insights-runtime-extractor-4d49l"
Oct 30 18:36:18.522223 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.521992    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0dc90efb-2709-4894-b457-6cadb8abd801-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tjtmv\" (UID: \"0dc90efb-2709-4894-b457-6cadb8abd801\") " pod="openshift-multus/multus-additional-cni-plugins-tjtmv"
Oct 30 18:36:18.522223 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.522027    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8bcb59a4-381f-49ba-b732-f5e4658707de-host-slash\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:36:18.522223 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.522089    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/bd08a0fa-46bc-4250-985c-d2580a2225b8-konnectivity-ca\") pod \"konnectivity-agent-dbbbq\" (UID: \"bd08a0fa-46bc-4250-985c-d2580a2225b8\") " pod="kube-system/konnectivity-agent-dbbbq"
Oct 30 18:36:18.522223 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.522115    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8bcb59a4-381f-49ba-b732-f5e4658707de-run-systemd\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:36:18.522223 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.522158    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8bcb59a4-381f-49ba-b732-f5e4658707de-host-run-ovn-kubernetes\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:36:18.522223 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.522186    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/25255277-4f3b-418c-9e85-2a82e7a13107-signing-key\") pod \"service-ca-5fffc97bdf-5qghx\" (UID: \"25255277-4f3b-418c-9e85-2a82e7a13107\") " pod="openshift-service-ca/service-ca-5fffc97bdf-5qghx"
Oct 30 18:36:18.522223 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.522193    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dltbj\" (UniqueName: \"kubernetes.io/projected/8bcb59a4-381f-49ba-b732-f5e4658707de-kube-api-access-dltbj\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:36:18.522223 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.522218    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/545dd3b9-01a9-40be-8762-5476a7a9ede4-etc-modprobe-d\") pod \"tuned-f77jk\" (UID: \"545dd3b9-01a9-40be-8762-5476a7a9ede4\") " pod="openshift-cluster-node-tuning-operator/tuned-f77jk"
Oct 30 18:36:18.523108 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.522248    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq2h8\" (UniqueName: \"kubernetes.io/projected/545dd3b9-01a9-40be-8762-5476a7a9ede4-kube-api-access-cq2h8\") pod \"tuned-f77jk\" (UID: \"545dd3b9-01a9-40be-8762-5476a7a9ede4\") " pod="openshift-cluster-node-tuning-operator/tuned-f77jk"
Oct 30 18:36:18.523108 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.522267    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0dc90efb-2709-4894-b457-6cadb8abd801-system-cni-dir\") pod \"multus-additional-cni-plugins-tjtmv\" (UID: \"0dc90efb-2709-4894-b457-6cadb8abd801\") " pod="openshift-multus/multus-additional-cni-plugins-tjtmv"
Oct 30 18:36:18.523108 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.522314    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/545dd3b9-01a9-40be-8762-5476a7a9ede4-etc-tuned\") pod \"tuned-f77jk\" (UID: \"545dd3b9-01a9-40be-8762-5476a7a9ede4\") " pod="openshift-cluster-node-tuning-operator/tuned-f77jk"
Oct 30 18:36:18.523108 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.522347    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2ca560f4-9aa8-4bc6-9db1-6992a1fa7c1f-etc-selinux\") pod \"aws-ebs-csi-driver-node-zb4pz\" (UID: \"2ca560f4-9aa8-4bc6-9db1-6992a1fa7c1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4pz"
Oct 30 18:36:18.523108 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.522374    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/545dd3b9-01a9-40be-8762-5476a7a9ede4-etc-tuned\") pod \"tuned-f77jk\" (UID: \"545dd3b9-01a9-40be-8762-5476a7a9ede4\") " pod="openshift-cluster-node-tuning-operator/tuned-f77jk"
Oct 30 18:36:18.523108 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.522381    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d8f39e97-7f68-47da-a247-adf78b523afd-multus-cni-dir\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:36:18.523108 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.522426    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d8f39e97-7f68-47da-a247-adf78b523afd-cni-binary-copy\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:36:18.523108 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.522468    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d8f39e97-7f68-47da-a247-adf78b523afd-host-run-netns\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:36:18.523108 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.522505    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d8f39e97-7f68-47da-a247-adf78b523afd-multus-conf-dir\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:36:18.523108 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.522532    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42b71e2d-a8f2-43cf-8487-0e5da2338c54-tmp-dir\") pod \"node-resolver-7z9gn\" (UID: \"42b71e2d-a8f2-43cf-8487-0e5da2338c54\") " pod="openshift-dns/node-resolver-7z9gn"
Oct 30 18:36:18.523108 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.522555    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/089fdd4f-daa0-4b3e-b81e-26a764a526bb-metrics-certs\") pod \"network-metrics-daemon-9p48q\" (UID: \"089fdd4f-daa0-4b3e-b81e-26a764a526bb\") " pod="openshift-multus/network-metrics-daemon-9p48q"
Oct 30 18:36:18.523108 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.522590    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d8f39e97-7f68-47da-a247-adf78b523afd-cni-binary-copy\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:36:18.523108 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.522593    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a-tmp-dir\") pod \"dns-default-lcst7\" (UID: \"5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a\") " pod="openshift-dns/dns-default-lcst7"
Oct 30 18:36:18.523108 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.522630    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42b71e2d-a8f2-43cf-8487-0e5da2338c54-tmp-dir\") pod \"node-resolver-7z9gn\" (UID: \"42b71e2d-a8f2-43cf-8487-0e5da2338c54\") " pod="openshift-dns/node-resolver-7z9gn"
Oct 30 18:36:18.523108 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.522667    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1315feec-3a59-4cae-ad7e-8ada8c5e4788-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4d49l\" (UID: \"1315feec-3a59-4cae-ad7e-8ada8c5e4788\") " pod="openshift-insights/insights-runtime-extractor-4d49l"
Oct 30 18:36:18.523108 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.522717    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a-tmp-dir\") pod \"dns-default-lcst7\" (UID: \"5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a\") " pod="openshift-dns/dns-default-lcst7"
Oct 30 18:36:18.523108 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.522745    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2ca560f4-9aa8-4bc6-9db1-6992a1fa7c1f-sys-fs\") pod \"aws-ebs-csi-driver-node-zb4pz\" (UID: \"2ca560f4-9aa8-4bc6-9db1-6992a1fa7c1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4pz"
Oct 30 18:36:18.523108 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.522763    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ce43bfa-1e19-4fb5-90ff-42751cdd1a50-trusted-ca-bundle\") pod \"periodic-gathering-q4kks-q2hfp\" (UID: \"9ce43bfa-1e19-4fb5-90ff-42751cdd1a50\") " pod="openshift-insights/periodic-gathering-q4kks-q2hfp"
Oct 30 18:36:18.523108 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.522796    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d8f39e97-7f68-47da-a247-adf78b523afd-system-cni-dir\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:36:18.523108 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.522842    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plzcb\" (UniqueName: \"kubernetes.io/projected/093a372e-74ea-4820-82c8-0dff7c201f10-kube-api-access-plzcb\") pod \"node-ca-mhtvp\" (UID: \"093a372e-74ea-4820-82c8-0dff7c201f10\") " pod="openshift-image-registry/node-ca-mhtvp"
Oct 30 18:36:18.523108 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.522886    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8bcb59a4-381f-49ba-b732-f5e4658707de-host-run-netns\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:36:18.523108 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.522902    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/545dd3b9-01a9-40be-8762-5476a7a9ede4-etc-sysctl-conf\") pod \"tuned-f77jk\" (UID: \"545dd3b9-01a9-40be-8762-5476a7a9ede4\") " pod="openshift-cluster-node-tuning-operator/tuned-f77jk"
Oct 30 18:36:18.523108 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.522917    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d8f39e97-7f68-47da-a247-adf78b523afd-cnibin\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:36:18.523108 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.522947    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d8f39e97-7f68-47da-a247-adf78b523afd-host-var-lib-cni-multus\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:36:18.523108 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.523038    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ce43bfa-1e19-4fb5-90ff-42751cdd1a50-trusted-ca-bundle\") pod \"periodic-gathering-q4kks-q2hfp\" (UID: \"9ce43bfa-1e19-4fb5-90ff-42751cdd1a50\") " pod="openshift-insights/periodic-gathering-q4kks-q2hfp"
Oct 30 18:36:18.533528 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.533511    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vl5cp\" (UniqueName: \"kubernetes.io/projected/5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a-kube-api-access-vl5cp\") pod \"dns-default-lcst7\" (UID: \"5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a\") " pod="openshift-dns/dns-default-lcst7"
Oct 30 18:36:18.535490 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.535461    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t65n\" (UniqueName: \"kubernetes.io/projected/97f514c8-0d31-4103-81c5-45886ff831f0-kube-api-access-5t65n\") pod \"migrator-66df5d496b-mhxzk\" (UID: \"97f514c8-0d31-4103-81c5-45886ff831f0\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-mhxzk"
Oct 30 18:36:18.535589 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.535558    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq9k2\" (UniqueName: \"kubernetes.io/projected/d8f39e97-7f68-47da-a247-adf78b523afd-kube-api-access-fq9k2\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:36:18.535658 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.535602    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm9cn\" (UniqueName: \"kubernetes.io/projected/42b71e2d-a8f2-43cf-8487-0e5da2338c54-kube-api-access-gm9cn\") pod \"node-resolver-7z9gn\" (UID: \"42b71e2d-a8f2-43cf-8487-0e5da2338c54\") " pod="openshift-dns/node-resolver-7z9gn"
Oct 30 18:36:18.535798 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.535778    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq2h8\" (UniqueName: \"kubernetes.io/projected/545dd3b9-01a9-40be-8762-5476a7a9ede4-kube-api-access-cq2h8\") pod \"tuned-f77jk\" (UID: \"545dd3b9-01a9-40be-8762-5476a7a9ede4\") " pod="openshift-cluster-node-tuning-operator/tuned-f77jk"
Oct 30 18:36:18.535947 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.535932    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx6qn\" (UniqueName: \"kubernetes.io/projected/089fdd4f-daa0-4b3e-b81e-26a764a526bb-kube-api-access-sx6qn\") pod \"network-metrics-daemon-9p48q\" (UID: \"089fdd4f-daa0-4b3e-b81e-26a764a526bb\") " pod="openshift-multus/network-metrics-daemon-9p48q"
Oct 30 18:36:18.536161 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.536141    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfrpn\" (UniqueName: \"kubernetes.io/projected/4b27cd9e-69b2-4480-a84f-5b463cec1a0e-kube-api-access-rfrpn\") pod \"network-check-target-mcc4d\" (UID: \"4b27cd9e-69b2-4480-a84f-5b463cec1a0e\") " pod="openshift-network-diagnostics/network-check-target-mcc4d"
Oct 30 18:36:18.536273 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.536257    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lbr9\" (UniqueName: \"kubernetes.io/projected/0dc90efb-2709-4894-b457-6cadb8abd801-kube-api-access-7lbr9\") pod \"multus-additional-cni-plugins-tjtmv\" (UID: \"0dc90efb-2709-4894-b457-6cadb8abd801\") " pod="openshift-multus/multus-additional-cni-plugins-tjtmv"
Oct 30 18:36:18.536765 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.536748    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dltbj\" (UniqueName: \"kubernetes.io/projected/8bcb59a4-381f-49ba-b732-f5e4658707de-kube-api-access-dltbj\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:36:18.536843 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.536827    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-862ff\" (UniqueName: \"kubernetes.io/projected/9ce43bfa-1e19-4fb5-90ff-42751cdd1a50-kube-api-access-862ff\") pod \"periodic-gathering-q4kks-q2hfp\" (UID: \"9ce43bfa-1e19-4fb5-90ff-42751cdd1a50\") " pod="openshift-insights/periodic-gathering-q4kks-q2hfp"
Oct 30 18:36:18.537396 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.537380    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kczq\" (UniqueName: \"kubernetes.io/projected/2ca560f4-9aa8-4bc6-9db1-6992a1fa7c1f-kube-api-access-7kczq\") pod \"aws-ebs-csi-driver-node-zb4pz\" (UID: \"2ca560f4-9aa8-4bc6-9db1-6992a1fa7c1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4pz"
Oct 30 18:36:18.537899 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.537884    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7rn7\" (UniqueName: \"kubernetes.io/projected/25255277-4f3b-418c-9e85-2a82e7a13107-kube-api-access-j7rn7\") pod \"service-ca-5fffc97bdf-5qghx\" (UID: \"25255277-4f3b-418c-9e85-2a82e7a13107\") " pod="openshift-service-ca/service-ca-5fffc97bdf-5qghx"
Oct 30 18:36:18.537954 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.537897    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp8kg\" (UniqueName: \"kubernetes.io/projected/1315feec-3a59-4cae-ad7e-8ada8c5e4788-kube-api-access-tp8kg\") pod \"insights-runtime-extractor-4d49l\" (UID: \"1315feec-3a59-4cae-ad7e-8ada8c5e4788\") " pod="openshift-insights/insights-runtime-extractor-4d49l"
Oct 30 18:36:18.537988 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.537977    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plzcb\" (UniqueName: \"kubernetes.io/projected/093a372e-74ea-4820-82c8-0dff7c201f10-kube-api-access-plzcb\") pod \"node-ca-mhtvp\" (UID: \"093a372e-74ea-4820-82c8-0dff7c201f10\") " pod="openshift-image-registry/node-ca-mhtvp"
Oct 30 18:36:18.538016 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.537989    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2phv\" (UniqueName: \"kubernetes.io/projected/77c2eb22-4b45-486f-8c62-bd14086942f5-kube-api-access-d2phv\") pod \"iptables-alerter-rxbc9\" (UID: \"77c2eb22-4b45-486f-8c62-bd14086942f5\") " pod="openshift-network-operator/iptables-alerter-rxbc9"
Oct 30 18:36:18.538539 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.538523    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7jl8\" (UniqueName: \"kubernetes.io/projected/f32f2498-2fc8-4bc6-b368-b53d156755c5-kube-api-access-t7jl8\") pod \"ingress-canary-dqxm8\" (UID: \"f32f2498-2fc8-4bc6-b368-b53d156755c5\") " pod="openshift-ingress-canary/ingress-canary-dqxm8"
Oct 30 18:36:18.567160 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.567113    4525 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbbbq_bd08a0fa-46bc-4250-985c-d2580a2225b8/konnectivity-agent/0.log" limit=1024
Oct 30 18:36:18.573271 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.573252    4525 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-135.us-east-2.compute.internal"
Oct 30 18:36:18.573379 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.573367    4525 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-135.us-east-2.compute.internal"
Oct 30 18:36:18.584926 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.584899    4525 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 18:36:18.585017 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:36:18.584943    4525 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-137-135.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-135.us-east-2.compute.internal"
Oct 30 18:36:18.586760 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.586025    4525 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 18:36:18.586760 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:36:18.586083    4525 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-137-135.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-135.us-east-2.compute.internal"
Oct 30 18:36:18.623611 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.623578    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8bcb59a4-381f-49ba-b732-f5e4658707de-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:36:18.623759 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.623616    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0dc90efb-2709-4894-b457-6cadb8abd801-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tjtmv\" (UID: \"0dc90efb-2709-4894-b457-6cadb8abd801\") " pod="openshift-multus/multus-additional-cni-plugins-tjtmv"
Oct 30 18:36:18.623759 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.623641    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8bcb59a4-381f-49ba-b732-f5e4658707de-run-openvswitch\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:36:18.623759 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.623661    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8bcb59a4-381f-49ba-b732-f5e4658707de-run-ovn\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:36:18.623759 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.623705    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8bcb59a4-381f-49ba-b732-f5e4658707de-host-slash\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:36:18.623759 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.623712    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8bcb59a4-381f-49ba-b732-f5e4658707de-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:36:18.623759 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.623727    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8bcb59a4-381f-49ba-b732-f5e4658707de-run-systemd\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:36:18.623759 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.623732    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8bcb59a4-381f-49ba-b732-f5e4658707de-run-openvswitch\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:36:18.623759 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.623755    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0dc90efb-2709-4894-b457-6cadb8abd801-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tjtmv\" (UID: \"0dc90efb-2709-4894-b457-6cadb8abd801\") " pod="openshift-multus/multus-additional-cni-plugins-tjtmv"
Oct 30 18:36:18.624101 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.623760    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8bcb59a4-381f-49ba-b732-f5e4658707de-host-run-ovn-kubernetes\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:36:18.624101 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.623763    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8bcb59a4-381f-49ba-b732-f5e4658707de-run-systemd\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:36:18.624101 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.623785    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8bcb59a4-381f-49ba-b732-f5e4658707de-run-ovn\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:36:18.624101 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.623802    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8bcb59a4-381f-49ba-b732-f5e4658707de-host-run-ovn-kubernetes\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:36:18.624101 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.623822    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8bcb59a4-381f-49ba-b732-f5e4658707de-host-slash\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:36:18.624101 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.623866    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/545dd3b9-01a9-40be-8762-5476a7a9ede4-etc-modprobe-d\") pod \"tuned-f77jk\" (UID: \"545dd3b9-01a9-40be-8762-5476a7a9ede4\") " pod="openshift-cluster-node-tuning-operator/tuned-f77jk"
Oct 30 18:36:18.624101 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.623897    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0dc90efb-2709-4894-b457-6cadb8abd801-system-cni-dir\") pod \"multus-additional-cni-plugins-tjtmv\" (UID: \"0dc90efb-2709-4894-b457-6cadb8abd801\") " pod="openshift-multus/multus-additional-cni-plugins-tjtmv"
Oct 30 18:36:18.624101 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.623920    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2ca560f4-9aa8-4bc6-9db1-6992a1fa7c1f-etc-selinux\") pod \"aws-ebs-csi-driver-node-zb4pz\" (UID: \"2ca560f4-9aa8-4bc6-9db1-6992a1fa7c1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4pz"
Oct 30 18:36:18.624101 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.623935    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d8f39e97-7f68-47da-a247-adf78b523afd-multus-cni-dir\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:36:18.624101 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.623946    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0dc90efb-2709-4894-b457-6cadb8abd801-system-cni-dir\") pod \"multus-additional-cni-plugins-tjtmv\" (UID: \"0dc90efb-2709-4894-b457-6cadb8abd801\") " pod="openshift-multus/multus-additional-cni-plugins-tjtmv"
Oct 30 18:36:18.624101 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.623949    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d8f39e97-7f68-47da-a247-adf78b523afd-host-run-netns\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:36:18.624101 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.623971    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d8f39e97-7f68-47da-a247-adf78b523afd-host-run-netns\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:36:18.624101 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.623974    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d8f39e97-7f68-47da-a247-adf78b523afd-multus-conf-dir\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:36:18.624101 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.623991    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/089fdd4f-daa0-4b3e-b81e-26a764a526bb-metrics-certs\") pod \"network-metrics-daemon-9p48q\" (UID: \"089fdd4f-daa0-4b3e-b81e-26a764a526bb\") " pod="openshift-multus/network-metrics-daemon-9p48q"
Oct 30 18:36:18.624101 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.624013    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d8f39e97-7f68-47da-a247-adf78b523afd-multus-cni-dir\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:36:18.624101 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.624010    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1315feec-3a59-4cae-ad7e-8ada8c5e4788-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4d49l\" (UID: \"1315feec-3a59-4cae-ad7e-8ada8c5e4788\") " pod="openshift-insights/insights-runtime-extractor-4d49l"
Oct 30 18:36:18.624101 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.624034    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2ca560f4-9aa8-4bc6-9db1-6992a1fa7c1f-sys-fs\") pod \"aws-ebs-csi-driver-node-zb4pz\" (UID: \"2ca560f4-9aa8-4bc6-9db1-6992a1fa7c1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4pz"
Oct 30 18:36:18.624101 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.624047    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d8f39e97-7f68-47da-a247-adf78b523afd-multus-conf-dir\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:36:18.624101 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.624012    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/545dd3b9-01a9-40be-8762-5476a7a9ede4-etc-modprobe-d\") pod \"tuned-f77jk\" (UID: \"545dd3b9-01a9-40be-8762-5476a7a9ede4\") " pod="openshift-cluster-node-tuning-operator/tuned-f77jk"
Oct 30 18:36:18.624101 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.624012    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2ca560f4-9aa8-4bc6-9db1-6992a1fa7c1f-etc-selinux\") pod \"aws-ebs-csi-driver-node-zb4pz\" (UID: \"2ca560f4-9aa8-4bc6-9db1-6992a1fa7c1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4pz"
Oct 30 18:36:18.624101 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.624049    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d8f39e97-7f68-47da-a247-adf78b523afd-system-cni-dir\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:36:18.624101 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.624082    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d8f39e97-7f68-47da-a247-adf78b523afd-system-cni-dir\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:36:18.624101 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:36:18.624100    4525 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 18:36:18.624101 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:36:18.624102    4525 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 18:36:18.625149 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.624104    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2ca560f4-9aa8-4bc6-9db1-6992a1fa7c1f-sys-fs\") pod \"aws-ebs-csi-driver-node-zb4pz\" (UID: \"2ca560f4-9aa8-4bc6-9db1-6992a1fa7c1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4pz"
Oct 30 18:36:18.625149 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.624137    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8bcb59a4-381f-49ba-b732-f5e4658707de-host-run-netns\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:36:18.625149 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.624107    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8bcb59a4-381f-49ba-b732-f5e4658707de-host-run-netns\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:36:18.625149 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:36:18.624186    4525 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/089fdd4f-daa0-4b3e-b81e-26a764a526bb-metrics-certs podName:089fdd4f-daa0-4b3e-b81e-26a764a526bb nodeName:}" failed. No retries permitted until 2026-10-30 18:36:19.124152875 +0000 UTC m=+1.749822852 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/089fdd4f-daa0-4b3e-b81e-26a764a526bb-metrics-certs") pod "network-metrics-daemon-9p48q" (UID: "089fdd4f-daa0-4b3e-b81e-26a764a526bb") : secret "metrics-daemon-secret" not found
Oct 30 18:36:18.625149 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:36:18.624229    4525 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1315feec-3a59-4cae-ad7e-8ada8c5e4788-insights-runtime-extractor-tls podName:1315feec-3a59-4cae-ad7e-8ada8c5e4788 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:19.124211334 +0000 UTC m=+1.749881283 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1315feec-3a59-4cae-ad7e-8ada8c5e4788-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4d49l" (UID: "1315feec-3a59-4cae-ad7e-8ada8c5e4788") : secret "insights-runtime-extractor-tls" not found
Oct 30 18:36:18.625149 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.624246    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/545dd3b9-01a9-40be-8762-5476a7a9ede4-etc-sysctl-conf\") pod \"tuned-f77jk\" (UID: \"545dd3b9-01a9-40be-8762-5476a7a9ede4\") " pod="openshift-cluster-node-tuning-operator/tuned-f77jk"
Oct 30 18:36:18.625149 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.624270    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d8f39e97-7f68-47da-a247-adf78b523afd-cnibin\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:36:18.625149 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.624297    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d8f39e97-7f68-47da-a247-adf78b523afd-host-var-lib-cni-multus\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:36:18.625149 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.624307    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d8f39e97-7f68-47da-a247-adf78b523afd-cnibin\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:36:18.625149 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.624321    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d8f39e97-7f68-47da-a247-adf78b523afd-etc-kubernetes\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:36:18.625149 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.624348    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d8f39e97-7f68-47da-a247-adf78b523afd-host-var-lib-cni-multus\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:36:18.625149 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.624348    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8bcb59a4-381f-49ba-b732-f5e4658707de-host-cni-bin\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:36:18.625149 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.624367    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/545dd3b9-01a9-40be-8762-5476a7a9ede4-etc-sysctl-conf\") pod \"tuned-f77jk\" (UID: \"545dd3b9-01a9-40be-8762-5476a7a9ede4\") " pod="openshift-cluster-node-tuning-operator/tuned-f77jk"
Oct 30 18:36:18.625149 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.624377    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/545dd3b9-01a9-40be-8762-5476a7a9ede4-lib-modules\") pod \"tuned-f77jk\" (UID: \"545dd3b9-01a9-40be-8762-5476a7a9ede4\") " pod="openshift-cluster-node-tuning-operator/tuned-f77jk"
Oct 30 18:36:18.625149 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.624381    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8bcb59a4-381f-49ba-b732-f5e4658707de-host-cni-bin\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:36:18.625149 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.624388    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d8f39e97-7f68-47da-a247-adf78b523afd-etc-kubernetes\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:36:18.625149 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.624401    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/545dd3b9-01a9-40be-8762-5476a7a9ede4-host\") pod \"tuned-f77jk\" (UID: \"545dd3b9-01a9-40be-8762-5476a7a9ede4\") " pod="openshift-cluster-node-tuning-operator/tuned-f77jk"
Oct 30 18:36:18.625149 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.624417    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d8f39e97-7f68-47da-a247-adf78b523afd-host-run-k8s-cni-cncf-io\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:36:18.625149 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.624433    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1315feec-3a59-4cae-ad7e-8ada8c5e4788-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4d49l\" (UID: \"1315feec-3a59-4cae-ad7e-8ada8c5e4788\") " pod="openshift-insights/insights-runtime-extractor-4d49l"
Oct 30 18:36:18.625149 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.624447    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8bcb59a4-381f-49ba-b732-f5e4658707de-systemd-units\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:36:18.625149 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.624454    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/545dd3b9-01a9-40be-8762-5476a7a9ede4-lib-modules\") pod \"tuned-f77jk\" (UID: \"545dd3b9-01a9-40be-8762-5476a7a9ede4\") " pod="openshift-cluster-node-tuning-operator/tuned-f77jk"
Oct 30 18:36:18.625149 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.624468    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8bcb59a4-381f-49ba-b732-f5e4658707de-host-cni-netd\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:36:18.625149 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.624489    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/545dd3b9-01a9-40be-8762-5476a7a9ede4-host\") pod \"tuned-f77jk\" (UID: \"545dd3b9-01a9-40be-8762-5476a7a9ede4\") " pod="openshift-cluster-node-tuning-operator/tuned-f77jk"
Oct 30 18:36:18.625149 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.624491    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8bcb59a4-381f-49ba-b732-f5e4658707de-systemd-units\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:36:18.625149 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.624492    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2ca560f4-9aa8-4bc6-9db1-6992a1fa7c1f-device-dir\") pod \"aws-ebs-csi-driver-node-zb4pz\" (UID: \"2ca560f4-9aa8-4bc6-9db1-6992a1fa7c1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4pz"
Oct 30 18:36:18.625149 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.624519    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d8f39e97-7f68-47da-a247-adf78b523afd-host-run-k8s-cni-cncf-io\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:36:18.625149 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.624524    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8bcb59a4-381f-49ba-b732-f5e4658707de-host-cni-netd\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:36:18.625149 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.624529    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1315feec-3a59-4cae-ad7e-8ada8c5e4788-crio-socket\") pod \"insights-runtime-extractor-4d49l\" (UID: \"1315feec-3a59-4cae-ad7e-8ada8c5e4788\") " pod="openshift-insights/insights-runtime-extractor-4d49l"
Oct 30 18:36:18.625149 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.624553    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8bcb59a4-381f-49ba-b732-f5e4658707de-var-lib-openvswitch\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:36:18.625149 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.624561    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2ca560f4-9aa8-4bc6-9db1-6992a1fa7c1f-device-dir\") pod \"aws-ebs-csi-driver-node-zb4pz\" (UID: \"2ca560f4-9aa8-4bc6-9db1-6992a1fa7c1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4pz"
Oct 30 18:36:18.625149 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.624569    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/42b71e2d-a8f2-43cf-8487-0e5da2338c54-hosts-file\") pod \"node-resolver-7z9gn\" (UID: \"42b71e2d-a8f2-43cf-8487-0e5da2338c54\") " pod="openshift-dns/node-resolver-7z9gn"
Oct 30 18:36:18.625149 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.624576    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1315feec-3a59-4cae-ad7e-8ada8c5e4788-crio-socket\") pod \"insights-runtime-extractor-4d49l\" (UID: \"1315feec-3a59-4cae-ad7e-8ada8c5e4788\") " pod="openshift-insights/insights-runtime-extractor-4d49l"
Oct 30 18:36:18.625149 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.624583    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/545dd3b9-01a9-40be-8762-5476a7a9ede4-etc-systemd\") pod \"tuned-f77jk\" (UID: \"545dd3b9-01a9-40be-8762-5476a7a9ede4\") " pod="openshift-cluster-node-tuning-operator/tuned-f77jk"
Oct 30 18:36:18.625149 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.624601    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8bcb59a4-381f-49ba-b732-f5e4658707de-var-lib-openvswitch\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:36:18.625149 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.624610    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/545dd3b9-01a9-40be-8762-5476a7a9ede4-etc-systemd\") pod \"tuned-f77jk\" (UID: \"545dd3b9-01a9-40be-8762-5476a7a9ede4\") " pod="openshift-cluster-node-tuning-operator/tuned-f77jk"
Oct 30 18:36:18.625149 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.624610    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0dc90efb-2709-4894-b457-6cadb8abd801-cnibin\") pod \"multus-additional-cni-plugins-tjtmv\" (UID: \"0dc90efb-2709-4894-b457-6cadb8abd801\") " pod="openshift-multus/multus-additional-cni-plugins-tjtmv"
Oct 30 18:36:18.625149 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.624635    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0dc90efb-2709-4894-b457-6cadb8abd801-cnibin\") pod \"multus-additional-cni-plugins-tjtmv\" (UID: \"0dc90efb-2709-4894-b457-6cadb8abd801\") " pod="openshift-multus/multus-additional-cni-plugins-tjtmv"
Oct 30 18:36:18.625149 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.624636    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8bcb59a4-381f-49ba-b732-f5e4658707de-log-socket\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:36:18.625149 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.624653    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8bcb59a4-381f-49ba-b732-f5e4658707de-log-socket\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:36:18.625149 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.624659    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d8f39e97-7f68-47da-a247-adf78b523afd-hostroot\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:36:18.625149 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.624702    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8bcb59a4-381f-49ba-b732-f5e4658707de-host-kubelet\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:36:18.625149 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.624717    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d8f39e97-7f68-47da-a247-adf78b523afd-hostroot\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:36:18.625149 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.624716    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/42b71e2d-a8f2-43cf-8487-0e5da2338c54-hosts-file\") pod \"node-resolver-7z9gn\" (UID: \"42b71e2d-a8f2-43cf-8487-0e5da2338c54\") " pod="openshift-dns/node-resolver-7z9gn"
Oct 30 18:36:18.625149 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.624730    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d8f39e97-7f68-47da-a247-adf78b523afd-multus-socket-dir-parent\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:36:18.625149 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.624762    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8bcb59a4-381f-49ba-b732-f5e4658707de-host-kubelet\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:36:18.625149 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.624767    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7c3ae3e0-ddf1-414f-bb79-e4afc0527b26-kubelet-config\") pod \"global-pull-secret-syncer-vhtmb\" (UID: \"7c3ae3e0-ddf1-414f-bb79-e4afc0527b26\") " pod="kube-system/global-pull-secret-syncer-vhtmb"
Oct 30 18:36:18.625149 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.624768    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d8f39e97-7f68-47da-a247-adf78b523afd-multus-socket-dir-parent\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:36:18.625149 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.624793    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a-metrics-tls\") pod \"dns-default-lcst7\" (UID: \"5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a\") " pod="openshift-dns/dns-default-lcst7"
Oct 30 18:36:18.625149 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.624810    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/545dd3b9-01a9-40be-8762-5476a7a9ede4-etc-sysconfig\") pod \"tuned-f77jk\" (UID: \"545dd3b9-01a9-40be-8762-5476a7a9ede4\") " pod="openshift-cluster-node-tuning-operator/tuned-f77jk"
Oct 30 18:36:18.625149 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.624825    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d8f39e97-7f68-47da-a247-adf78b523afd-host-var-lib-kubelet\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:36:18.625149 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.624794    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7c3ae3e0-ddf1-414f-bb79-e4afc0527b26-kubelet-config\") pod \"global-pull-secret-syncer-vhtmb\" (UID: \"7c3ae3e0-ddf1-414f-bb79-e4afc0527b26\") " pod="kube-system/global-pull-secret-syncer-vhtmb"
Oct 30 18:36:18.625149 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.624844    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d8f39e97-7f68-47da-a247-adf78b523afd-host-run-multus-certs\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:36:18.625149 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.624868    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d8f39e97-7f68-47da-a247-adf78b523afd-host-var-lib-kubelet\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:36:18.625149 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.624869    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7c3ae3e0-ddf1-414f-bb79-e4afc0527b26-dbus\") pod \"global-pull-secret-syncer-vhtmb\" (UID: \"7c3ae3e0-ddf1-414f-bb79-e4afc0527b26\") " pod="kube-system/global-pull-secret-syncer-vhtmb"
Oct 30 18:36:18.625149 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.624868    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d8f39e97-7f68-47da-a247-adf78b523afd-host-run-multus-certs\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:36:18.625149 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.624878    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/545dd3b9-01a9-40be-8762-5476a7a9ede4-etc-sysconfig\") pod \"tuned-f77jk\" (UID: \"545dd3b9-01a9-40be-8762-5476a7a9ede4\") " pod="openshift-cluster-node-tuning-operator/tuned-f77jk"
Oct 30 18:36:18.625149 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:36:18.624847    4525 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 18:36:18.625149 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.624893    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/77c2eb22-4b45-486f-8c62-bd14086942f5-host-slash\") pod \"iptables-alerter-rxbc9\" (UID: \"77c2eb22-4b45-486f-8c62-bd14086942f5\") " pod="openshift-network-operator/iptables-alerter-rxbc9"
Oct 30 18:36:18.625149 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.624913    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7c3ae3e0-ddf1-414f-bb79-e4afc0527b26-dbus\") pod \"global-pull-secret-syncer-vhtmb\" (UID: \"7c3ae3e0-ddf1-414f-bb79-e4afc0527b26\") " pod="kube-system/global-pull-secret-syncer-vhtmb"
Oct 30 18:36:18.625149 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.624916    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8bcb59a4-381f-49ba-b732-f5e4658707de-node-log\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:36:18.625149 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:36:18.624943    4525 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a-metrics-tls podName:5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a nodeName:}" failed. No retries permitted until 2026-10-30 18:36:19.124928978 +0000 UTC m=+1.750598939 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a-metrics-tls") pod "dns-default-lcst7" (UID: "5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a") : secret "dns-default-metrics-tls" not found
Oct 30 18:36:18.625149 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.624951    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/77c2eb22-4b45-486f-8c62-bd14086942f5-host-slash\") pod \"iptables-alerter-rxbc9\" (UID: \"77c2eb22-4b45-486f-8c62-bd14086942f5\") " pod="openshift-network-operator/iptables-alerter-rxbc9"
Oct 30 18:36:18.625149 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.624954    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8bcb59a4-381f-49ba-b732-f5e4658707de-node-log\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:36:18.625149 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.624960    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/545dd3b9-01a9-40be-8762-5476a7a9ede4-etc-sysctl-d\") pod \"tuned-f77jk\" (UID: \"545dd3b9-01a9-40be-8762-5476a7a9ede4\") " pod="openshift-cluster-node-tuning-operator/tuned-f77jk"
Oct 30 18:36:18.625149 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.624983    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8bcb59a4-381f-49ba-b732-f5e4658707de-etc-openvswitch\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:36:18.625149 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.624998    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/545dd3b9-01a9-40be-8762-5476a7a9ede4-etc-kubernetes\") pod \"tuned-f77jk\" (UID: \"545dd3b9-01a9-40be-8762-5476a7a9ede4\") " pod="openshift-cluster-node-tuning-operator/tuned-f77jk"
Oct 30 18:36:18.625149 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.625007    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1315feec-3a59-4cae-ad7e-8ada8c5e4788-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4d49l\" (UID: \"1315feec-3a59-4cae-ad7e-8ada8c5e4788\") " pod="openshift-insights/insights-runtime-extractor-4d49l"
Oct 30 18:36:18.625149 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.625012    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2ca560f4-9aa8-4bc6-9db1-6992a1fa7c1f-socket-dir\") pod \"aws-ebs-csi-driver-node-zb4pz\" (UID: \"2ca560f4-9aa8-4bc6-9db1-6992a1fa7c1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4pz"
Oct 30 18:36:18.625149 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.625038    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8bcb59a4-381f-49ba-b732-f5e4658707de-etc-openvswitch\") pod \"ovnkube-node-kwd4p\" (UID: \"8bcb59a4-381f-49ba-b732-f5e4658707de\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:36:18.625149 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.625052    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2ca560f4-9aa8-4bc6-9db1-6992a1fa7c1f-registration-dir\") pod \"aws-ebs-csi-driver-node-zb4pz\" (UID: \"2ca560f4-9aa8-4bc6-9db1-6992a1fa7c1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4pz"
Oct 30 18:36:18.625149 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.625069    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/545dd3b9-01a9-40be-8762-5476a7a9ede4-etc-sysctl-d\") pod \"tuned-f77jk\" (UID: \"545dd3b9-01a9-40be-8762-5476a7a9ede4\") " pod="openshift-cluster-node-tuning-operator/tuned-f77jk"
Oct 30 18:36:18.625149 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.625074    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/545dd3b9-01a9-40be-8762-5476a7a9ede4-etc-kubernetes\") pod \"tuned-f77jk\" (UID: \"545dd3b9-01a9-40be-8762-5476a7a9ede4\") " pod="openshift-cluster-node-tuning-operator/tuned-f77jk"
Oct 30 18:36:18.625149 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.625051    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2ca560f4-9aa8-4bc6-9db1-6992a1fa7c1f-socket-dir\") pod \"aws-ebs-csi-driver-node-zb4pz\" (UID: \"2ca560f4-9aa8-4bc6-9db1-6992a1fa7c1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4pz"
Oct 30 18:36:18.625149 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.625083    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d8f39e97-7f68-47da-a247-adf78b523afd-host-var-lib-cni-bin\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:36:18.625149 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.625092    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2ca560f4-9aa8-4bc6-9db1-6992a1fa7c1f-registration-dir\") pod \"aws-ebs-csi-driver-node-zb4pz\" (UID: \"2ca560f4-9aa8-4bc6-9db1-6992a1fa7c1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4pz"
Oct 30 18:36:18.625149 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.625119    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d8f39e97-7f68-47da-a247-adf78b523afd-host-var-lib-cni-bin\") pod \"multus-4lmrw\" (UID: \"d8f39e97-7f68-47da-a247-adf78b523afd\") " pod="openshift-multus/multus-4lmrw"
Oct 30 18:36:18.625149 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.625109    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/093a372e-74ea-4820-82c8-0dff7c201f10-host\") pod \"node-ca-mhtvp\" (UID: \"093a372e-74ea-4820-82c8-0dff7c201f10\") " pod="openshift-image-registry/node-ca-mhtvp"
Oct 30 18:36:18.625149 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.625147    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/545dd3b9-01a9-40be-8762-5476a7a9ede4-run\") pod \"tuned-f77jk\" (UID: \"545dd3b9-01a9-40be-8762-5476a7a9ede4\") " pod="openshift-cluster-node-tuning-operator/tuned-f77jk"
Oct 30 18:36:18.625149 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.625173    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/545dd3b9-01a9-40be-8762-5476a7a9ede4-sys\") pod \"tuned-f77jk\" (UID: \"545dd3b9-01a9-40be-8762-5476a7a9ede4\") " pod="openshift-cluster-node-tuning-operator/tuned-f77jk"
Oct 30 18:36:18.625149 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.625194    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/545dd3b9-01a9-40be-8762-5476a7a9ede4-var-lib-kubelet\") pod \"tuned-f77jk\" (UID: \"545dd3b9-01a9-40be-8762-5476a7a9ede4\") " pod="openshift-cluster-node-tuning-operator/tuned-f77jk"
Oct 30 18:36:18.625149 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.625203    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/545dd3b9-01a9-40be-8762-5476a7a9ede4-run\") pod \"tuned-f77jk\" (UID: \"545dd3b9-01a9-40be-8762-5476a7a9ede4\") " pod="openshift-cluster-node-tuning-operator/tuned-f77jk"
Oct 30 18:36:18.625149 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.625214    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2ca560f4-9aa8-4bc6-9db1-6992a1fa7c1f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zb4pz\" (UID: \"2ca560f4-9aa8-4bc6-9db1-6992a1fa7c1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4pz"
Oct 30 18:36:18.627340 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.625220    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/545dd3b9-01a9-40be-8762-5476a7a9ede4-sys\") pod \"tuned-f77jk\" (UID: \"545dd3b9-01a9-40be-8762-5476a7a9ede4\") " pod="openshift-cluster-node-tuning-operator/tuned-f77jk"
Oct 30 18:36:18.627340 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.625195    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/093a372e-74ea-4820-82c8-0dff7c201f10-host\") pod \"node-ca-mhtvp\" (UID: \"093a372e-74ea-4820-82c8-0dff7c201f10\") " pod="openshift-image-registry/node-ca-mhtvp"
Oct 30 18:36:18.627340 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.625242    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/545dd3b9-01a9-40be-8762-5476a7a9ede4-var-lib-kubelet\") pod \"tuned-f77jk\" (UID: \"545dd3b9-01a9-40be-8762-5476a7a9ede4\") " pod="openshift-cluster-node-tuning-operator/tuned-f77jk"
Oct 30 18:36:18.627340 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.625268    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2ca560f4-9aa8-4bc6-9db1-6992a1fa7c1f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zb4pz\" (UID: \"2ca560f4-9aa8-4bc6-9db1-6992a1fa7c1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4pz"
Oct 30 18:36:18.749636 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.749556    4525 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbbbq_bd08a0fa-46bc-4250-985c-d2580a2225b8/konnectivity-agent/0.log" limit=1024
Oct 30 18:36:18.784989 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.784971    4525 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 18:36:18.805548 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:18.805518    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:36:19.128046 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:19.128016    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a-metrics-tls\") pod \"dns-default-lcst7\" (UID: \"5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a\") " pod="openshift-dns/dns-default-lcst7"
Oct 30 18:36:19.128240 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:19.128072    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/089fdd4f-daa0-4b3e-b81e-26a764a526bb-metrics-certs\") pod \"network-metrics-daemon-9p48q\" (UID: \"089fdd4f-daa0-4b3e-b81e-26a764a526bb\") " pod="openshift-multus/network-metrics-daemon-9p48q"
Oct 30 18:36:19.128240 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:19.128092    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1315feec-3a59-4cae-ad7e-8ada8c5e4788-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4d49l\" (UID: \"1315feec-3a59-4cae-ad7e-8ada8c5e4788\") " pod="openshift-insights/insights-runtime-extractor-4d49l"
Oct 30 18:36:19.128240 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:36:19.128177    4525 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 18:36:19.128240 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:36:19.128178    4525 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 18:36:19.128240 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:36:19.128224    4525 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1315feec-3a59-4cae-ad7e-8ada8c5e4788-insights-runtime-extractor-tls podName:1315feec-3a59-4cae-ad7e-8ada8c5e4788 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:20.128211224 +0000 UTC m=+2.753881170 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1315feec-3a59-4cae-ad7e-8ada8c5e4788-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4d49l" (UID: "1315feec-3a59-4cae-ad7e-8ada8c5e4788") : secret "insights-runtime-extractor-tls" not found
Oct 30 18:36:19.128441 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:36:19.128245    4525 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a-metrics-tls podName:5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a nodeName:}" failed. No retries permitted until 2026-10-30 18:36:20.128228873 +0000 UTC m=+2.753898834 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a-metrics-tls") pod "dns-default-lcst7" (UID: "5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a") : secret "dns-default-metrics-tls" not found
Oct 30 18:36:19.128441 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:36:19.128177    4525 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 18:36:19.128441 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:36:19.128290    4525 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/089fdd4f-daa0-4b3e-b81e-26a764a526bb-metrics-certs podName:089fdd4f-daa0-4b3e-b81e-26a764a526bb nodeName:}" failed. No retries permitted until 2026-10-30 18:36:20.128278205 +0000 UTC m=+2.753948161 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/089fdd4f-daa0-4b3e-b81e-26a764a526bb-metrics-certs") pod "network-metrics-daemon-9p48q" (UID: "089fdd4f-daa0-4b3e-b81e-26a764a526bb") : secret "metrics-daemon-secret" not found
Oct 30 18:36:20.135979 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:20.135941    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a-metrics-tls\") pod \"dns-default-lcst7\" (UID: \"5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a\") " pod="openshift-dns/dns-default-lcst7"
Oct 30 18:36:20.136342 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:20.135996    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/089fdd4f-daa0-4b3e-b81e-26a764a526bb-metrics-certs\") pod \"network-metrics-daemon-9p48q\" (UID: \"089fdd4f-daa0-4b3e-b81e-26a764a526bb\") " pod="openshift-multus/network-metrics-daemon-9p48q"
Oct 30 18:36:20.136342 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:36:20.136082    4525 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 18:36:20.136342 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:36:20.136087    4525 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 18:36:20.136342 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:20.136126    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1315feec-3a59-4cae-ad7e-8ada8c5e4788-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4d49l\" (UID: \"1315feec-3a59-4cae-ad7e-8ada8c5e4788\") " pod="openshift-insights/insights-runtime-extractor-4d49l"
Oct 30 18:36:20.136342 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:36:20.136138    4525 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/089fdd4f-daa0-4b3e-b81e-26a764a526bb-metrics-certs podName:089fdd4f-daa0-4b3e-b81e-26a764a526bb nodeName:}" failed. No retries permitted until 2026-10-30 18:36:22.136125208 +0000 UTC m=+4.761795166 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/089fdd4f-daa0-4b3e-b81e-26a764a526bb-metrics-certs") pod "network-metrics-daemon-9p48q" (UID: "089fdd4f-daa0-4b3e-b81e-26a764a526bb") : secret "metrics-daemon-secret" not found
Oct 30 18:36:20.136342 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:36:20.136197    4525 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a-metrics-tls podName:5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a nodeName:}" failed. No retries permitted until 2026-10-30 18:36:22.136180135 +0000 UTC m=+4.761850094 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a-metrics-tls") pod "dns-default-lcst7" (UID: "5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a") : secret "dns-default-metrics-tls" not found
Oct 30 18:36:20.136342 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:36:20.136236    4525 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 18:36:20.136342 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:36:20.136285    4525 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1315feec-3a59-4cae-ad7e-8ada8c5e4788-insights-runtime-extractor-tls podName:1315feec-3a59-4cae-ad7e-8ada8c5e4788 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:22.136272972 +0000 UTC m=+4.761942913 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1315feec-3a59-4cae-ad7e-8ada8c5e4788-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4d49l" (UID: "1315feec-3a59-4cae-ad7e-8ada8c5e4788") : secret "insights-runtime-extractor-tls" not found
Oct 30 18:36:20.580019 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:20.579922    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-5qghx" event={"ID":"25255277-4f3b-418c-9e85-2a82e7a13107","Type":"ContainerStarted","Data":"9ba0e9a949df4a01afcf2b33b305edbac1eda342942dd88c9544e859ff25f98d"}
Oct 30 18:36:20.581614 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:20.581578    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-q4kks-q2hfp" event={"ID":"9ce43bfa-1e19-4fb5-90ff-42751cdd1a50","Type":"ContainerStarted","Data":"d52ae502590170ff357847f9726284480792f0b3d0a67699c1544b4db74bca54"}
Oct 30 18:36:20.692709 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:20.692667    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:36:20.692904 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:20.692889    4525 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 18:36:22.150139 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:22.150098    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/089fdd4f-daa0-4b3e-b81e-26a764a526bb-metrics-certs\") pod \"network-metrics-daemon-9p48q\" (UID: \"089fdd4f-daa0-4b3e-b81e-26a764a526bb\") " pod="openshift-multus/network-metrics-daemon-9p48q"
Oct 30 18:36:22.150139 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:22.150139    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1315feec-3a59-4cae-ad7e-8ada8c5e4788-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4d49l\" (UID: \"1315feec-3a59-4cae-ad7e-8ada8c5e4788\") " pod="openshift-insights/insights-runtime-extractor-4d49l"
Oct 30 18:36:22.150630 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:22.150196    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a-metrics-tls\") pod \"dns-default-lcst7\" (UID: \"5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a\") " pod="openshift-dns/dns-default-lcst7"
Oct 30 18:36:22.150630 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:36:22.150230    4525 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 18:36:22.150630 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:36:22.150302    4525 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 18:36:22.150630 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:36:22.150307    4525 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/089fdd4f-daa0-4b3e-b81e-26a764a526bb-metrics-certs podName:089fdd4f-daa0-4b3e-b81e-26a764a526bb nodeName:}" failed. No retries permitted until 2026-10-30 18:36:26.150280099 +0000 UTC m=+8.775950042 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/089fdd4f-daa0-4b3e-b81e-26a764a526bb-metrics-certs") pod "network-metrics-daemon-9p48q" (UID: "089fdd4f-daa0-4b3e-b81e-26a764a526bb") : secret "metrics-daemon-secret" not found
Oct 30 18:36:22.150630 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:36:22.150372    4525 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1315feec-3a59-4cae-ad7e-8ada8c5e4788-insights-runtime-extractor-tls podName:1315feec-3a59-4cae-ad7e-8ada8c5e4788 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:26.150354817 +0000 UTC m=+8.776024774 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1315feec-3a59-4cae-ad7e-8ada8c5e4788-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4d49l" (UID: "1315feec-3a59-4cae-ad7e-8ada8c5e4788") : secret "insights-runtime-extractor-tls" not found
Oct 30 18:36:22.150630 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:22.150562    4525 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 18:36:22.152675 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:22.152654    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a-metrics-tls\") pod \"dns-default-lcst7\" (UID: \"5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a\") " pod="openshift-dns/dns-default-lcst7"
Oct 30 18:36:22.213143 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:22.213121    4525 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7z9gn_42b71e2d-a8f2-43cf-8487-0e5da2338c54/dns-node-resolver/0.log"
Oct 30 18:36:22.382959 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:22.382923    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lcst7"
Oct 30 18:36:22.650516 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:22.650492    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lcst7"]
Oct 30 18:36:22.652891 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:22.652861    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e44ab8a_1e33_4c96_a4d4_9f3c71e5883a.slice/crio-1180b727f1fc84c89ddb6d75e5b065f88cfa8e21e5a96b61cb3fa9e624b1d136 WatchSource:0}: Error finding container 1180b727f1fc84c89ddb6d75e5b065f88cfa8e21e5a96b61cb3fa9e624b1d136: Status 404 returned error can't find the container with id 1180b727f1fc84c89ddb6d75e5b065f88cfa8e21e5a96b61cb3fa9e624b1d136
Oct 30 18:36:23.214008 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:23.213983    4525 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mhtvp_093a372e-74ea-4820-82c8-0dff7c201f10/node-ca/0.log"
Oct 30 18:36:23.590598 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:23.590505    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lcst7" event={"ID":"5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a","Type":"ContainerStarted","Data":"1180b727f1fc84c89ddb6d75e5b065f88cfa8e21e5a96b61cb3fa9e624b1d136"}
Oct 30 18:36:23.814046 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:23.814013    4525 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dqxm8_f32f2498-2fc8-4bc6-b368-b53d156755c5/serve-healthcheck-canary/0.log"
Oct 30 18:36:24.414361 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:24.414326    4525 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-mhxzk_97f514c8-0d31-4103-81c5-45886ff831f0/migrator/0.log"
Oct 30 18:36:24.596245 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:24.596169    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lcst7" event={"ID":"5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a","Type":"ContainerStarted","Data":"3e2953c8cce97d0efdc32c5fb33b7bf7182c83c8d16e7a29b915b57d249619f3"}
Oct 30 18:36:24.596245 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:24.596214    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lcst7" event={"ID":"5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a","Type":"ContainerStarted","Data":"9725f0e7f670961c853e8626726730b58436923fdd81d0306a334fd11d824618"}
Oct 30 18:36:24.596455 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:24.596323    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-lcst7"
Oct 30 18:36:26.182102 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:26.182066    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/089fdd4f-daa0-4b3e-b81e-26a764a526bb-metrics-certs\") pod \"network-metrics-daemon-9p48q\" (UID: \"089fdd4f-daa0-4b3e-b81e-26a764a526bb\") " pod="openshift-multus/network-metrics-daemon-9p48q"
Oct 30 18:36:26.182102 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:26.182105    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1315feec-3a59-4cae-ad7e-8ada8c5e4788-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4d49l\" (UID: \"1315feec-3a59-4cae-ad7e-8ada8c5e4788\") " pod="openshift-insights/insights-runtime-extractor-4d49l"
Oct 30 18:36:26.182555 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:36:26.182239    4525 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 18:36:26.182555 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:36:26.182342    4525 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/089fdd4f-daa0-4b3e-b81e-26a764a526bb-metrics-certs podName:089fdd4f-daa0-4b3e-b81e-26a764a526bb nodeName:}" failed. No retries permitted until 2026-10-30 18:36:34.182326988 +0000 UTC m=+16.807996931 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/089fdd4f-daa0-4b3e-b81e-26a764a526bb-metrics-certs") pod "network-metrics-daemon-9p48q" (UID: "089fdd4f-daa0-4b3e-b81e-26a764a526bb") : secret "metrics-daemon-secret" not found
Oct 30 18:36:26.184599 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:26.184574    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1315feec-3a59-4cae-ad7e-8ada8c5e4788-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4d49l\" (UID: \"1315feec-3a59-4cae-ad7e-8ada8c5e4788\") " pod="openshift-insights/insights-runtime-extractor-4d49l"
Oct 30 18:36:26.282482 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:26.282449    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4d49l"
Oct 30 18:36:26.552068 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:26.551991    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4d49l"]
Oct 30 18:36:26.555444 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:26.555418    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1315feec_3a59_4cae_ad7e_8ada8c5e4788.slice/crio-1ac5636c6d9d3e9be23e4072f1594bef9ae65fa0433b8a45b6e59756ffec96bd WatchSource:0}: Error finding container 1ac5636c6d9d3e9be23e4072f1594bef9ae65fa0433b8a45b6e59756ffec96bd: Status 404 returned error can't find the container with id 1ac5636c6d9d3e9be23e4072f1594bef9ae65fa0433b8a45b6e59756ffec96bd
Oct 30 18:36:26.604024 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:26.603985    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4d49l" event={"ID":"1315feec-3a59-4cae-ad7e-8ada8c5e4788","Type":"ContainerStarted","Data":"1ac5636c6d9d3e9be23e4072f1594bef9ae65fa0433b8a45b6e59756ffec96bd"}
Oct 30 18:36:27.608614 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:27.608526    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4d49l" event={"ID":"1315feec-3a59-4cae-ad7e-8ada8c5e4788","Type":"ContainerStarted","Data":"d3707e648ecd405f4316a67ae5cd2118f073dd2a08eb7ee6b33ad8783a1c916f"}
Oct 30 18:36:27.608614 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:27.608587    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4d49l" event={"ID":"1315feec-3a59-4cae-ad7e-8ada8c5e4788","Type":"ContainerStarted","Data":"db6a4371d0ea9191b6129e64e61c1657604889871052b1a24d8e3a888e401df0"}
Oct 30 18:36:28.645263 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:28.645230    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-mcc4d"
Oct 30 18:36:29.621086 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:29.621046    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4d49l" event={"ID":"1315feec-3a59-4cae-ad7e-8ada8c5e4788","Type":"ContainerStarted","Data":"717c3f806a0fd42d6518268b550546b1a6c56368566745b65af48f0688cf9d49"}
Oct 30 18:36:34.239004 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:34.238962    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/089fdd4f-daa0-4b3e-b81e-26a764a526bb-metrics-certs\") pod \"network-metrics-daemon-9p48q\" (UID: \"089fdd4f-daa0-4b3e-b81e-26a764a526bb\") " pod="openshift-multus/network-metrics-daemon-9p48q"
Oct 30 18:36:34.241350 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:34.241329    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/089fdd4f-daa0-4b3e-b81e-26a764a526bb-metrics-certs\") pod \"network-metrics-daemon-9p48q\" (UID: \"089fdd4f-daa0-4b3e-b81e-26a764a526bb\") " pod="openshift-multus/network-metrics-daemon-9p48q"
Oct 30 18:36:34.379204 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:34.379159    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9p48q"
Oct 30 18:36:34.454740 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:34.454714    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-86qts"]
Oct 30 18:36:34.457807 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:34.457788    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-86qts"
Oct 30 18:36:34.459799 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:34.459775    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-l98wz"]
Oct 30 18:36:34.462496 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:34.462480    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-l98wz"
Oct 30 18:36:34.469656 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:34.469627    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Oct 30 18:36:34.469786 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:34.469759    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-k7q7h\""
Oct 30 18:36:34.471583 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:34.471562    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-86qts"]
Oct 30 18:36:34.475088 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:34.475044    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 30 18:36:34.475225 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:34.475210    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 30 18:36:34.476897 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:34.476877    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-l98wz"]
Oct 30 18:36:34.541947 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:34.541874    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvdm9\" (UniqueName: \"kubernetes.io/projected/356d85d5-c222-49e7-916b-bfd94d2438e1-kube-api-access-fvdm9\") pod \"downloads-6dd789c8b7-l98wz\" (UID: \"356d85d5-c222-49e7-916b-bfd94d2438e1\") " pod="openshift-console/downloads-6dd789c8b7-l98wz"
Oct 30 18:36:34.541947 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:34.541913    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7815c72b-c24d-4720-9400-2482edce2255-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-86qts\" (UID: \"7815c72b-c24d-4720-9400-2482edce2255\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-86qts"
Oct 30 18:36:34.541947 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:34.541937    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/356d85d5-c222-49e7-916b-bfd94d2438e1-tmp\") pod \"downloads-6dd789c8b7-l98wz\" (UID: \"356d85d5-c222-49e7-916b-bfd94d2438e1\") " pod="openshift-console/downloads-6dd789c8b7-l98wz"
Oct 30 18:36:34.564914 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:34.564890    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-b89865498-mr5st"]
Oct 30 18:36:34.567753 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:34.567738    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b89865498-mr5st"
Oct 30 18:36:34.573927 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:34.573909    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-snxck\""
Oct 30 18:36:34.582290 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:34.582271    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 18:36:34.585763 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:34.585743    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-b89865498-mr5st"]
Oct 30 18:36:34.602755 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:34.602734    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-lcst7"
Oct 30 18:36:34.642759 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:34.642737    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/64563635-ed73-4e02-9333-2527c071bb00-registry-certificates\") pod \"image-registry-b89865498-mr5st\" (UID: \"64563635-ed73-4e02-9333-2527c071bb00\") " pod="openshift-image-registry/image-registry-b89865498-mr5st"
Oct 30 18:36:34.642892 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:34.642766    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/64563635-ed73-4e02-9333-2527c071bb00-bound-sa-token\") pod \"image-registry-b89865498-mr5st\" (UID: \"64563635-ed73-4e02-9333-2527c071bb00\") " pod="openshift-image-registry/image-registry-b89865498-mr5st"
Oct 30 18:36:34.642892 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:34.642789    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vp6s2\" (UniqueName: \"kubernetes.io/projected/64563635-ed73-4e02-9333-2527c071bb00-kube-api-access-vp6s2\") pod \"image-registry-b89865498-mr5st\" (UID: \"64563635-ed73-4e02-9333-2527c071bb00\") " pod="openshift-image-registry/image-registry-b89865498-mr5st"
Oct 30 18:36:34.642892 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:34.642803    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/64563635-ed73-4e02-9333-2527c071bb00-trusted-ca\") pod \"image-registry-b89865498-mr5st\" (UID: \"64563635-ed73-4e02-9333-2527c071bb00\") " pod="openshift-image-registry/image-registry-b89865498-mr5st"
Oct 30 18:36:34.643121 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:34.643104    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7815c72b-c24d-4720-9400-2482edce2255-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-86qts\" (UID: \"7815c72b-c24d-4720-9400-2482edce2255\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-86qts"
Oct 30 18:36:34.643298 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:34.643135    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/356d85d5-c222-49e7-916b-bfd94d2438e1-tmp\") pod \"downloads-6dd789c8b7-l98wz\" (UID: \"356d85d5-c222-49e7-916b-bfd94d2438e1\") " pod="openshift-console/downloads-6dd789c8b7-l98wz"
Oct 30 18:36:34.643298 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:34.643161    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/64563635-ed73-4e02-9333-2527c071bb00-registry-tls\") pod \"image-registry-b89865498-mr5st\" (UID: \"64563635-ed73-4e02-9333-2527c071bb00\") " pod="openshift-image-registry/image-registry-b89865498-mr5st"
Oct 30 18:36:34.643298 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:34.643203    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/64563635-ed73-4e02-9333-2527c071bb00-ca-trust-extracted\") pod \"image-registry-b89865498-mr5st\" (UID: \"64563635-ed73-4e02-9333-2527c071bb00\") " pod="openshift-image-registry/image-registry-b89865498-mr5st"
Oct 30 18:36:34.643298 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:34.643253    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvdm9\" (UniqueName: \"kubernetes.io/projected/356d85d5-c222-49e7-916b-bfd94d2438e1-kube-api-access-fvdm9\") pod \"downloads-6dd789c8b7-l98wz\" (UID: \"356d85d5-c222-49e7-916b-bfd94d2438e1\") " pod="openshift-console/downloads-6dd789c8b7-l98wz"
Oct 30 18:36:34.643298 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:34.643283    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/64563635-ed73-4e02-9333-2527c071bb00-installation-pull-secrets\") pod \"image-registry-b89865498-mr5st\" (UID: \"64563635-ed73-4e02-9333-2527c071bb00\") " pod="openshift-image-registry/image-registry-b89865498-mr5st"
Oct 30 18:36:34.643566 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:34.643308    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/64563635-ed73-4e02-9333-2527c071bb00-image-registry-private-configuration\") pod \"image-registry-b89865498-mr5st\" (UID: \"64563635-ed73-4e02-9333-2527c071bb00\") " pod="openshift-image-registry/image-registry-b89865498-mr5st"
Oct 30 18:36:34.643566 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:34.643523    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/356d85d5-c222-49e7-916b-bfd94d2438e1-tmp\") pod \"downloads-6dd789c8b7-l98wz\" (UID: \"356d85d5-c222-49e7-916b-bfd94d2438e1\") " pod="openshift-console/downloads-6dd789c8b7-l98wz"
Oct 30 18:36:34.645802 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:34.645783    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7815c72b-c24d-4720-9400-2482edce2255-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-86qts\" (UID: \"7815c72b-c24d-4720-9400-2482edce2255\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-86qts"
Oct 30 18:36:34.657107 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:34.657089    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvdm9\" (UniqueName: \"kubernetes.io/projected/356d85d5-c222-49e7-916b-bfd94d2438e1-kube-api-access-fvdm9\") pod \"downloads-6dd789c8b7-l98wz\" (UID: \"356d85d5-c222-49e7-916b-bfd94d2438e1\") " pod="openshift-console/downloads-6dd789c8b7-l98wz"
Oct 30 18:36:34.665416 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:34.665393    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9p48q"]
Oct 30 18:36:34.671072 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:34.671045    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod089fdd4f_daa0_4b3e_b81e_26a764a526bb.slice/crio-0595d20cd5d11fe66ef90ae551e983bf345aacd1fee21122ca966d98efc4d5d8 WatchSource:0}: Error finding container 0595d20cd5d11fe66ef90ae551e983bf345aacd1fee21122ca966d98efc4d5d8: Status 404 returned error can't find the container with id 0595d20cd5d11fe66ef90ae551e983bf345aacd1fee21122ca966d98efc4d5d8
Oct 30 18:36:34.671858 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:34.671834    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 18:36:34.672168 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:34.672149    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 18:36:34.672168 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:34.672162    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-d6n4c\""
Oct 30 18:36:34.672295 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:34.672285    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 18:36:34.743831 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:34.743806    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/64563635-ed73-4e02-9333-2527c071bb00-registry-tls\") pod \"image-registry-b89865498-mr5st\" (UID: \"64563635-ed73-4e02-9333-2527c071bb00\") " pod="openshift-image-registry/image-registry-b89865498-mr5st"
Oct 30 18:36:34.743982 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:34.743844    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/64563635-ed73-4e02-9333-2527c071bb00-ca-trust-extracted\") pod \"image-registry-b89865498-mr5st\" (UID: \"64563635-ed73-4e02-9333-2527c071bb00\") " pod="openshift-image-registry/image-registry-b89865498-mr5st"
Oct 30 18:36:34.743982 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:34.743864    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/64563635-ed73-4e02-9333-2527c071bb00-installation-pull-secrets\") pod \"image-registry-b89865498-mr5st\" (UID: \"64563635-ed73-4e02-9333-2527c071bb00\") " pod="openshift-image-registry/image-registry-b89865498-mr5st"
Oct 30 18:36:34.743982 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:34.743964    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/64563635-ed73-4e02-9333-2527c071bb00-image-registry-private-configuration\") pod \"image-registry-b89865498-mr5st\" (UID: \"64563635-ed73-4e02-9333-2527c071bb00\") " pod="openshift-image-registry/image-registry-b89865498-mr5st"
Oct 30 18:36:34.744111 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:34.743991    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/64563635-ed73-4e02-9333-2527c071bb00-registry-certificates\") pod \"image-registry-b89865498-mr5st\" (UID: \"64563635-ed73-4e02-9333-2527c071bb00\") " pod="openshift-image-registry/image-registry-b89865498-mr5st"
Oct 30 18:36:34.744111 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:34.744009    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/64563635-ed73-4e02-9333-2527c071bb00-bound-sa-token\") pod \"image-registry-b89865498-mr5st\" (UID: \"64563635-ed73-4e02-9333-2527c071bb00\") " pod="openshift-image-registry/image-registry-b89865498-mr5st"
Oct 30 18:36:34.744111 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:34.744031    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp6s2\" (UniqueName: \"kubernetes.io/projected/64563635-ed73-4e02-9333-2527c071bb00-kube-api-access-vp6s2\") pod \"image-registry-b89865498-mr5st\" (UID: \"64563635-ed73-4e02-9333-2527c071bb00\") " pod="openshift-image-registry/image-registry-b89865498-mr5st"
Oct 30 18:36:34.744111 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:34.744053    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/64563635-ed73-4e02-9333-2527c071bb00-trusted-ca\") pod \"image-registry-b89865498-mr5st\" (UID: \"64563635-ed73-4e02-9333-2527c071bb00\") " pod="openshift-image-registry/image-registry-b89865498-mr5st"
Oct 30 18:36:34.744994 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:34.744948    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/64563635-ed73-4e02-9333-2527c071bb00-ca-trust-extracted\") pod \"image-registry-b89865498-mr5st\" (UID: \"64563635-ed73-4e02-9333-2527c071bb00\") " pod="openshift-image-registry/image-registry-b89865498-mr5st"
Oct 30 18:36:34.745130 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:34.745012    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/64563635-ed73-4e02-9333-2527c071bb00-registry-certificates\") pod \"image-registry-b89865498-mr5st\" (UID: \"64563635-ed73-4e02-9333-2527c071bb00\") " pod="openshift-image-registry/image-registry-b89865498-mr5st"
Oct 30 18:36:34.745130 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:34.745056    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/64563635-ed73-4e02-9333-2527c071bb00-trusted-ca\") pod \"image-registry-b89865498-mr5st\" (UID: \"64563635-ed73-4e02-9333-2527c071bb00\") " pod="openshift-image-registry/image-registry-b89865498-mr5st"
Oct 30 18:36:34.746519 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:34.746493    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/64563635-ed73-4e02-9333-2527c071bb00-registry-tls\") pod \"image-registry-b89865498-mr5st\" (UID: \"64563635-ed73-4e02-9333-2527c071bb00\") " pod="openshift-image-registry/image-registry-b89865498-mr5st"
Oct 30 18:36:34.746519 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:34.746513    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/64563635-ed73-4e02-9333-2527c071bb00-installation-pull-secrets\") pod \"image-registry-b89865498-mr5st\" (UID: \"64563635-ed73-4e02-9333-2527c071bb00\") " pod="openshift-image-registry/image-registry-b89865498-mr5st"
Oct 30 18:36:34.746632 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:34.746567    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/64563635-ed73-4e02-9333-2527c071bb00-image-registry-private-configuration\") pod \"image-registry-b89865498-mr5st\" (UID: \"64563635-ed73-4e02-9333-2527c071bb00\") " pod="openshift-image-registry/image-registry-b89865498-mr5st"
Oct 30 18:36:34.760561 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:34.760535    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp6s2\" (UniqueName: \"kubernetes.io/projected/64563635-ed73-4e02-9333-2527c071bb00-kube-api-access-vp6s2\") pod \"image-registry-b89865498-mr5st\" (UID: \"64563635-ed73-4e02-9333-2527c071bb00\") " pod="openshift-image-registry/image-registry-b89865498-mr5st"
Oct 30 18:36:34.760727 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:34.760712    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/64563635-ed73-4e02-9333-2527c071bb00-bound-sa-token\") pod \"image-registry-b89865498-mr5st\" (UID: \"64563635-ed73-4e02-9333-2527c071bb00\") " pod="openshift-image-registry/image-registry-b89865498-mr5st"
Oct 30 18:36:34.770633 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:34.770617    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-86qts"
Oct 30 18:36:34.776564 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:34.776547    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-l98wz"
Oct 30 18:36:34.878645 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:34.878621    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b89865498-mr5st"
Oct 30 18:36:35.050060 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:35.047828    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-86qts"]
Oct 30 18:36:35.051606 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:35.051577    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-l98wz"]
Oct 30 18:36:35.052661 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:35.052623    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7815c72b_c24d_4720_9400_2482edce2255.slice/crio-6438f9b0d9bcbf487e99fe75e5b06f0764d87ed8fced930143d1f49f84ff6c3c WatchSource:0}: Error finding container 6438f9b0d9bcbf487e99fe75e5b06f0764d87ed8fced930143d1f49f84ff6c3c: Status 404 returned error can't find the container with id 6438f9b0d9bcbf487e99fe75e5b06f0764d87ed8fced930143d1f49f84ff6c3c
Oct 30 18:36:35.055436 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:35.055417    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod356d85d5_c222_49e7_916b_bfd94d2438e1.slice/crio-82399845baf41895d4e01bddc969c96c4cc9503d69d5a18428ce937663a7a39b WatchSource:0}: Error finding container 82399845baf41895d4e01bddc969c96c4cc9503d69d5a18428ce937663a7a39b: Status 404 returned error can't find the container with id 82399845baf41895d4e01bddc969c96c4cc9503d69d5a18428ce937663a7a39b
Oct 30 18:36:35.150844 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:35.150815    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-b89865498-mr5st"]
Oct 30 18:36:35.154123 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:35.154093    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64563635_ed73_4e02_9333_2527c071bb00.slice/crio-0e56db5f7518521ceca0daff5275b6e7fe89cea3a7c43418a0cf321e8249638d WatchSource:0}: Error finding container 0e56db5f7518521ceca0daff5275b6e7fe89cea3a7c43418a0cf321e8249638d: Status 404 returned error can't find the container with id 0e56db5f7518521ceca0daff5275b6e7fe89cea3a7c43418a0cf321e8249638d
Oct 30 18:36:35.644321 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:35.644277    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b89865498-mr5st" event={"ID":"64563635-ed73-4e02-9333-2527c071bb00","Type":"ContainerStarted","Data":"eff3993e21951a4fef15910706a06903f02fdd7928eefdbdb065b928b870333b"}
Oct 30 18:36:35.644816 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:35.644332    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b89865498-mr5st" event={"ID":"64563635-ed73-4e02-9333-2527c071bb00","Type":"ContainerStarted","Data":"0e56db5f7518521ceca0daff5275b6e7fe89cea3a7c43418a0cf321e8249638d"}
Oct 30 18:36:35.644816 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:35.644770    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-b89865498-mr5st"
Oct 30 18:36:35.646349 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:35.646316    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-l98wz" event={"ID":"356d85d5-c222-49e7-916b-bfd94d2438e1","Type":"ContainerStarted","Data":"82399845baf41895d4e01bddc969c96c4cc9503d69d5a18428ce937663a7a39b"}
Oct 30 18:36:35.647603 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:35.647579    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-86qts" event={"ID":"7815c72b-c24d-4720-9400-2482edce2255","Type":"ContainerStarted","Data":"6438f9b0d9bcbf487e99fe75e5b06f0764d87ed8fced930143d1f49f84ff6c3c"}
Oct 30 18:36:35.650418 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:35.650392    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9p48q" event={"ID":"089fdd4f-daa0-4b3e-b81e-26a764a526bb","Type":"ContainerStarted","Data":"0595d20cd5d11fe66ef90ae551e983bf345aacd1fee21122ca966d98efc4d5d8"}
Oct 30 18:36:35.667386 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:35.667333    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-b89865498-mr5st" podStartSLOduration=1.667317292 podStartE2EDuration="1.667317292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:36:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:35.666341187 +0000 UTC m=+18.292011152" watchObservedRunningTime="2026-10-30 18:36:35.667317292 +0000 UTC m=+18.292987258"
Oct 30 18:36:36.656385 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:36.656346    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-86qts" event={"ID":"7815c72b-c24d-4720-9400-2482edce2255","Type":"ContainerStarted","Data":"57ccf6b9e44feb3b1352ca359795f52f1fc3e23b1fa55e151e8cfaae891777d0"}
Oct 30 18:36:36.657000 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:36.656523    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-86qts"
Oct 30 18:36:36.658573 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:36.658538    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9p48q" event={"ID":"089fdd4f-daa0-4b3e-b81e-26a764a526bb","Type":"ContainerStarted","Data":"ed143e5cc63dfe43ffeb46827a0aaf0488ab7eef68800874bc031c7805c2f5ee"}
Oct 30 18:36:36.658696 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:36.658586    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9p48q" event={"ID":"089fdd4f-daa0-4b3e-b81e-26a764a526bb","Type":"ContainerStarted","Data":"5eed64715049ab9180d8e4025416435a9f0711429c782246afa6379b34bcb77d"}
Oct 30 18:36:36.661819 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:36.661796    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-86qts"
Oct 30 18:36:36.673016 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:36.672964    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-86qts" podStartSLOduration=1.559220198 podStartE2EDuration="2.672952761s" totalImagesPullingTime="1.113732563s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:36:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:36.671610378 +0000 UTC m=+19.297280356" watchObservedRunningTime="2026-10-30 18:36:36.672952761 +0000 UTC m=+19.298622725"
Oct 30 18:36:39.979853 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:39.979819    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kwd4p"
Oct 30 18:36:41.265957 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:41.265926    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-rwkfn"]
Oct 30 18:36:41.271314 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:41.271283    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rwkfn"
Oct 30 18:36:41.323212 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:41.323159    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 18:36:41.323594 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:41.323570    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 18:36:41.323736 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:41.323606    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-dgrqn\""
Oct 30 18:36:41.354664 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:41.354612    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 18:36:41.354843 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:41.354804    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 18:36:41.354843 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:41.354826    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 18:36:41.354948 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:41.354846    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 18:36:41.402233 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:41.402192    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/62619ba0-8477-4864-a107-7c2fd2de3273-node-exporter-tls\") pod \"node-exporter-rwkfn\" (UID: \"62619ba0-8477-4864-a107-7c2fd2de3273\") " pod="openshift-monitoring/node-exporter-rwkfn"
Oct 30 18:36:41.402233 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:41.402238    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/62619ba0-8477-4864-a107-7c2fd2de3273-node-exporter-wtmp\") pod \"node-exporter-rwkfn\" (UID: \"62619ba0-8477-4864-a107-7c2fd2de3273\") " pod="openshift-monitoring/node-exporter-rwkfn"
Oct 30 18:36:41.402457 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:41.402261    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/62619ba0-8477-4864-a107-7c2fd2de3273-root\") pod \"node-exporter-rwkfn\" (UID: \"62619ba0-8477-4864-a107-7c2fd2de3273\") " pod="openshift-monitoring/node-exporter-rwkfn"
Oct 30 18:36:41.402457 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:41.402337    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/62619ba0-8477-4864-a107-7c2fd2de3273-node-exporter-textfile\") pod \"node-exporter-rwkfn\" (UID: \"62619ba0-8477-4864-a107-7c2fd2de3273\") " pod="openshift-monitoring/node-exporter-rwkfn"
Oct 30 18:36:41.402457 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:41.402446    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/62619ba0-8477-4864-a107-7c2fd2de3273-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rwkfn\" (UID: \"62619ba0-8477-4864-a107-7c2fd2de3273\") " pod="openshift-monitoring/node-exporter-rwkfn"
Oct 30 18:36:41.402574 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:41.402480    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/62619ba0-8477-4864-a107-7c2fd2de3273-node-exporter-accelerators-collector-config\") pod \"node-exporter-rwkfn\" (UID: \"62619ba0-8477-4864-a107-7c2fd2de3273\") " pod="openshift-monitoring/node-exporter-rwkfn"
Oct 30 18:36:41.402621 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:41.402577    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/62619ba0-8477-4864-a107-7c2fd2de3273-sys\") pod \"node-exporter-rwkfn\" (UID: \"62619ba0-8477-4864-a107-7c2fd2de3273\") " pod="openshift-monitoring/node-exporter-rwkfn"
Oct 30 18:36:41.402669 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:41.402627    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2r42d\" (UniqueName: \"kubernetes.io/projected/62619ba0-8477-4864-a107-7c2fd2de3273-kube-api-access-2r42d\") pod \"node-exporter-rwkfn\" (UID: \"62619ba0-8477-4864-a107-7c2fd2de3273\") " pod="openshift-monitoring/node-exporter-rwkfn"
Oct 30 18:36:41.402770 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:41.402729    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/62619ba0-8477-4864-a107-7c2fd2de3273-metrics-client-ca\") pod \"node-exporter-rwkfn\" (UID: \"62619ba0-8477-4864-a107-7c2fd2de3273\") " pod="openshift-monitoring/node-exporter-rwkfn"
Oct 30 18:36:41.503791 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:41.503756    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/62619ba0-8477-4864-a107-7c2fd2de3273-sys\") pod \"node-exporter-rwkfn\" (UID: \"62619ba0-8477-4864-a107-7c2fd2de3273\") " pod="openshift-monitoring/node-exporter-rwkfn"
Oct 30 18:36:41.503791 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:41.503798    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r42d\" (UniqueName: \"kubernetes.io/projected/62619ba0-8477-4864-a107-7c2fd2de3273-kube-api-access-2r42d\") pod \"node-exporter-rwkfn\" (UID: \"62619ba0-8477-4864-a107-7c2fd2de3273\") " pod="openshift-monitoring/node-exporter-rwkfn"
Oct 30 18:36:41.504048 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:41.503822    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/62619ba0-8477-4864-a107-7c2fd2de3273-metrics-client-ca\") pod \"node-exporter-rwkfn\" (UID: \"62619ba0-8477-4864-a107-7c2fd2de3273\") " pod="openshift-monitoring/node-exporter-rwkfn"
Oct 30 18:36:41.504048 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:41.503868    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/62619ba0-8477-4864-a107-7c2fd2de3273-node-exporter-tls\") pod \"node-exporter-rwkfn\" (UID: \"62619ba0-8477-4864-a107-7c2fd2de3273\") " pod="openshift-monitoring/node-exporter-rwkfn"
Oct 30 18:36:41.504048 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:41.503898    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/62619ba0-8477-4864-a107-7c2fd2de3273-node-exporter-wtmp\") pod \"node-exporter-rwkfn\" (UID: \"62619ba0-8477-4864-a107-7c2fd2de3273\") " pod="openshift-monitoring/node-exporter-rwkfn"
Oct 30 18:36:41.504048 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:41.503897    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/62619ba0-8477-4864-a107-7c2fd2de3273-sys\") pod \"node-exporter-rwkfn\" (UID: \"62619ba0-8477-4864-a107-7c2fd2de3273\") " pod="openshift-monitoring/node-exporter-rwkfn"
Oct 30 18:36:41.504048 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:41.503916    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/62619ba0-8477-4864-a107-7c2fd2de3273-root\") pod \"node-exporter-rwkfn\" (UID: \"62619ba0-8477-4864-a107-7c2fd2de3273\") " pod="openshift-monitoring/node-exporter-rwkfn"
Oct 30 18:36:41.504048 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:41.503931    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/62619ba0-8477-4864-a107-7c2fd2de3273-node-exporter-textfile\") pod \"node-exporter-rwkfn\" (UID: \"62619ba0-8477-4864-a107-7c2fd2de3273\") " pod="openshift-monitoring/node-exporter-rwkfn"
Oct 30 18:36:41.504048 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:41.503950    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/62619ba0-8477-4864-a107-7c2fd2de3273-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rwkfn\" (UID: \"62619ba0-8477-4864-a107-7c2fd2de3273\") " pod="openshift-monitoring/node-exporter-rwkfn"
Oct 30 18:36:41.504048 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:41.503970    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/62619ba0-8477-4864-a107-7c2fd2de3273-node-exporter-accelerators-collector-config\") pod \"node-exporter-rwkfn\" (UID: \"62619ba0-8477-4864-a107-7c2fd2de3273\") " pod="openshift-monitoring/node-exporter-rwkfn"
Oct 30 18:36:41.504416 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:41.504172    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/62619ba0-8477-4864-a107-7c2fd2de3273-root\") pod \"node-exporter-rwkfn\" (UID: \"62619ba0-8477-4864-a107-7c2fd2de3273\") " pod="openshift-monitoring/node-exporter-rwkfn"
Oct 30 18:36:41.504416 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:41.504290    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/62619ba0-8477-4864-a107-7c2fd2de3273-node-exporter-wtmp\") pod \"node-exporter-rwkfn\" (UID: \"62619ba0-8477-4864-a107-7c2fd2de3273\") " pod="openshift-monitoring/node-exporter-rwkfn"
Oct 30 18:36:41.504579 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:41.504559    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/62619ba0-8477-4864-a107-7c2fd2de3273-node-exporter-accelerators-collector-config\") pod \"node-exporter-rwkfn\" (UID: \"62619ba0-8477-4864-a107-7c2fd2de3273\") " pod="openshift-monitoring/node-exporter-rwkfn"
Oct 30 18:36:41.504746 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:41.504673    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/62619ba0-8477-4864-a107-7c2fd2de3273-node-exporter-textfile\") pod \"node-exporter-rwkfn\" (UID: \"62619ba0-8477-4864-a107-7c2fd2de3273\") " pod="openshift-monitoring/node-exporter-rwkfn"
Oct 30 18:36:41.504829 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:41.504780    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/62619ba0-8477-4864-a107-7c2fd2de3273-metrics-client-ca\") pod \"node-exporter-rwkfn\" (UID: \"62619ba0-8477-4864-a107-7c2fd2de3273\") " pod="openshift-monitoring/node-exporter-rwkfn"
Oct 30 18:36:41.507012 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:41.506959    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/62619ba0-8477-4864-a107-7c2fd2de3273-node-exporter-tls\") pod \"node-exporter-rwkfn\" (UID: \"62619ba0-8477-4864-a107-7c2fd2de3273\") " pod="openshift-monitoring/node-exporter-rwkfn"
Oct 30 18:36:41.507335 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:41.507315    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/62619ba0-8477-4864-a107-7c2fd2de3273-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rwkfn\" (UID: \"62619ba0-8477-4864-a107-7c2fd2de3273\") " pod="openshift-monitoring/node-exporter-rwkfn"
Oct 30 18:36:41.511785 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:41.511758    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2r42d\" (UniqueName: \"kubernetes.io/projected/62619ba0-8477-4864-a107-7c2fd2de3273-kube-api-access-2r42d\") pod \"node-exporter-rwkfn\" (UID: \"62619ba0-8477-4864-a107-7c2fd2de3273\") " pod="openshift-monitoring/node-exporter-rwkfn"
Oct 30 18:36:41.589944 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:41.589868    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rwkfn"
Oct 30 18:36:41.607094 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:41.607059    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62619ba0_8477_4864_a107_7c2fd2de3273.slice/crio-b2abac5aaa63060074c14badb23e87faa52eff6625ab46fc5bb19b99fe47d4ee WatchSource:0}: Error finding container b2abac5aaa63060074c14badb23e87faa52eff6625ab46fc5bb19b99fe47d4ee: Status 404 returned error can't find the container with id b2abac5aaa63060074c14badb23e87faa52eff6625ab46fc5bb19b99fe47d4ee
Oct 30 18:36:41.676200 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:41.676155    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rwkfn" event={"ID":"62619ba0-8477-4864-a107-7c2fd2de3273","Type":"ContainerStarted","Data":"b2abac5aaa63060074c14badb23e87faa52eff6625ab46fc5bb19b99fe47d4ee"}
Oct 30 18:36:43.684461 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:43.684424    4525 generic.go:356] "Generic (PLEG): container finished" podID="62619ba0-8477-4864-a107-7c2fd2de3273" containerID="e721b78e864f34d95c7fff9437ae06ee81394628e5144bb6b61f4eb49aa65d27" exitCode=0
Oct 30 18:36:43.684905 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:43.684514    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rwkfn" event={"ID":"62619ba0-8477-4864-a107-7c2fd2de3273","Type":"ContainerDied","Data":"e721b78e864f34d95c7fff9437ae06ee81394628e5144bb6b61f4eb49aa65d27"}
Oct 30 18:36:44.480843 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:44.480808    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-dbbbq"
Oct 30 18:36:44.691309 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:44.691269    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rwkfn" event={"ID":"62619ba0-8477-4864-a107-7c2fd2de3273","Type":"ContainerStarted","Data":"49d934a4ad3a63a35bba1b430052dfd4d42ad8130e5680ca6a0607f6b256949b"}
Oct 30 18:36:44.691720 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:44.691335    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rwkfn" event={"ID":"62619ba0-8477-4864-a107-7c2fd2de3273","Type":"ContainerStarted","Data":"ab263bc66fe2a23fbc9bd83f1df7d4f6470c52828fd9600d0c320926ad94318c"}
Oct 30 18:36:44.711798 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:44.711725    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-rwkfn" podStartSLOduration=2.604692215 podStartE2EDuration="3.711711308s" totalImagesPullingTime="1.022371163s" totalInitContainerRuntime="84.64793ms" isStatefulPod=true podCreationTimestamp="2026-10-30 18:36:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:44.710470443 +0000 UTC m=+27.336140409" watchObservedRunningTime="2026-10-30 18:36:44.711711308 +0000 UTC m=+27.337381271"
Oct 30 18:36:46.012566 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:46.012533    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-xl7v4"]
Oct 30 18:36:46.017158 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:46.016222    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-xl7v4"
Oct 30 18:36:46.028785 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:46.027504    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-xl7v4"]
Oct 30 18:36:46.029634 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:46.029613    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Oct 30 18:36:46.029771 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:46.029698    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-2nhdt\""
Oct 30 18:36:46.147046 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:46.147010    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/47a5f9fe-b519-43a6-9973-4479f504421f-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-xl7v4\" (UID: \"47a5f9fe-b519-43a6-9973-4479f504421f\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-xl7v4"
Oct 30 18:36:46.147241 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:46.147062    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqmp6\" (UniqueName: \"kubernetes.io/projected/47a5f9fe-b519-43a6-9973-4479f504421f-kube-api-access-zqmp6\") pod \"monitoring-plugin-77f4f76594-xl7v4\" (UID: \"47a5f9fe-b519-43a6-9973-4479f504421f\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-xl7v4"
Oct 30 18:36:46.247997 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:46.247954    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/47a5f9fe-b519-43a6-9973-4479f504421f-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-xl7v4\" (UID: \"47a5f9fe-b519-43a6-9973-4479f504421f\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-xl7v4"
Oct 30 18:36:46.248177 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:46.248020    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqmp6\" (UniqueName: \"kubernetes.io/projected/47a5f9fe-b519-43a6-9973-4479f504421f-kube-api-access-zqmp6\") pod \"monitoring-plugin-77f4f76594-xl7v4\" (UID: \"47a5f9fe-b519-43a6-9973-4479f504421f\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-xl7v4"
Oct 30 18:36:46.250693 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:46.250649    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/47a5f9fe-b519-43a6-9973-4479f504421f-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-xl7v4\" (UID: \"47a5f9fe-b519-43a6-9973-4479f504421f\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-xl7v4"
Oct 30 18:36:46.257291 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:46.257265    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqmp6\" (UniqueName: \"kubernetes.io/projected/47a5f9fe-b519-43a6-9973-4479f504421f-kube-api-access-zqmp6\") pod \"monitoring-plugin-77f4f76594-xl7v4\" (UID: \"47a5f9fe-b519-43a6-9973-4479f504421f\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-xl7v4"
Oct 30 18:36:46.331864 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:46.331784    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-xl7v4"
Oct 30 18:36:46.607603 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:46.607569    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-xl7v4"]
Oct 30 18:36:46.611054 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:36:46.611021    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47a5f9fe_b519_43a6_9973_4479f504421f.slice/crio-ee01bc8f2196e66a872b9403c9bb7fc931a9035009823275a26c377530ce5527 WatchSource:0}: Error finding container ee01bc8f2196e66a872b9403c9bb7fc931a9035009823275a26c377530ce5527: Status 404 returned error can't find the container with id ee01bc8f2196e66a872b9403c9bb7fc931a9035009823275a26c377530ce5527
Oct 30 18:36:46.699776 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:46.699738    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-xl7v4" event={"ID":"47a5f9fe-b519-43a6-9973-4479f504421f","Type":"ContainerStarted","Data":"ee01bc8f2196e66a872b9403c9bb7fc931a9035009823275a26c377530ce5527"}
Oct 30 18:36:47.408605 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:36:47.408511    4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d49l" containerName="extractor"
Oct 30 18:36:48.647935 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:48.647908    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-mcc4d"
Oct 30 18:36:48.709636 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:48.709605    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-xl7v4" event={"ID":"47a5f9fe-b519-43a6-9973-4479f504421f","Type":"ContainerStarted","Data":"f13af2c25deb8287ed8167b845551da7b69ef35662981ab824fc06c9fc8f2e37"}
Oct 30 18:36:48.709907 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:48.709887    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-xl7v4"
Oct 30 18:36:48.715471 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:48.715448    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-xl7v4"
Oct 30 18:36:48.725439 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:48.725388    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-xl7v4" podStartSLOduration=2.371472855 podStartE2EDuration="3.725371414s" totalImagesPullingTime="1.353898559s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:36:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:48.724407606 +0000 UTC m=+31.350077571" watchObservedRunningTime="2026-10-30 18:36:48.725371414 +0000 UTC m=+31.351041375"
Oct 30 18:36:52.726545 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:52.726457    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-l98wz" event={"ID":"356d85d5-c222-49e7-916b-bfd94d2438e1","Type":"ContainerStarted","Data":"315717254dd152a93ffe67844cc3f4e09a63e390712ed1d2d1287fa386e6dba5"}
Oct 30 18:36:52.727055 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:52.726726    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-l98wz"
Oct 30 18:36:52.728454 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:52.728395    4525 patch_prober.go:29] interesting pod/downloads-6dd789c8b7-l98wz container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.133.0.13:8080/\": dial tcp 10.133.0.13:8080: connect: connection refused" start-of-body=
Oct 30 18:36:52.728614 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:52.728459    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-6dd789c8b7-l98wz" podUID="356d85d5-c222-49e7-916b-bfd94d2438e1" containerName="download-server" probeResult="failure" output="Get \"http://10.133.0.13:8080/\": dial tcp 10.133.0.13:8080: connect: connection refused"
Oct 30 18:36:52.746559 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:52.746502    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-l98wz" podStartSLOduration=1.384641943 podStartE2EDuration="18.746485833s" totalImagesPullingTime="17.36184389s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:36:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:52.74484959 +0000 UTC m=+35.370519554" watchObservedRunningTime="2026-10-30 18:36:52.746485833 +0000 UTC m=+35.372155798"
Oct 30 18:36:53.744207 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:53.744176    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-l98wz"
Oct 30 18:36:56.664026 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:36:56.663987    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-b89865498-mr5st"
Oct 30 18:37:17.396142 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:37:17.396099    4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d49l" containerName="extractor"
Oct 30 18:37:34.126729 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:37:34.126692    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-q4kks-q2hfp"]
Oct 30 18:37:34.127240 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:37:34.126938    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-q4kks-q2hfp" podUID="9ce43bfa-1e19-4fb5-90ff-42751cdd1a50" containerName="insights-gathering" containerID="cri-o://d52ae502590170ff357847f9726284480792f0b3d0a67699c1544b4db74bca54" gracePeriod=30
Oct 30 18:37:34.192978 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:37:34.192943    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-jjtxw-2lz7h"]
Oct 30 18:37:34.207246 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:37:34.207223    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-jjtxw-2lz7h"
Oct 30 18:37:34.207906 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:37:34.207883    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-jjtxw-2lz7h"]
Oct 30 18:37:34.299583 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:37:34.299551    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/89dc8053-f6fa-4b87-ae63-c757b4a68921-archives-path\") pod \"periodic-gathering-jjtxw-2lz7h\" (UID: \"89dc8053-f6fa-4b87-ae63-c757b4a68921\") " pod="openshift-insights/periodic-gathering-jjtxw-2lz7h"
Oct 30 18:37:34.299774 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:37:34.299592    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89dc8053-f6fa-4b87-ae63-c757b4a68921-service-ca-bundle\") pod \"periodic-gathering-jjtxw-2lz7h\" (UID: \"89dc8053-f6fa-4b87-ae63-c757b4a68921\") " pod="openshift-insights/periodic-gathering-jjtxw-2lz7h"
Oct 30 18:37:34.299774 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:37:34.299619    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zbrz\" (UniqueName: \"kubernetes.io/projected/89dc8053-f6fa-4b87-ae63-c757b4a68921-kube-api-access-7zbrz\") pod \"periodic-gathering-jjtxw-2lz7h\" (UID: \"89dc8053-f6fa-4b87-ae63-c757b4a68921\") " pod="openshift-insights/periodic-gathering-jjtxw-2lz7h"
Oct 30 18:37:34.299887 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:37:34.299799    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89dc8053-f6fa-4b87-ae63-c757b4a68921-trusted-ca-bundle\") pod \"periodic-gathering-jjtxw-2lz7h\" (UID: \"89dc8053-f6fa-4b87-ae63-c757b4a68921\") " pod="openshift-insights/periodic-gathering-jjtxw-2lz7h"
Oct 30 18:37:34.396182 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:37:34.396127    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-q4kks-q2hfp"
Oct 30 18:37:34.400361 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:37:34.400337    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/89dc8053-f6fa-4b87-ae63-c757b4a68921-archives-path\") pod \"periodic-gathering-jjtxw-2lz7h\" (UID: \"89dc8053-f6fa-4b87-ae63-c757b4a68921\") " pod="openshift-insights/periodic-gathering-jjtxw-2lz7h"
Oct 30 18:37:34.400544 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:37:34.400375    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89dc8053-f6fa-4b87-ae63-c757b4a68921-service-ca-bundle\") pod \"periodic-gathering-jjtxw-2lz7h\" (UID: \"89dc8053-f6fa-4b87-ae63-c757b4a68921\") " pod="openshift-insights/periodic-gathering-jjtxw-2lz7h"
Oct 30 18:37:34.400544 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:37:34.400403    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zbrz\" (UniqueName: \"kubernetes.io/projected/89dc8053-f6fa-4b87-ae63-c757b4a68921-kube-api-access-7zbrz\") pod \"periodic-gathering-jjtxw-2lz7h\" (UID: \"89dc8053-f6fa-4b87-ae63-c757b4a68921\") " pod="openshift-insights/periodic-gathering-jjtxw-2lz7h"
Oct 30 18:37:34.400544 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:37:34.400455    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89dc8053-f6fa-4b87-ae63-c757b4a68921-trusted-ca-bundle\") pod \"periodic-gathering-jjtxw-2lz7h\" (UID: \"89dc8053-f6fa-4b87-ae63-c757b4a68921\") " pod="openshift-insights/periodic-gathering-jjtxw-2lz7h"
Oct 30 18:37:34.400778 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:37:34.400765    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/89dc8053-f6fa-4b87-ae63-c757b4a68921-archives-path\") pod \"periodic-gathering-jjtxw-2lz7h\" (UID: \"89dc8053-f6fa-4b87-ae63-c757b4a68921\") " pod="openshift-insights/periodic-gathering-jjtxw-2lz7h"
Oct 30 18:37:34.401051 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:37:34.401028    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89dc8053-f6fa-4b87-ae63-c757b4a68921-service-ca-bundle\") pod \"periodic-gathering-jjtxw-2lz7h\" (UID: \"89dc8053-f6fa-4b87-ae63-c757b4a68921\") " pod="openshift-insights/periodic-gathering-jjtxw-2lz7h"
Oct 30 18:37:34.401379 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:37:34.401358    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89dc8053-f6fa-4b87-ae63-c757b4a68921-trusted-ca-bundle\") pod \"periodic-gathering-jjtxw-2lz7h\" (UID: \"89dc8053-f6fa-4b87-ae63-c757b4a68921\") " pod="openshift-insights/periodic-gathering-jjtxw-2lz7h"
Oct 30 18:37:34.408636 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:37:34.408617    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zbrz\" (UniqueName: \"kubernetes.io/projected/89dc8053-f6fa-4b87-ae63-c757b4a68921-kube-api-access-7zbrz\") pod \"periodic-gathering-jjtxw-2lz7h\" (UID: \"89dc8053-f6fa-4b87-ae63-c757b4a68921\") " pod="openshift-insights/periodic-gathering-jjtxw-2lz7h"
Oct 30 18:37:34.501331 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:37:34.501305    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ce43bfa-1e19-4fb5-90ff-42751cdd1a50-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ce43bfa-1e19-4fb5-90ff-42751cdd1a50-trusted-ca-bundle\") pod \"9ce43bfa-1e19-4fb5-90ff-42751cdd1a50\" (UID: \"9ce43bfa-1e19-4fb5-90ff-42751cdd1a50\") "
Oct 30 18:37:34.501480 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:37:34.501368    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ce43bfa-1e19-4fb5-90ff-42751cdd1a50-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9ce43bfa-1e19-4fb5-90ff-42751cdd1a50-archives-path\") pod \"9ce43bfa-1e19-4fb5-90ff-42751cdd1a50\" (UID: \"9ce43bfa-1e19-4fb5-90ff-42751cdd1a50\") "
Oct 30 18:37:34.501480 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:37:34.501395    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ce43bfa-1e19-4fb5-90ff-42751cdd1a50-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ce43bfa-1e19-4fb5-90ff-42751cdd1a50-service-ca-bundle\") pod \"9ce43bfa-1e19-4fb5-90ff-42751cdd1a50\" (UID: \"9ce43bfa-1e19-4fb5-90ff-42751cdd1a50\") "
Oct 30 18:37:34.501480 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:37:34.501437    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ce43bfa-1e19-4fb5-90ff-42751cdd1a50-kube-api-access-862ff\" (UniqueName: \"kubernetes.io/projected/9ce43bfa-1e19-4fb5-90ff-42751cdd1a50-kube-api-access-862ff\") pod \"9ce43bfa-1e19-4fb5-90ff-42751cdd1a50\" (UID: \"9ce43bfa-1e19-4fb5-90ff-42751cdd1a50\") "
Oct 30 18:37:34.501766 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:37:34.501742    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ce43bfa-1e19-4fb5-90ff-42751cdd1a50-trusted-ca-bundle" pod "9ce43bfa-1e19-4fb5-90ff-42751cdd1a50" (UID: "9ce43bfa-1e19-4fb5-90ff-42751cdd1a50"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:37:34.501834 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:37:34.501763    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ce43bfa-1e19-4fb5-90ff-42751cdd1a50-service-ca-bundle" pod "9ce43bfa-1e19-4fb5-90ff-42751cdd1a50" (UID: "9ce43bfa-1e19-4fb5-90ff-42751cdd1a50"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:37:34.501834 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:37:34.501748    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ce43bfa-1e19-4fb5-90ff-42751cdd1a50-archives-path" pod "9ce43bfa-1e19-4fb5-90ff-42751cdd1a50" (UID: "9ce43bfa-1e19-4fb5-90ff-42751cdd1a50"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:37:34.503719 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:37:34.503701    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ce43bfa-1e19-4fb5-90ff-42751cdd1a50-kube-api-access-862ff" pod "9ce43bfa-1e19-4fb5-90ff-42751cdd1a50" (UID: "9ce43bfa-1e19-4fb5-90ff-42751cdd1a50"). InnerVolumeSpecName "kube-api-access-862ff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:37:34.524774 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:37:34.524749    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-jjtxw-2lz7h"
Oct 30 18:37:34.604120 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:37:34.603112    4525 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ce43bfa-1e19-4fb5-90ff-42751cdd1a50-service-ca-bundle\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:37:34.604120 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:37:34.603368    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-862ff\" (UniqueName: \"kubernetes.io/projected/9ce43bfa-1e19-4fb5-90ff-42751cdd1a50-kube-api-access-862ff\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:37:34.604120 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:37:34.603389    4525 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ce43bfa-1e19-4fb5-90ff-42751cdd1a50-trusted-ca-bundle\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:37:34.604120 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:37:34.603405    4525 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9ce43bfa-1e19-4fb5-90ff-42751cdd1a50-archives-path\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:37:34.811419 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:37:34.811337    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-jjtxw-2lz7h"]
Oct 30 18:37:34.817949 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:37:34.817923    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89dc8053_f6fa_4b87_ae63_c757b4a68921.slice/crio-8bda67e5fc306cc448ffa9ce5cd30057622872bea7158222ae8627d49c8ac405 WatchSource:0}: Error finding container 8bda67e5fc306cc448ffa9ce5cd30057622872bea7158222ae8627d49c8ac405: Status 404 returned error can't find the container with id 8bda67e5fc306cc448ffa9ce5cd30057622872bea7158222ae8627d49c8ac405
Oct 30 18:37:34.862902 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:37:34.862875    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-jjtxw-2lz7h" event={"ID":"89dc8053-f6fa-4b87-ae63-c757b4a68921","Type":"ContainerStarted","Data":"8bda67e5fc306cc448ffa9ce5cd30057622872bea7158222ae8627d49c8ac405"}
Oct 30 18:37:34.864126 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:37:34.864102    4525 generic.go:356] "Generic (PLEG): container finished" podID="9ce43bfa-1e19-4fb5-90ff-42751cdd1a50" containerID="d52ae502590170ff357847f9726284480792f0b3d0a67699c1544b4db74bca54" exitCode=2
Oct 30 18:37:34.864216 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:37:34.864144    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-q4kks-q2hfp" event={"ID":"9ce43bfa-1e19-4fb5-90ff-42751cdd1a50","Type":"ContainerDied","Data":"d52ae502590170ff357847f9726284480792f0b3d0a67699c1544b4db74bca54"}
Oct 30 18:37:34.864216 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:37:34.864154    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-q4kks-q2hfp"
Oct 30 18:37:34.864216 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:37:34.864176    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-q4kks-q2hfp" event={"ID":"9ce43bfa-1e19-4fb5-90ff-42751cdd1a50","Type":"ContainerDied","Data":"1fe07a3980476efe57f370a3f8ffd99d740abe566fde8bf60f8a712e032faac7"}
Oct 30 18:37:34.864216 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:37:34.864206    4525 scope.go:122] "RemoveContainer" containerID="d52ae502590170ff357847f9726284480792f0b3d0a67699c1544b4db74bca54"
Oct 30 18:37:34.872335 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:37:34.872314    4525 scope.go:122] "RemoveContainer" containerID="d52ae502590170ff357847f9726284480792f0b3d0a67699c1544b4db74bca54"
Oct 30 18:37:34.872622 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:37:34.872598    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d52ae502590170ff357847f9726284480792f0b3d0a67699c1544b4db74bca54\": container with ID starting with d52ae502590170ff357847f9726284480792f0b3d0a67699c1544b4db74bca54 not found: ID does not exist" containerID="d52ae502590170ff357847f9726284480792f0b3d0a67699c1544b4db74bca54"
Oct 30 18:37:34.872671 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:37:34.872630    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d52ae502590170ff357847f9726284480792f0b3d0a67699c1544b4db74bca54"} err="failed to get container status \"d52ae502590170ff357847f9726284480792f0b3d0a67699c1544b4db74bca54\": rpc error: code = NotFound desc = could not find container \"d52ae502590170ff357847f9726284480792f0b3d0a67699c1544b4db74bca54\": container with ID starting with d52ae502590170ff357847f9726284480792f0b3d0a67699c1544b4db74bca54 not found: ID does not exist"
Oct 30 18:37:34.885637 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:37:34.885612    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-q4kks-q2hfp"]
Oct 30 18:37:34.890411 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:37:34.890390    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-q4kks-q2hfp"]
Oct 30 18:37:35.541105 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:37:35.541074    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ce43bfa-1e19-4fb5-90ff-42751cdd1a50" path="/var/lib/kubelet/pods/9ce43bfa-1e19-4fb5-90ff-42751cdd1a50/volumes"
Oct 30 18:37:35.869980 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:37:35.869945    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-jjtxw-2lz7h" event={"ID":"89dc8053-f6fa-4b87-ae63-c757b4a68921","Type":"ContainerStarted","Data":"40f76e66e88cfbb67777e7b0b663c08b2bc6b970109b42385eedadd917846c0c"}
Oct 30 18:37:35.886947 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:37:35.886898    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-jjtxw-2lz7h" podStartSLOduration=1.886885953 podStartE2EDuration="1.886885953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:37:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:37:35.886344634 +0000 UTC m=+78.512014598" watchObservedRunningTime="2026-10-30 18:37:35.886885953 +0000 UTC m=+78.512555917"
Oct 30 18:37:47.395229 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:37:47.395191    4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d49l" containerName="extractor"
Oct 30 18:38:17.395331 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:38:17.395291    4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d49l" containerName="extractor"
Oct 30 18:38:47.395785 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:38:47.395745    4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d49l" containerName="extractor"
Oct 30 18:38:58.128353 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:38:58.128319    4525 generic.go:356] "Generic (PLEG): container finished" podID="89dc8053-f6fa-4b87-ae63-c757b4a68921" containerID="40f76e66e88cfbb67777e7b0b663c08b2bc6b970109b42385eedadd917846c0c" exitCode=0
Oct 30 18:38:58.128753 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:38:58.128395    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-jjtxw-2lz7h" event={"ID":"89dc8053-f6fa-4b87-ae63-c757b4a68921","Type":"ContainerDied","Data":"40f76e66e88cfbb67777e7b0b663c08b2bc6b970109b42385eedadd917846c0c"}
Oct 30 18:38:59.252068 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:38:59.252044    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-jjtxw-2lz7h"
Oct 30 18:38:59.303875 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:38:59.303849    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89dc8053-f6fa-4b87-ae63-c757b4a68921-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/89dc8053-f6fa-4b87-ae63-c757b4a68921-archives-path\") pod \"89dc8053-f6fa-4b87-ae63-c757b4a68921\" (UID: \"89dc8053-f6fa-4b87-ae63-c757b4a68921\") "
Oct 30 18:38:59.303999 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:38:59.303893    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89dc8053-f6fa-4b87-ae63-c757b4a68921-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89dc8053-f6fa-4b87-ae63-c757b4a68921-service-ca-bundle\") pod \"89dc8053-f6fa-4b87-ae63-c757b4a68921\" (UID: \"89dc8053-f6fa-4b87-ae63-c757b4a68921\") "
Oct 30 18:38:59.303999 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:38:59.303914    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89dc8053-f6fa-4b87-ae63-c757b4a68921-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89dc8053-f6fa-4b87-ae63-c757b4a68921-trusted-ca-bundle\") pod \"89dc8053-f6fa-4b87-ae63-c757b4a68921\" (UID: \"89dc8053-f6fa-4b87-ae63-c757b4a68921\") "
Oct 30 18:38:59.303999 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:38:59.303964    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89dc8053-f6fa-4b87-ae63-c757b4a68921-kube-api-access-7zbrz\" (UniqueName: \"kubernetes.io/projected/89dc8053-f6fa-4b87-ae63-c757b4a68921-kube-api-access-7zbrz\") pod \"89dc8053-f6fa-4b87-ae63-c757b4a68921\" (UID: \"89dc8053-f6fa-4b87-ae63-c757b4a68921\") "
Oct 30 18:38:59.304216 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:38:59.304194    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89dc8053-f6fa-4b87-ae63-c757b4a68921-archives-path" pod "89dc8053-f6fa-4b87-ae63-c757b4a68921" (UID: "89dc8053-f6fa-4b87-ae63-c757b4a68921"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:38:59.304262 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:38:59.304243    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89dc8053-f6fa-4b87-ae63-c757b4a68921-service-ca-bundle" pod "89dc8053-f6fa-4b87-ae63-c757b4a68921" (UID: "89dc8053-f6fa-4b87-ae63-c757b4a68921"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:38:59.304295 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:38:59.304277    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89dc8053-f6fa-4b87-ae63-c757b4a68921-trusted-ca-bundle" pod "89dc8053-f6fa-4b87-ae63-c757b4a68921" (UID: "89dc8053-f6fa-4b87-ae63-c757b4a68921"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:38:59.306136 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:38:59.306116    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89dc8053-f6fa-4b87-ae63-c757b4a68921-kube-api-access-7zbrz" pod "89dc8053-f6fa-4b87-ae63-c757b4a68921" (UID: "89dc8053-f6fa-4b87-ae63-c757b4a68921"). InnerVolumeSpecName "kube-api-access-7zbrz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:38:59.404808 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:38:59.404737    4525 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89dc8053-f6fa-4b87-ae63-c757b4a68921-service-ca-bundle\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:38:59.404808 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:38:59.404765    4525 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89dc8053-f6fa-4b87-ae63-c757b4a68921-trusted-ca-bundle\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:38:59.404808 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:38:59.404775    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zbrz\" (UniqueName: \"kubernetes.io/projected/89dc8053-f6fa-4b87-ae63-c757b4a68921-kube-api-access-7zbrz\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:38:59.404808 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:38:59.404785    4525 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/89dc8053-f6fa-4b87-ae63-c757b4a68921-archives-path\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:39:00.136443 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:39:00.136404    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-jjtxw-2lz7h" event={"ID":"89dc8053-f6fa-4b87-ae63-c757b4a68921","Type":"ContainerDied","Data":"8bda67e5fc306cc448ffa9ce5cd30057622872bea7158222ae8627d49c8ac405"}
Oct 30 18:39:00.136619 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:39:00.136454    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-jjtxw-2lz7h"
Oct 30 18:39:00.136619 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:39:00.136487    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8bda67e5fc306cc448ffa9ce5cd30057622872bea7158222ae8627d49c8ac405"
Oct 30 18:39:17.395904 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:39:17.395866    4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d49l" containerName="extractor"
Oct 30 18:39:47.395583 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:39:47.395539    4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d49l" containerName="extractor"
Oct 30 18:40:17.397200 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:40:17.397163    4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d49l" containerName="extractor"
Oct 30 18:40:47.395524 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:40:47.395479    4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d49l" containerName="extractor"
Oct 30 18:41:17.395600 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:41:17.395553    4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d49l" containerName="extractor"
Oct 30 18:41:47.395317 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:41:47.395277    4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d49l" containerName="extractor"
Oct 30 18:42:06.449101 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:06.449065    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5h8fw"]
Oct 30 18:42:06.449500 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:06.449295    4525 cpu_manager.go:414] "containerMap: removing container" podUID="9ce43bfa-1e19-4fb5-90ff-42751cdd1a50" containerName="insights-gathering"
Oct 30 18:42:06.449500 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:06.449306    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ce43bfa-1e19-4fb5-90ff-42751cdd1a50" containerName="insights-gathering"
Oct 30 18:42:06.449500 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:06.449327    4525 cpu_manager.go:414] "containerMap: removing container" podUID="89dc8053-f6fa-4b87-ae63-c757b4a68921" containerName="insights-gathering"
Oct 30 18:42:06.449500 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:06.449332    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89dc8053-f6fa-4b87-ae63-c757b4a68921" containerName="insights-gathering"
Oct 30 18:42:06.449500 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:06.449406    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ce43bfa-1e19-4fb5-90ff-42751cdd1a50" containerName="insights-gathering"
Oct 30 18:42:06.449500 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:06.449417    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="89dc8053-f6fa-4b87-ae63-c757b4a68921" containerName="insights-gathering"
Oct 30 18:42:06.452302 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:06.452286    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5h8fw"
Oct 30 18:42:06.457771 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:06.457750    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 30 18:42:06.457930 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:06.457911    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-jr79f\""
Oct 30 18:42:06.458308 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:06.458009    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 30 18:42:06.470053 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:06.470027    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5h8fw"]
Oct 30 18:42:06.581113 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:06.581075    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e2853c1-7992-446c-a335-2d2e63565ff3-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5h8fw\" (UID: \"5e2853c1-7992-446c-a335-2d2e63565ff3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5h8fw"
Oct 30 18:42:06.581113 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:06.581112    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxgcs\" (UniqueName: \"kubernetes.io/projected/5e2853c1-7992-446c-a335-2d2e63565ff3-kube-api-access-mxgcs\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5h8fw\" (UID: \"5e2853c1-7992-446c-a335-2d2e63565ff3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5h8fw"
Oct 30 18:42:06.581370 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:06.581234    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e2853c1-7992-446c-a335-2d2e63565ff3-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5h8fw\" (UID: \"5e2853c1-7992-446c-a335-2d2e63565ff3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5h8fw"
Oct 30 18:42:06.634445 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:06.634404    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7125bzv"]
Oct 30 18:42:06.637603 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:06.637581    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7125bzv"
Oct 30 18:42:06.641663 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:06.641640    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-s7c2h\""
Oct 30 18:42:06.646535 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:06.646511    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7125bzv"]
Oct 30 18:42:06.681623 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:06.681591    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e2853c1-7992-446c-a335-2d2e63565ff3-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5h8fw\" (UID: \"5e2853c1-7992-446c-a335-2d2e63565ff3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5h8fw"
Oct 30 18:42:06.681807 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:06.681633    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e2853c1-7992-446c-a335-2d2e63565ff3-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5h8fw\" (UID: \"5e2853c1-7992-446c-a335-2d2e63565ff3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5h8fw"
Oct 30 18:42:06.681807 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:06.681652    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxgcs\" (UniqueName: \"kubernetes.io/projected/5e2853c1-7992-446c-a335-2d2e63565ff3-kube-api-access-mxgcs\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5h8fw\" (UID: \"5e2853c1-7992-446c-a335-2d2e63565ff3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5h8fw"
Oct 30 18:42:06.682117 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:06.682093    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e2853c1-7992-446c-a335-2d2e63565ff3-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5h8fw\" (UID: \"5e2853c1-7992-446c-a335-2d2e63565ff3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5h8fw"
Oct 30 18:42:06.682155 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:06.682100    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e2853c1-7992-446c-a335-2d2e63565ff3-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5h8fw\" (UID: \"5e2853c1-7992-446c-a335-2d2e63565ff3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5h8fw"
Oct 30 18:42:06.689967 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:06.689942    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxgcs\" (UniqueName: \"kubernetes.io/projected/5e2853c1-7992-446c-a335-2d2e63565ff3-kube-api-access-mxgcs\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5h8fw\" (UID: \"5e2853c1-7992-446c-a335-2d2e63565ff3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5h8fw"
Oct 30 18:42:06.767130 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:06.767046    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5h8fw"
Oct 30 18:42:06.782320 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:06.782296    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f78d8915-08e0-4559-8cfe-a63505f35f46-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7125bzv\" (UID: \"f78d8915-08e0-4559-8cfe-a63505f35f46\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7125bzv"
Oct 30 18:42:06.782442 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:06.782377    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f78d8915-08e0-4559-8cfe-a63505f35f46-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7125bzv\" (UID: \"f78d8915-08e0-4559-8cfe-a63505f35f46\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7125bzv"
Oct 30 18:42:06.782536 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:06.782506    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pkls\" (UniqueName: \"kubernetes.io/projected/f78d8915-08e0-4559-8cfe-a63505f35f46-kube-api-access-9pkls\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7125bzv\" (UID: \"f78d8915-08e0-4559-8cfe-a63505f35f46\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7125bzv"
Oct 30 18:42:06.883397 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:06.883362    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f78d8915-08e0-4559-8cfe-a63505f35f46-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7125bzv\" (UID: \"f78d8915-08e0-4559-8cfe-a63505f35f46\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7125bzv"
Oct 30 18:42:06.883589 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:06.883413    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pkls\" (UniqueName: \"kubernetes.io/projected/f78d8915-08e0-4559-8cfe-a63505f35f46-kube-api-access-9pkls\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7125bzv\" (UID: \"f78d8915-08e0-4559-8cfe-a63505f35f46\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7125bzv"
Oct 30 18:42:06.883589 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:06.883444    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f78d8915-08e0-4559-8cfe-a63505f35f46-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7125bzv\" (UID: \"f78d8915-08e0-4559-8cfe-a63505f35f46\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7125bzv"
Oct 30 18:42:06.883880 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:06.883857    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f78d8915-08e0-4559-8cfe-a63505f35f46-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7125bzv\" (UID: \"f78d8915-08e0-4559-8cfe-a63505f35f46\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7125bzv"
Oct 30 18:42:06.883941 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:06.883881    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f78d8915-08e0-4559-8cfe-a63505f35f46-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7125bzv\" (UID: \"f78d8915-08e0-4559-8cfe-a63505f35f46\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7125bzv"
Oct 30 18:42:06.893122 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:06.893097    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pkls\" (UniqueName: \"kubernetes.io/projected/f78d8915-08e0-4559-8cfe-a63505f35f46-kube-api-access-9pkls\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7125bzv\" (UID: \"f78d8915-08e0-4559-8cfe-a63505f35f46\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7125bzv"
Oct 30 18:42:06.953722 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:06.953675    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7125bzv"
Oct 30 18:42:07.038929 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:07.038841    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5h8fw"]
Oct 30 18:42:07.041881 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:42:07.041847    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e2853c1_7992_446c_a335_2d2e63565ff3.slice/crio-68036ae21b5d6e24c548ac5ff0870b48838f9540cf9e5a81c6c6064d41b4ceae WatchSource:0}: Error finding container 68036ae21b5d6e24c548ac5ff0870b48838f9540cf9e5a81c6c6064d41b4ceae: Status 404 returned error can't find the container with id 68036ae21b5d6e24c548ac5ff0870b48838f9540cf9e5a81c6c6064d41b4ceae
Oct 30 18:42:07.043853 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:07.043836    4525 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 18:42:07.228287 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:07.228255    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7125bzv"]
Oct 30 18:42:07.231925 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:42:07.231893    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf78d8915_08e0_4559_8cfe_a63505f35f46.slice/crio-a74851a69b721cf96641286e890e204656631c66d61996cb3d4cbaded9005e17 WatchSource:0}: Error finding container a74851a69b721cf96641286e890e204656631c66d61996cb3d4cbaded9005e17: Status 404 returned error can't find the container with id a74851a69b721cf96641286e890e204656631c66d61996cb3d4cbaded9005e17
Oct 30 18:42:07.724625 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:07.724579    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7125bzv" event={"ID":"f78d8915-08e0-4559-8cfe-a63505f35f46","Type":"ContainerStarted","Data":"a74851a69b721cf96641286e890e204656631c66d61996cb3d4cbaded9005e17"}
Oct 30 18:42:07.726405 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:07.726371    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5h8fw" event={"ID":"5e2853c1-7992-446c-a335-2d2e63565ff3","Type":"ContainerStarted","Data":"68036ae21b5d6e24c548ac5ff0870b48838f9540cf9e5a81c6c6064d41b4ceae"}
Oct 30 18:42:08.762863 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:08.762822    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644sn5hk"]
Oct 30 18:42:08.767300 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:08.766916    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644sn5hk"
Oct 30 18:42:08.776936 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:08.776909    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-wxnbc\""
Oct 30 18:42:08.779365 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:08.779339    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644sn5hk"]
Oct 30 18:42:08.900320 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:08.900278    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tmdq\" (UniqueName: \"kubernetes.io/projected/dba7dabb-2faf-4472-9b77-786f5c61decb-kube-api-access-5tmdq\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644sn5hk\" (UID: \"dba7dabb-2faf-4472-9b77-786f5c61decb\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644sn5hk"
Oct 30 18:42:08.900518 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:08.900345    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dba7dabb-2faf-4472-9b77-786f5c61decb-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644sn5hk\" (UID: \"dba7dabb-2faf-4472-9b77-786f5c61decb\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644sn5hk"
Oct 30 18:42:08.900518 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:08.900442    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dba7dabb-2faf-4472-9b77-786f5c61decb-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644sn5hk\" (UID: \"dba7dabb-2faf-4472-9b77-786f5c61decb\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644sn5hk"
Oct 30 18:42:09.001769 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:09.001724    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dba7dabb-2faf-4472-9b77-786f5c61decb-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644sn5hk\" (UID: \"dba7dabb-2faf-4472-9b77-786f5c61decb\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644sn5hk"
Oct 30 18:42:09.001962 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:09.001833    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tmdq\" (UniqueName: \"kubernetes.io/projected/dba7dabb-2faf-4472-9b77-786f5c61decb-kube-api-access-5tmdq\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644sn5hk\" (UID: \"dba7dabb-2faf-4472-9b77-786f5c61decb\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644sn5hk"
Oct 30 18:42:09.001962 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:09.001876    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dba7dabb-2faf-4472-9b77-786f5c61decb-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644sn5hk\" (UID: \"dba7dabb-2faf-4472-9b77-786f5c61decb\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644sn5hk"
Oct 30 18:42:09.002173 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:09.002154    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dba7dabb-2faf-4472-9b77-786f5c61decb-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644sn5hk\" (UID: \"dba7dabb-2faf-4472-9b77-786f5c61decb\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644sn5hk"
Oct 30 18:42:09.002212 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:09.002182    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dba7dabb-2faf-4472-9b77-786f5c61decb-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644sn5hk\" (UID: \"dba7dabb-2faf-4472-9b77-786f5c61decb\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644sn5hk"
Oct 30 18:42:09.013003 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:09.012941    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tmdq\" (UniqueName: \"kubernetes.io/projected/dba7dabb-2faf-4472-9b77-786f5c61decb-kube-api-access-5tmdq\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644sn5hk\" (UID: \"dba7dabb-2faf-4472-9b77-786f5c61decb\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644sn5hk"
Oct 30 18:42:09.086863 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:09.086810    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644sn5hk"
Oct 30 18:42:09.382481 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:09.382449    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644sn5hk"]
Oct 30 18:42:09.386999 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:42:09.386951    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddba7dabb_2faf_4472_9b77_786f5c61decb.slice/crio-63b04b979693fc01dc2a73b7efed7488d8216b4e9a551eba95da66bbcb5a4331 WatchSource:0}: Error finding container 63b04b979693fc01dc2a73b7efed7488d8216b4e9a551eba95da66bbcb5a4331: Status 404 returned error can't find the container with id 63b04b979693fc01dc2a73b7efed7488d8216b4e9a551eba95da66bbcb5a4331
Oct 30 18:42:09.681904 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:09.681830    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2gq6q"]
Oct 30 18:42:09.686423 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:09.686399    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2gq6q"
Oct 30 18:42:09.694158 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:09.694135    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-6mqjj\""
Oct 30 18:42:09.694437 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:09.694407    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2gq6q"]
Oct 30 18:42:09.736382 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:09.736333    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644sn5hk" event={"ID":"dba7dabb-2faf-4472-9b77-786f5c61decb","Type":"ContainerStarted","Data":"63b04b979693fc01dc2a73b7efed7488d8216b4e9a551eba95da66bbcb5a4331"}
Oct 30 18:42:09.807796 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:09.807753    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rm9w\" (UniqueName: \"kubernetes.io/projected/5332c799-1c59-4995-8c40-146f82223d75-kube-api-access-6rm9w\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2gq6q\" (UID: \"5332c799-1c59-4995-8c40-146f82223d75\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2gq6q"
Oct 30 18:42:09.808372 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:09.807811    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5332c799-1c59-4995-8c40-146f82223d75-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2gq6q\" (UID: \"5332c799-1c59-4995-8c40-146f82223d75\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2gq6q"
Oct 30 18:42:09.808372 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:09.807850    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5332c799-1c59-4995-8c40-146f82223d75-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2gq6q\" (UID: \"5332c799-1c59-4995-8c40-146f82223d75\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2gq6q"
Oct 30 18:42:09.835020 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:09.834980    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c9w86"]
Oct 30 18:42:09.838670 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:09.838646    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c9w86"
Oct 30 18:42:09.842538 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:09.842502    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-lvfwl\""
Oct 30 18:42:09.850985 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:09.850911    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c9w86"]
Oct 30 18:42:09.901096 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:09.901062    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chnnzp"]
Oct 30 18:42:09.904871 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:09.904848    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chnnzp"
Oct 30 18:42:09.908245 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:09.908208    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rm9w\" (UniqueName: \"kubernetes.io/projected/5332c799-1c59-4995-8c40-146f82223d75-kube-api-access-6rm9w\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2gq6q\" (UID: \"5332c799-1c59-4995-8c40-146f82223d75\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2gq6q"
Oct 30 18:42:09.908364 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:09.908250    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5332c799-1c59-4995-8c40-146f82223d75-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2gq6q\" (UID: \"5332c799-1c59-4995-8c40-146f82223d75\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2gq6q"
Oct 30 18:42:09.908364 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:09.908289    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5332c799-1c59-4995-8c40-146f82223d75-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2gq6q\" (UID: \"5332c799-1c59-4995-8c40-146f82223d75\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2gq6q"
Oct 30 18:42:09.908757 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:09.908735    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5332c799-1c59-4995-8c40-146f82223d75-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2gq6q\" (UID: \"5332c799-1c59-4995-8c40-146f82223d75\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2gq6q"
Oct 30 18:42:09.909250 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:09.909225    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5332c799-1c59-4995-8c40-146f82223d75-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2gq6q\" (UID: \"5332c799-1c59-4995-8c40-146f82223d75\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2gq6q"
Oct 30 18:42:09.909543 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:09.909525    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-ggfjv\""
Oct 30 18:42:09.915760 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:09.915310    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chnnzp"]
Oct 30 18:42:09.930079 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:09.930037    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rm9w\" (UniqueName: \"kubernetes.io/projected/5332c799-1c59-4995-8c40-146f82223d75-kube-api-access-6rm9w\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2gq6q\" (UID: \"5332c799-1c59-4995-8c40-146f82223d75\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2gq6q"
Oct 30 18:42:09.988012 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:09.987945    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lr5h6"]
Oct 30 18:42:10.006892 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:10.006656    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2gq6q"
Oct 30 18:42:10.007846 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:10.007820    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lr5h6"]
Oct 30 18:42:10.008066 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:10.008032    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lr5h6"
Oct 30 18:42:10.008776 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:10.008746    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9eb75e76-3366-457a-972f-e445bb34328e-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c9w86\" (UID: \"9eb75e76-3366-457a-972f-e445bb34328e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c9w86"
Oct 30 18:42:10.008868 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:10.008782    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nw8w\" (UniqueName: \"kubernetes.io/projected/54116276-13c3-4e07-b8e9-65943636d1e5-kube-api-access-8nw8w\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chnnzp\" (UID: \"54116276-13c3-4e07-b8e9-65943636d1e5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chnnzp"
Oct 30 18:42:10.008868 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:10.008828    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/54116276-13c3-4e07-b8e9-65943636d1e5-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chnnzp\" (UID: \"54116276-13c3-4e07-b8e9-65943636d1e5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chnnzp"
Oct 30 18:42:10.008868 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:10.008854    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/54116276-13c3-4e07-b8e9-65943636d1e5-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chnnzp\" (UID: \"54116276-13c3-4e07-b8e9-65943636d1e5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chnnzp"
Oct 30 18:42:10.009040 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:10.008894    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9eb75e76-3366-457a-972f-e445bb34328e-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c9w86\" (UID: \"9eb75e76-3366-457a-972f-e445bb34328e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c9w86"
Oct 30 18:42:10.009040 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:10.008975    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtpvn\" (UniqueName: \"kubernetes.io/projected/9eb75e76-3366-457a-972f-e445bb34328e-kube-api-access-jtpvn\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c9w86\" (UID: \"9eb75e76-3366-457a-972f-e445bb34328e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c9w86"
Oct 30 18:42:10.013221 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:10.013201    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-j85rw\""
Oct 30 18:42:10.109643 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:10.109613    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtpvn\" (UniqueName: \"kubernetes.io/projected/9eb75e76-3366-457a-972f-e445bb34328e-kube-api-access-jtpvn\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c9w86\" (UID: \"9eb75e76-3366-457a-972f-e445bb34328e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c9w86"
Oct 30 18:42:10.109811 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:10.109652    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9eb75e76-3366-457a-972f-e445bb34328e-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c9w86\" (UID: \"9eb75e76-3366-457a-972f-e445bb34328e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c9w86"
Oct 30 18:42:10.109811 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:10.109703    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nw8w\" (UniqueName: \"kubernetes.io/projected/54116276-13c3-4e07-b8e9-65943636d1e5-kube-api-access-8nw8w\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chnnzp\" (UID: \"54116276-13c3-4e07-b8e9-65943636d1e5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chnnzp"
Oct 30 18:42:10.109811 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:10.109737    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aab475f7-3a68-4c19-930b-b055767366a9-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lr5h6\" (UID: \"aab475f7-3a68-4c19-930b-b055767366a9\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lr5h6"
Oct 30 18:42:10.109811 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:10.109779    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/54116276-13c3-4e07-b8e9-65943636d1e5-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chnnzp\" (UID: \"54116276-13c3-4e07-b8e9-65943636d1e5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chnnzp"
Oct 30 18:42:10.109811 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:10.109801    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aab475f7-3a68-4c19-930b-b055767366a9-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lr5h6\" (UID: \"aab475f7-3a68-4c19-930b-b055767366a9\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lr5h6"
Oct 30 18:42:10.110082 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:10.109827    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/54116276-13c3-4e07-b8e9-65943636d1e5-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chnnzp\" (UID: \"54116276-13c3-4e07-b8e9-65943636d1e5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chnnzp"
Oct 30 18:42:10.110082 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:10.109868    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9eb75e76-3366-457a-972f-e445bb34328e-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c9w86\" (UID: \"9eb75e76-3366-457a-972f-e445bb34328e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c9w86"
Oct 30 18:42:10.110082 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:10.109891    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpqg2\" (UniqueName: \"kubernetes.io/projected/aab475f7-3a68-4c19-930b-b055767366a9-kube-api-access-xpqg2\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lr5h6\" (UID: \"aab475f7-3a68-4c19-930b-b055767366a9\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lr5h6"
Oct 30 18:42:10.110082 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:10.110030    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9eb75e76-3366-457a-972f-e445bb34328e-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c9w86\" (UID: \"9eb75e76-3366-457a-972f-e445bb34328e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c9w86"
Oct 30 18:42:10.110282 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:10.110265    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/54116276-13c3-4e07-b8e9-65943636d1e5-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chnnzp\" (UID: \"54116276-13c3-4e07-b8e9-65943636d1e5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chnnzp"
Oct 30 18:42:10.110453 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:10.110418    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9eb75e76-3366-457a-972f-e445bb34328e-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c9w86\" (UID: \"9eb75e76-3366-457a-972f-e445bb34328e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c9w86"
Oct 30 18:42:10.110453 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:10.110428    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/54116276-13c3-4e07-b8e9-65943636d1e5-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chnnzp\" (UID: \"54116276-13c3-4e07-b8e9-65943636d1e5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chnnzp"
Oct 30 18:42:10.119920 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:10.119894    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nw8w\" (UniqueName: \"kubernetes.io/projected/54116276-13c3-4e07-b8e9-65943636d1e5-kube-api-access-8nw8w\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chnnzp\" (UID: \"54116276-13c3-4e07-b8e9-65943636d1e5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chnnzp"
Oct 30 18:42:10.120823 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:10.120781    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtpvn\" (UniqueName: \"kubernetes.io/projected/9eb75e76-3366-457a-972f-e445bb34328e-kube-api-access-jtpvn\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c9w86\" (UID: \"9eb75e76-3366-457a-972f-e445bb34328e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c9w86"
Oct 30 18:42:10.162694 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:10.162658    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c9w86"
Oct 30 18:42:10.211163 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:10.211123    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aab475f7-3a68-4c19-930b-b055767366a9-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lr5h6\" (UID: \"aab475f7-3a68-4c19-930b-b055767366a9\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lr5h6"
Oct 30 18:42:10.211943 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:10.211812    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpqg2\" (UniqueName: \"kubernetes.io/projected/aab475f7-3a68-4c19-930b-b055767366a9-kube-api-access-xpqg2\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lr5h6\" (UID: \"aab475f7-3a68-4c19-930b-b055767366a9\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lr5h6"
Oct 30 18:42:10.211943 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:10.211702    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aab475f7-3a68-4c19-930b-b055767366a9-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lr5h6\" (UID: \"aab475f7-3a68-4c19-930b-b055767366a9\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lr5h6"
Oct 30 18:42:10.212414 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:10.212384    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aab475f7-3a68-4c19-930b-b055767366a9-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lr5h6\" (UID: \"aab475f7-3a68-4c19-930b-b055767366a9\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lr5h6"
Oct 30 18:42:10.212727 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:10.212707    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aab475f7-3a68-4c19-930b-b055767366a9-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lr5h6\" (UID: \"aab475f7-3a68-4c19-930b-b055767366a9\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lr5h6"
Oct 30 18:42:10.221589 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:10.221562    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpqg2\" (UniqueName: \"kubernetes.io/projected/aab475f7-3a68-4c19-930b-b055767366a9-kube-api-access-xpqg2\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lr5h6\" (UID: \"aab475f7-3a68-4c19-930b-b055767366a9\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lr5h6"
Oct 30 18:42:10.238473 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:10.238420    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chnnzp"
Oct 30 18:42:10.306875 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:10.306837    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2gq6q"]
Oct 30 18:42:10.312300 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:42:10.312271    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5332c799_1c59_4995_8c40_146f82223d75.slice/crio-637157f9a8e5adbecaefef3f8676f2a7c5f7780ae0cabc55ce4b2e492a921cb5 WatchSource:0}: Error finding container 637157f9a8e5adbecaefef3f8676f2a7c5f7780ae0cabc55ce4b2e492a921cb5: Status 404 returned error can't find the container with id 637157f9a8e5adbecaefef3f8676f2a7c5f7780ae0cabc55ce4b2e492a921cb5
Oct 30 18:42:10.344432 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:10.344406    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lr5h6"
Oct 30 18:42:10.462701 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:10.462653    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c9w86"]
Oct 30 18:42:10.466985 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:42:10.466951    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9eb75e76_3366_457a_972f_e445bb34328e.slice/crio-14fb4c9862c83f4a5318029de9617565ba81c2dfa1d6bca6cfa3c51ef557a287 WatchSource:0}: Error finding container 14fb4c9862c83f4a5318029de9617565ba81c2dfa1d6bca6cfa3c51ef557a287: Status 404 returned error can't find the container with id 14fb4c9862c83f4a5318029de9617565ba81c2dfa1d6bca6cfa3c51ef557a287
Oct 30 18:42:10.548888 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:10.548807    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chnnzp"]
Oct 30 18:42:10.552889 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:42:10.552858    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54116276_13c3_4e07_b8e9_65943636d1e5.slice/crio-0e87b8d9b43f322c38bb6f66caa80558a6e140094b0cfe8e3a50a5687b150d18 WatchSource:0}: Error finding container 0e87b8d9b43f322c38bb6f66caa80558a6e140094b0cfe8e3a50a5687b150d18: Status 404 returned error can't find the container with id 0e87b8d9b43f322c38bb6f66caa80558a6e140094b0cfe8e3a50a5687b150d18
Oct 30 18:42:10.645789 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:10.644050    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lr5h6"]
Oct 30 18:42:10.649496 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:42:10.649418    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaab475f7_3a68_4c19_930b_b055767366a9.slice/crio-b441edd01d0a6af7bb9b755dc9705fc7ed8a3939dfa71a70125af525e44b34a6 WatchSource:0}: Error finding container b441edd01d0a6af7bb9b755dc9705fc7ed8a3939dfa71a70125af525e44b34a6: Status 404 returned error can't find the container with id b441edd01d0a6af7bb9b755dc9705fc7ed8a3939dfa71a70125af525e44b34a6
Oct 30 18:42:10.743481 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:10.743401    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c9w86" event={"ID":"9eb75e76-3366-457a-972f-e445bb34328e","Type":"ContainerStarted","Data":"14fb4c9862c83f4a5318029de9617565ba81c2dfa1d6bca6cfa3c51ef557a287"}
Oct 30 18:42:10.745424 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:10.745390    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lr5h6" event={"ID":"aab475f7-3a68-4c19-930b-b055767366a9","Type":"ContainerStarted","Data":"b441edd01d0a6af7bb9b755dc9705fc7ed8a3939dfa71a70125af525e44b34a6"}
Oct 30 18:42:10.747468 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:10.747412    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chnnzp" event={"ID":"54116276-13c3-4e07-b8e9-65943636d1e5","Type":"ContainerStarted","Data":"0e87b8d9b43f322c38bb6f66caa80558a6e140094b0cfe8e3a50a5687b150d18"}
Oct 30 18:42:10.751124 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:10.751043    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2gq6q" event={"ID":"5332c799-1c59-4995-8c40-146f82223d75","Type":"ContainerStarted","Data":"637157f9a8e5adbecaefef3f8676f2a7c5f7780ae0cabc55ce4b2e492a921cb5"}
Oct 30 18:42:11.357451 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:11.356263    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-sgtll"]
Oct 30 18:42:11.360867 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:11.359631    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-sgtll"
Oct 30 18:42:11.365205 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:11.364851    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-sj2rm\""
Oct 30 18:42:11.365205 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:11.364951    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 18:42:11.365205 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:11.365007    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 18:42:11.365205 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:11.365078    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 18:42:11.377661 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:11.376177    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-sgtll"]
Oct 30 18:42:11.454933 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:11.453469    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-864cbfbb6c-qhbfx"]
Oct 30 18:42:11.458720 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:11.457119    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-864cbfbb6c-qhbfx"
Oct 30 18:42:11.476739 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:11.475936    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-g7cd4\""
Oct 30 18:42:11.479462 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:11.479379    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-864cbfbb6c-qhbfx"]
Oct 30 18:42:11.529360 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:11.529324    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85ce0a7f-b7fb-4909-bfd6-affca6da6d49-workload-certs\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-sgtll\" (UID: \"85ce0a7f-b7fb-4909-bfd6-affca6da6d49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-sgtll"
Oct 30 18:42:11.529531 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:11.529372    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85ce0a7f-b7fb-4909-bfd6-affca6da6d49-workload-socket\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-sgtll\" (UID: \"85ce0a7f-b7fb-4909-bfd6-affca6da6d49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-sgtll"
Oct 30 18:42:11.529531 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:11.529409    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85ce0a7f-b7fb-4909-bfd6-affca6da6d49-istio-data\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-sgtll\" (UID: \"85ce0a7f-b7fb-4909-bfd6-affca6da6d49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-sgtll"
Oct 30 18:42:11.529531 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:11.529450    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9dbq\" (UniqueName: \"kubernetes.io/projected/85ce0a7f-b7fb-4909-bfd6-affca6da6d49-kube-api-access-q9dbq\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-sgtll\" (UID: \"85ce0a7f-b7fb-4909-bfd6-affca6da6d49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-sgtll"
Oct 30 18:42:11.529531 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:11.529516    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85ce0a7f-b7fb-4909-bfd6-affca6da6d49-credential-socket\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-sgtll\" (UID: \"85ce0a7f-b7fb-4909-bfd6-affca6da6d49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-sgtll"
Oct 30 18:42:11.529792 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:11.529551    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85ce0a7f-b7fb-4909-bfd6-affca6da6d49-istio-podinfo\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-sgtll\" (UID: \"85ce0a7f-b7fb-4909-bfd6-affca6da6d49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-sgtll"
Oct 30 18:42:11.529792 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:11.529593    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85ce0a7f-b7fb-4909-bfd6-affca6da6d49-istio-envoy\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-sgtll\" (UID: \"85ce0a7f-b7fb-4909-bfd6-affca6da6d49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-sgtll"
Oct 30 18:42:11.529792 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:11.529618    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85ce0a7f-b7fb-4909-bfd6-affca6da6d49-istio-token\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-sgtll\" (UID: \"85ce0a7f-b7fb-4909-bfd6-affca6da6d49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-sgtll"
Oct 30 18:42:11.529792 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:11.529644    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85ce0a7f-b7fb-4909-bfd6-affca6da6d49-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-sgtll\" (UID: \"85ce0a7f-b7fb-4909-bfd6-affca6da6d49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-sgtll"
Oct 30 18:42:11.631111 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:11.631024    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6jdx\" (UniqueName: \"kubernetes.io/projected/d9a5292b-dacb-4a0f-af6d-f3da1b7843c2-kube-api-access-g6jdx\") pod \"openshift-ai-inference-openshift-ai-inference-864cbfbb6c-qhbfx\" (UID: \"d9a5292b-dacb-4a0f-af6d-f3da1b7843c2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-864cbfbb6c-qhbfx"
Oct 30 18:42:11.631111 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:11.631082    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9a5292b-dacb-4a0f-af6d-f3da1b7843c2-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-864cbfbb6c-qhbfx\" (UID: \"d9a5292b-dacb-4a0f-af6d-f3da1b7843c2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-864cbfbb6c-qhbfx"
Oct 30 18:42:11.631276 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:11.631116    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85ce0a7f-b7fb-4909-bfd6-affca6da6d49-credential-socket\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-sgtll\" (UID: \"85ce0a7f-b7fb-4909-bfd6-affca6da6d49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-sgtll"
Oct 30 18:42:11.631276 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:11.631148    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85ce0a7f-b7fb-4909-bfd6-affca6da6d49-istio-podinfo\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-sgtll\" (UID: \"85ce0a7f-b7fb-4909-bfd6-affca6da6d49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-sgtll"
Oct 30 18:42:11.631276 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:11.631175    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9a5292b-dacb-4a0f-af6d-f3da1b7843c2-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-864cbfbb6c-qhbfx\" (UID: \"d9a5292b-dacb-4a0f-af6d-f3da1b7843c2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-864cbfbb6c-qhbfx"
Oct 30 18:42:11.631276 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:11.631199    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9a5292b-dacb-4a0f-af6d-f3da1b7843c2-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-864cbfbb6c-qhbfx\" (UID: \"d9a5292b-dacb-4a0f-af6d-f3da1b7843c2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-864cbfbb6c-qhbfx"
Oct 30 18:42:11.631276 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:11.631240    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85ce0a7f-b7fb-4909-bfd6-affca6da6d49-istio-envoy\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-sgtll\" (UID: \"85ce0a7f-b7fb-4909-bfd6-affca6da6d49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-sgtll"
Oct 30 18:42:11.631417 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:11.631277    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85ce0a7f-b7fb-4909-bfd6-affca6da6d49-istio-token\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-sgtll\" (UID: \"85ce0a7f-b7fb-4909-bfd6-affca6da6d49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-sgtll"
Oct 30 18:42:11.631417 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:11.631301    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85ce0a7f-b7fb-4909-bfd6-affca6da6d49-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-sgtll\" (UID: \"85ce0a7f-b7fb-4909-bfd6-affca6da6d49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-sgtll"
Oct 30 18:42:11.631417 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:11.631332    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9a5292b-dacb-4a0f-af6d-f3da1b7843c2-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-864cbfbb6c-qhbfx\" (UID: \"d9a5292b-dacb-4a0f-af6d-f3da1b7843c2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-864cbfbb6c-qhbfx"
Oct 30 18:42:11.631417 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:11.631355    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9a5292b-dacb-4a0f-af6d-f3da1b7843c2-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-864cbfbb6c-qhbfx\" (UID: \"d9a5292b-dacb-4a0f-af6d-f3da1b7843c2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-864cbfbb6c-qhbfx"
Oct 30 18:42:11.631417 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:11.631384    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85ce0a7f-b7fb-4909-bfd6-affca6da6d49-workload-certs\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-sgtll\" (UID: \"85ce0a7f-b7fb-4909-bfd6-affca6da6d49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-sgtll"
Oct 30 18:42:11.631550 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:11.631420    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85ce0a7f-b7fb-4909-bfd6-affca6da6d49-workload-socket\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-sgtll\" (UID: \"85ce0a7f-b7fb-4909-bfd6-affca6da6d49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-sgtll"
Oct 30 18:42:11.631550 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:11.631444    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9a5292b-dacb-4a0f-af6d-f3da1b7843c2-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-864cbfbb6c-qhbfx\" (UID: \"d9a5292b-dacb-4a0f-af6d-f3da1b7843c2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-864cbfbb6c-qhbfx"
Oct 30 18:42:11.631550 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:11.631499    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85ce0a7f-b7fb-4909-bfd6-affca6da6d49-istio-data\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-sgtll\" (UID: \"85ce0a7f-b7fb-4909-bfd6-affca6da6d49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-sgtll"
Oct 30 18:42:11.631550 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:11.631532    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9dbq\" (UniqueName: \"kubernetes.io/projected/85ce0a7f-b7fb-4909-bfd6-affca6da6d49-kube-api-access-q9dbq\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-sgtll\" (UID: \"85ce0a7f-b7fb-4909-bfd6-affca6da6d49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-sgtll"
Oct 30 18:42:11.631661 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:11.631565    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9a5292b-dacb-4a0f-af6d-f3da1b7843c2-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-864cbfbb6c-qhbfx\" (UID: \"d9a5292b-dacb-4a0f-af6d-f3da1b7843c2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-864cbfbb6c-qhbfx"
Oct 30 18:42:11.631661 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:11.631589    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9a5292b-dacb-4a0f-af6d-f3da1b7843c2-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-864cbfbb6c-qhbfx\" (UID: \"d9a5292b-dacb-4a0f-af6d-f3da1b7843c2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-864cbfbb6c-qhbfx"
Oct 30 18:42:11.632126 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:11.632094    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85ce0a7f-b7fb-4909-bfd6-affca6da6d49-credential-socket\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-sgtll\" (UID: \"85ce0a7f-b7fb-4909-bfd6-affca6da6d49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-sgtll"
Oct 30 18:42:11.633103 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:11.633080    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85ce0a7f-b7fb-4909-bfd6-affca6da6d49-workload-certs\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-sgtll\" (UID: \"85ce0a7f-b7fb-4909-bfd6-affca6da6d49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-sgtll"
Oct 30 18:42:11.633325 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:11.633307    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85ce0a7f-b7fb-4909-bfd6-affca6da6d49-workload-socket\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-sgtll\" (UID: \"85ce0a7f-b7fb-4909-bfd6-affca6da6d49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-sgtll"
Oct 30 18:42:11.633934 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:11.633910    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85ce0a7f-b7fb-4909-bfd6-affca6da6d49-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-sgtll\" (UID: \"85ce0a7f-b7fb-4909-bfd6-affca6da6d49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-sgtll"
Oct 30 18:42:11.634660 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:11.634638    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85ce0a7f-b7fb-4909-bfd6-affca6da6d49-istio-data\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-sgtll\" (UID: \"85ce0a7f-b7fb-4909-bfd6-affca6da6d49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-sgtll"
Oct 30 18:42:11.644712 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:11.644449    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9dbq\" (UniqueName: \"kubernetes.io/projected/85ce0a7f-b7fb-4909-bfd6-affca6da6d49-kube-api-access-q9dbq\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-sgtll\" (UID: \"85ce0a7f-b7fb-4909-bfd6-affca6da6d49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-sgtll"
Oct 30 18:42:11.649762 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:11.648007    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85ce0a7f-b7fb-4909-bfd6-affca6da6d49-istio-token\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-sgtll\" (UID: \"85ce0a7f-b7fb-4909-bfd6-affca6da6d49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-sgtll"
Oct 30 18:42:11.652011 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:11.651987    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85ce0a7f-b7fb-4909-bfd6-affca6da6d49-istio-envoy\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-sgtll\" (UID: \"85ce0a7f-b7fb-4909-bfd6-affca6da6d49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-sgtll"
Oct 30 18:42:11.661074 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:11.661014    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85ce0a7f-b7fb-4909-bfd6-affca6da6d49-istio-podinfo\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-sgtll\" (UID: \"85ce0a7f-b7fb-4909-bfd6-affca6da6d49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-sgtll"
Oct 30 18:42:11.685997 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:11.685747    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-sgtll"
Oct 30 18:42:11.735525 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:11.732585    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9a5292b-dacb-4a0f-af6d-f3da1b7843c2-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-864cbfbb6c-qhbfx\" (UID: \"d9a5292b-dacb-4a0f-af6d-f3da1b7843c2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-864cbfbb6c-qhbfx"
Oct 30 18:42:11.735525 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:11.732620    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9a5292b-dacb-4a0f-af6d-f3da1b7843c2-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-864cbfbb6c-qhbfx\" (UID: \"d9a5292b-dacb-4a0f-af6d-f3da1b7843c2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-864cbfbb6c-qhbfx"
Oct 30 18:42:11.735525 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:11.732663    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6jdx\" (UniqueName: \"kubernetes.io/projected/d9a5292b-dacb-4a0f-af6d-f3da1b7843c2-kube-api-access-g6jdx\") pod \"openshift-ai-inference-openshift-ai-inference-864cbfbb6c-qhbfx\" (UID: \"d9a5292b-dacb-4a0f-af6d-f3da1b7843c2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-864cbfbb6c-qhbfx"
Oct 30 18:42:11.735525 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:11.732709    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9a5292b-dacb-4a0f-af6d-f3da1b7843c2-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-864cbfbb6c-qhbfx\" (UID: \"d9a5292b-dacb-4a0f-af6d-f3da1b7843c2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-864cbfbb6c-qhbfx"
Oct 30 18:42:11.735525 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:11.732734    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9a5292b-dacb-4a0f-af6d-f3da1b7843c2-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-864cbfbb6c-qhbfx\" (UID: \"d9a5292b-dacb-4a0f-af6d-f3da1b7843c2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-864cbfbb6c-qhbfx"
Oct 30 18:42:11.735525 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:11.732753    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9a5292b-dacb-4a0f-af6d-f3da1b7843c2-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-864cbfbb6c-qhbfx\" (UID: \"d9a5292b-dacb-4a0f-af6d-f3da1b7843c2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-864cbfbb6c-qhbfx"
Oct 30 18:42:11.735525 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:11.732787    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9a5292b-dacb-4a0f-af6d-f3da1b7843c2-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-864cbfbb6c-qhbfx\" (UID: \"d9a5292b-dacb-4a0f-af6d-f3da1b7843c2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-864cbfbb6c-qhbfx"
Oct 30 18:42:11.735525 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:11.732805    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9a5292b-dacb-4a0f-af6d-f3da1b7843c2-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-864cbfbb6c-qhbfx\" (UID: \"d9a5292b-dacb-4a0f-af6d-f3da1b7843c2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-864cbfbb6c-qhbfx"
Oct 30 18:42:11.735525 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:11.732836    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9a5292b-dacb-4a0f-af6d-f3da1b7843c2-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-864cbfbb6c-qhbfx\" (UID: \"d9a5292b-dacb-4a0f-af6d-f3da1b7843c2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-864cbfbb6c-qhbfx"
Oct 30 18:42:11.735525 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:11.733659    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9a5292b-dacb-4a0f-af6d-f3da1b7843c2-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-864cbfbb6c-qhbfx\" (UID: \"d9a5292b-dacb-4a0f-af6d-f3da1b7843c2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-864cbfbb6c-qhbfx"
Oct 30 18:42:11.735525 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:11.734333    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9a5292b-dacb-4a0f-af6d-f3da1b7843c2-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-864cbfbb6c-qhbfx\" (UID: \"d9a5292b-dacb-4a0f-af6d-f3da1b7843c2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-864cbfbb6c-qhbfx"
Oct 30 18:42:11.735525 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:11.734572    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9a5292b-dacb-4a0f-af6d-f3da1b7843c2-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-864cbfbb6c-qhbfx\" (UID: \"d9a5292b-dacb-4a0f-af6d-f3da1b7843c2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-864cbfbb6c-qhbfx"
Oct 30 18:42:11.735525 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:11.735306    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9a5292b-dacb-4a0f-af6d-f3da1b7843c2-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-864cbfbb6c-qhbfx\" (UID: \"d9a5292b-dacb-4a0f-af6d-f3da1b7843c2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-864cbfbb6c-qhbfx"
Oct 30 18:42:11.735525 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:11.735497    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9a5292b-dacb-4a0f-af6d-f3da1b7843c2-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-864cbfbb6c-qhbfx\" (UID: \"d9a5292b-dacb-4a0f-af6d-f3da1b7843c2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-864cbfbb6c-qhbfx"
Oct 30 18:42:11.742916 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:11.742871    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9a5292b-dacb-4a0f-af6d-f3da1b7843c2-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-864cbfbb6c-qhbfx\" (UID: \"d9a5292b-dacb-4a0f-af6d-f3da1b7843c2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-864cbfbb6c-qhbfx"
Oct 30 18:42:11.743604 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:11.743561    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9a5292b-dacb-4a0f-af6d-f3da1b7843c2-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-864cbfbb6c-qhbfx\" (UID: \"d9a5292b-dacb-4a0f-af6d-f3da1b7843c2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-864cbfbb6c-qhbfx"
Oct 30 18:42:11.744639 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:11.744457    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6jdx\" (UniqueName: \"kubernetes.io/projected/d9a5292b-dacb-4a0f-af6d-f3da1b7843c2-kube-api-access-g6jdx\") pod \"openshift-ai-inference-openshift-ai-inference-864cbfbb6c-qhbfx\" (UID: \"d9a5292b-dacb-4a0f-af6d-f3da1b7843c2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-864cbfbb6c-qhbfx"
Oct 30 18:42:11.744639 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:11.744609    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9a5292b-dacb-4a0f-af6d-f3da1b7843c2-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-864cbfbb6c-qhbfx\" (UID: \"d9a5292b-dacb-4a0f-af6d-f3da1b7843c2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-864cbfbb6c-qhbfx"
Oct 30 18:42:11.787069 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:11.786750    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-864cbfbb6c-qhbfx"
Oct 30 18:42:12.025533 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:12.025498    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-sgtll"]
Oct 30 18:42:12.045702 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:42:12.045636    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85ce0a7f_b7fb_4909_bfd6_affca6da6d49.slice/crio-a282fd47dfc693839da0d65efbdb8bb11c055bab1c7322abe3e66a0136064304 WatchSource:0}: Error finding container a282fd47dfc693839da0d65efbdb8bb11c055bab1c7322abe3e66a0136064304: Status 404 returned error can't find the container with id a282fd47dfc693839da0d65efbdb8bb11c055bab1c7322abe3e66a0136064304
Oct 30 18:42:12.131728 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:12.131067    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-864cbfbb6c-qhbfx"]
Oct 30 18:42:12.137466 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:42:12.137433    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9a5292b_dacb_4a0f_af6d_f3da1b7843c2.slice/crio-11e3e82da01391decea38d94973a7cfd44165a65d58366b70f00b7cce0ffeae4 WatchSource:0}: Error finding container 11e3e82da01391decea38d94973a7cfd44165a65d58366b70f00b7cce0ffeae4: Status 404 returned error can't find the container with id 11e3e82da01391decea38d94973a7cfd44165a65d58366b70f00b7cce0ffeae4
Oct 30 18:42:12.173607 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:12.171777    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8sn8c"]
Oct 30 18:42:12.177857 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:12.177301    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8sn8c"
Oct 30 18:42:12.184712 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:12.184054    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8sn8c"]
Oct 30 18:42:12.340491 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:12.340248    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qm8p8\" (UniqueName: \"kubernetes.io/projected/555342d6-04c7-43e7-ba14-32e2c3b8fae0-kube-api-access-qm8p8\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8sn8c\" (UID: \"555342d6-04c7-43e7-ba14-32e2c3b8fae0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8sn8c"
Oct 30 18:42:12.340491 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:12.340305    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/555342d6-04c7-43e7-ba14-32e2c3b8fae0-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8sn8c\" (UID: \"555342d6-04c7-43e7-ba14-32e2c3b8fae0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8sn8c"
Oct 30 18:42:12.340491 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:12.340378    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/555342d6-04c7-43e7-ba14-32e2c3b8fae0-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8sn8c\" (UID: \"555342d6-04c7-43e7-ba14-32e2c3b8fae0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8sn8c"
Oct 30 18:42:12.441244 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:12.441152    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/555342d6-04c7-43e7-ba14-32e2c3b8fae0-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8sn8c\" (UID: \"555342d6-04c7-43e7-ba14-32e2c3b8fae0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8sn8c"
Oct 30 18:42:12.441673 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:12.441240    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qm8p8\" (UniqueName: \"kubernetes.io/projected/555342d6-04c7-43e7-ba14-32e2c3b8fae0-kube-api-access-qm8p8\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8sn8c\" (UID: \"555342d6-04c7-43e7-ba14-32e2c3b8fae0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8sn8c"
Oct 30 18:42:12.441673 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:12.441272    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/555342d6-04c7-43e7-ba14-32e2c3b8fae0-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8sn8c\" (UID: \"555342d6-04c7-43e7-ba14-32e2c3b8fae0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8sn8c"
Oct 30 18:42:12.441925 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:12.441844    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/555342d6-04c7-43e7-ba14-32e2c3b8fae0-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8sn8c\" (UID: \"555342d6-04c7-43e7-ba14-32e2c3b8fae0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8sn8c"
Oct 30 18:42:12.442140 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:12.442117    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/555342d6-04c7-43e7-ba14-32e2c3b8fae0-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8sn8c\" (UID: \"555342d6-04c7-43e7-ba14-32e2c3b8fae0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8sn8c"
Oct 30 18:42:12.464581 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:12.464533    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qm8p8\" (UniqueName: \"kubernetes.io/projected/555342d6-04c7-43e7-ba14-32e2c3b8fae0-kube-api-access-qm8p8\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8sn8c\" (UID: \"555342d6-04c7-43e7-ba14-32e2c3b8fae0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8sn8c"
Oct 30 18:42:12.472361 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:12.469367    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71crvs5"]
Oct 30 18:42:12.476100 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:12.476044    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71crvs5"
Oct 30 18:42:12.480779 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:12.479729    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71crvs5"]
Oct 30 18:42:12.506971 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:12.506946    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8sn8c"
Oct 30 18:42:12.649651 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:12.643084    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d6f9c6ac-4dbd-4080-9b06-4955c441311c-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71crvs5\" (UID: \"d6f9c6ac-4dbd-4080-9b06-4955c441311c\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71crvs5"
Oct 30 18:42:12.649651 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:12.643189    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d6f9c6ac-4dbd-4080-9b06-4955c441311c-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71crvs5\" (UID: \"d6f9c6ac-4dbd-4080-9b06-4955c441311c\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71crvs5"
Oct 30 18:42:12.649651 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:12.643223    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slrgc\" (UniqueName: \"kubernetes.io/projected/d6f9c6ac-4dbd-4080-9b06-4955c441311c-kube-api-access-slrgc\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71crvs5\" (UID: \"d6f9c6ac-4dbd-4080-9b06-4955c441311c\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71crvs5"
Oct 30 18:42:12.745893 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:12.744501    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d6f9c6ac-4dbd-4080-9b06-4955c441311c-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71crvs5\" (UID: \"d6f9c6ac-4dbd-4080-9b06-4955c441311c\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71crvs5"
Oct 30 18:42:12.745893 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:12.744558    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slrgc\" (UniqueName: \"kubernetes.io/projected/d6f9c6ac-4dbd-4080-9b06-4955c441311c-kube-api-access-slrgc\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71crvs5\" (UID: \"d6f9c6ac-4dbd-4080-9b06-4955c441311c\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71crvs5"
Oct 30 18:42:12.745893 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:12.744628    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d6f9c6ac-4dbd-4080-9b06-4955c441311c-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71crvs5\" (UID: \"d6f9c6ac-4dbd-4080-9b06-4955c441311c\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71crvs5"
Oct 30 18:42:12.745893 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:12.745221    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d6f9c6ac-4dbd-4080-9b06-4955c441311c-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71crvs5\" (UID: \"d6f9c6ac-4dbd-4080-9b06-4955c441311c\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71crvs5"
Oct 30 18:42:12.745893 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:12.745483    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d6f9c6ac-4dbd-4080-9b06-4955c441311c-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71crvs5\" (UID: \"d6f9c6ac-4dbd-4080-9b06-4955c441311c\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71crvs5"
Oct 30 18:42:12.762027 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:12.761943    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slrgc\" (UniqueName: \"kubernetes.io/projected/d6f9c6ac-4dbd-4080-9b06-4955c441311c-kube-api-access-slrgc\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71crvs5\" (UID: \"d6f9c6ac-4dbd-4080-9b06-4955c441311c\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71crvs5"
Oct 30 18:42:12.783090 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:12.783049    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-864cbfbb6c-qhbfx" event={"ID":"d9a5292b-dacb-4a0f-af6d-f3da1b7843c2","Type":"ContainerStarted","Data":"11e3e82da01391decea38d94973a7cfd44165a65d58366b70f00b7cce0ffeae4"}
Oct 30 18:42:12.785127 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:12.785103    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-sgtll" event={"ID":"85ce0a7f-b7fb-4909-bfd6-affca6da6d49","Type":"ContainerStarted","Data":"a282fd47dfc693839da0d65efbdb8bb11c055bab1c7322abe3e66a0136064304"}
Oct 30 18:42:12.805698 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:12.805655    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71crvs5"
Oct 30 18:42:12.861534 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:42:12.861492    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod555342d6_04c7_43e7_ba14_32e2c3b8fae0.slice/crio-78ee8f316b48b0e4dd13516cf4f268aa3ddd49f4d08cdc1c47bf863e0762e2ae WatchSource:0}: Error finding container 78ee8f316b48b0e4dd13516cf4f268aa3ddd49f4d08cdc1c47bf863e0762e2ae: Status 404 returned error can't find the container with id 78ee8f316b48b0e4dd13516cf4f268aa3ddd49f4d08cdc1c47bf863e0762e2ae
Oct 30 18:42:12.862843 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:12.862815    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8sn8c"]
Oct 30 18:42:13.115506 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:13.115110    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71crvs5"]
Oct 30 18:42:13.117735 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:42:13.117706    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6f9c6ac_4dbd_4080_9b06_4955c441311c.slice/crio-0b33254e978ab0bf233b9121593a11bd9f01d2c2612416c7d9f48316c769c6ab WatchSource:0}: Error finding container 0b33254e978ab0bf233b9121593a11bd9f01d2c2612416c7d9f48316c769c6ab: Status 404 returned error can't find the container with id 0b33254e978ab0bf233b9121593a11bd9f01d2c2612416c7d9f48316c769c6ab
Oct 30 18:42:13.805715 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:13.804707    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71crvs5" event={"ID":"d6f9c6ac-4dbd-4080-9b06-4955c441311c","Type":"ContainerStarted","Data":"0b33254e978ab0bf233b9121593a11bd9f01d2c2612416c7d9f48316c769c6ab"}
Oct 30 18:42:13.819069 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:13.818993    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8sn8c" event={"ID":"555342d6-04c7-43e7-ba14-32e2c3b8fae0","Type":"ContainerStarted","Data":"78ee8f316b48b0e4dd13516cf4f268aa3ddd49f4d08cdc1c47bf863e0762e2ae"}
Oct 30 18:42:17.423127 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:42:17.423083    4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d49l" containerName="extractor"
Oct 30 18:42:22.950958 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:22.950920    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5h8fw" event={"ID":"5e2853c1-7992-446c-a335-2d2e63565ff3","Type":"ContainerStarted","Data":"e2d1a590e942d2eb0f2bc34bee1fa31a2d416ecc690c90247348b6368bc140ff"}
Oct 30 18:42:23.191962 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:23.191238    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:42:23.191962 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:23.191323    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:42:23.191962 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:23.191365    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:42:23.306767 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:23.306648    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:42:23.306767 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:23.306732    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:42:23.306767 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:23.306763    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:42:23.958940 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:23.958904    4525 generic.go:356] "Generic (PLEG): container finished" podID="aab475f7-3a68-4c19-930b-b055767366a9" containerID="c2897126b4423fd36d23b52eaf03fe7f77a26a33d3bb353faebda3b65e4c2517" exitCode=0
Oct 30 18:42:23.959348 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:23.958991    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lr5h6" event={"ID":"aab475f7-3a68-4c19-930b-b055767366a9","Type":"ContainerDied","Data":"c2897126b4423fd36d23b52eaf03fe7f77a26a33d3bb353faebda3b65e4c2517"}
Oct 30 18:42:23.961762 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:23.961741    4525 generic.go:356] "Generic (PLEG): container finished" podID="54116276-13c3-4e07-b8e9-65943636d1e5" containerID="111d5dfd783cd901ede06219262f77d4e74107055e48ddc4865831c00aa10d42" exitCode=0
Oct 30 18:42:23.961841 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:23.961825    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chnnzp" event={"ID":"54116276-13c3-4e07-b8e9-65943636d1e5","Type":"ContainerDied","Data":"111d5dfd783cd901ede06219262f77d4e74107055e48ddc4865831c00aa10d42"}
Oct 30 18:42:23.964194 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:23.964171    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-864cbfbb6c-qhbfx" event={"ID":"d9a5292b-dacb-4a0f-af6d-f3da1b7843c2","Type":"ContainerStarted","Data":"3a8bda3f02e787c9d1764ac4bbe1e461decb006c764c535d55f8139664a551bf"}
Oct 30 18:42:23.966548 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:23.966006    4525 generic.go:356] "Generic (PLEG): container finished" podID="5332c799-1c59-4995-8c40-146f82223d75" containerID="42ceabd4e07d963f29938b2e4137c5d5e7ef07239db884d11198a1dd7ad7e540" exitCode=0
Oct 30 18:42:23.966548 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:23.966354    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2gq6q" event={"ID":"5332c799-1c59-4995-8c40-146f82223d75","Type":"ContainerDied","Data":"42ceabd4e07d963f29938b2e4137c5d5e7ef07239db884d11198a1dd7ad7e540"}
Oct 30 18:42:23.968777 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:23.968753    4525 generic.go:356] "Generic (PLEG): container finished" podID="f78d8915-08e0-4559-8cfe-a63505f35f46" containerID="6c69b8a3ddd8bdf7c96725a96cd0eed914c921654f2afaa71937dd1e3d38748f" exitCode=0
Oct 30 18:42:23.968874 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:23.968798    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7125bzv" event={"ID":"f78d8915-08e0-4559-8cfe-a63505f35f46","Type":"ContainerDied","Data":"6c69b8a3ddd8bdf7c96725a96cd0eed914c921654f2afaa71937dd1e3d38748f"}
Oct 30 18:42:23.971433 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:23.971409    4525 generic.go:356] "Generic (PLEG): container finished" podID="5e2853c1-7992-446c-a335-2d2e63565ff3" containerID="e2d1a590e942d2eb0f2bc34bee1fa31a2d416ecc690c90247348b6368bc140ff" exitCode=0
Oct 30 18:42:23.971522 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:23.971442    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5h8fw" event={"ID":"5e2853c1-7992-446c-a335-2d2e63565ff3","Type":"ContainerDied","Data":"e2d1a590e942d2eb0f2bc34bee1fa31a2d416ecc690c90247348b6368bc140ff"}
Oct 30 18:42:23.973748 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:23.973608    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-sgtll" event={"ID":"85ce0a7f-b7fb-4909-bfd6-affca6da6d49","Type":"ContainerStarted","Data":"6cf9b5c37b7c71638afc520527f10f9d8a5b3bc1d32b025d3f3507f70f0c8b44"}
Oct 30 18:42:23.975325 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:23.975266    4525 generic.go:356] "Generic (PLEG): container finished" podID="555342d6-04c7-43e7-ba14-32e2c3b8fae0" containerID="c9a57cd9cb6539427729bcdd024778237101081cc417f663e4b7a1e59777015c" exitCode=0
Oct 30 18:42:23.975421 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:23.975323    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8sn8c" event={"ID":"555342d6-04c7-43e7-ba14-32e2c3b8fae0","Type":"ContainerDied","Data":"c9a57cd9cb6539427729bcdd024778237101081cc417f663e4b7a1e59777015c"}
Oct 30 18:42:24.055970 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:24.055915    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-sgtll" podStartSLOduration=1.7984707100000001 podStartE2EDuration="13.05589442s" totalImagesPullingTime="11.25742371s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:42:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:42:24.05350638 +0000 UTC m=+366.679176345" watchObservedRunningTime="2026-10-30 18:42:24.05589442 +0000 UTC m=+366.681564386"
Oct 30 18:42:24.687061 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:24.687026    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-sgtll"
Oct 30 18:42:24.692710 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:24.692651    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-sgtll"
Oct 30 18:42:24.717648 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:24.717580    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-864cbfbb6c-qhbfx" podStartSLOduration=2.667581088 podStartE2EDuration="13.717559233s" totalImagesPullingTime="11.049978145s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:42:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:42:24.104770388 +0000 UTC m=+366.730440353" watchObservedRunningTime="2026-10-30 18:42:24.717559233 +0000 UTC m=+367.343229198"
Oct 30 18:42:24.788000 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:24.787966    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-864cbfbb6c-qhbfx"
Oct 30 18:42:24.793974 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:24.793709    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-864cbfbb6c-qhbfx"
Oct 30 18:42:24.983805 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:24.983673    4525 generic.go:356] "Generic (PLEG): container finished" podID="d6f9c6ac-4dbd-4080-9b06-4955c441311c" containerID="aaf7227460bc8d1fbe4a52dbb10d8e6d32f2bf0ca5ee393df9a42e8000e0de57" exitCode=0
Oct 30 18:42:24.984288 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:24.984167    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71crvs5" event={"ID":"d6f9c6ac-4dbd-4080-9b06-4955c441311c","Type":"ContainerDied","Data":"aaf7227460bc8d1fbe4a52dbb10d8e6d32f2bf0ca5ee393df9a42e8000e0de57"}
Oct 30 18:42:24.996542 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:24.996506    4525 generic.go:356] "Generic (PLEG): container finished" podID="f78d8915-08e0-4559-8cfe-a63505f35f46" containerID="99ac9c2d71961b3ab89a15e2727ff2c163314d74663a52a91d9f9e40439a9b5c" exitCode=0
Oct 30 18:42:24.996776 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:24.996752    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7125bzv" event={"ID":"f78d8915-08e0-4559-8cfe-a63505f35f46","Type":"ContainerDied","Data":"99ac9c2d71961b3ab89a15e2727ff2c163314d74663a52a91d9f9e40439a9b5c"}
Oct 30 18:42:24.996875 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:24.996807    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-864cbfbb6c-qhbfx"
Oct 30 18:42:24.997709 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:24.997151    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-sgtll"
Oct 30 18:42:24.997988 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:24.997967    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-864cbfbb6c-qhbfx"
Oct 30 18:42:24.998456 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:24.998413    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-sgtll"
Oct 30 18:42:26.004574 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:26.004538    4525 generic.go:356] "Generic (PLEG): container finished" podID="dba7dabb-2faf-4472-9b77-786f5c61decb" containerID="a6d661dab06e4df6109fbd5ddd5fb4c580ad09fdb9a1fa8f94859340aff2b7f7" exitCode=0
Oct 30 18:42:26.005057 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:26.004636    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644sn5hk" event={"ID":"dba7dabb-2faf-4472-9b77-786f5c61decb","Type":"ContainerDied","Data":"a6d661dab06e4df6109fbd5ddd5fb4c580ad09fdb9a1fa8f94859340aff2b7f7"}
Oct 30 18:42:26.011131 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:26.010992    4525 generic.go:356] "Generic (PLEG): container finished" podID="aab475f7-3a68-4c19-930b-b055767366a9" containerID="425eaa0634ce1a883c7149c4b694bdcefcdad766befba5d44a8d771f36ebc527" exitCode=0
Oct 30 18:42:26.011131 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:26.011072    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lr5h6" event={"ID":"aab475f7-3a68-4c19-930b-b055767366a9","Type":"ContainerDied","Data":"425eaa0634ce1a883c7149c4b694bdcefcdad766befba5d44a8d771f36ebc527"}
Oct 30 18:42:26.014286 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:26.014262    4525 generic.go:356] "Generic (PLEG): container finished" podID="54116276-13c3-4e07-b8e9-65943636d1e5" containerID="ff2e4f85afde59926d23df44a2dcafd03078a60a6306c9f3e4955650fe0858a5" exitCode=0
Oct 30 18:42:26.014378 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:26.014296    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chnnzp" event={"ID":"54116276-13c3-4e07-b8e9-65943636d1e5","Type":"ContainerDied","Data":"ff2e4f85afde59926d23df44a2dcafd03078a60a6306c9f3e4955650fe0858a5"}
Oct 30 18:42:26.018450 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:26.017822    4525 generic.go:356] "Generic (PLEG): container finished" podID="5332c799-1c59-4995-8c40-146f82223d75" containerID="97d5c54dc7fc7158bc194b17736917952fb037d1bd52a94a6d50ee6c805d8d8a" exitCode=0
Oct 30 18:42:26.018450 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:26.017902    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2gq6q" event={"ID":"5332c799-1c59-4995-8c40-146f82223d75","Type":"ContainerDied","Data":"97d5c54dc7fc7158bc194b17736917952fb037d1bd52a94a6d50ee6c805d8d8a"}
Oct 30 18:42:26.023042 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:26.023005    4525 generic.go:356] "Generic (PLEG): container finished" podID="d6f9c6ac-4dbd-4080-9b06-4955c441311c" containerID="0fd7574c122aa366d08263373079329dfcf6d36b5e2c80028acff7f15a0ec19f" exitCode=0
Oct 30 18:42:26.023176 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:26.023159    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71crvs5" event={"ID":"d6f9c6ac-4dbd-4080-9b06-4955c441311c","Type":"ContainerDied","Data":"0fd7574c122aa366d08263373079329dfcf6d36b5e2c80028acff7f15a0ec19f"}
Oct 30 18:42:27.034078 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:27.034033    4525 generic.go:356] "Generic (PLEG): container finished" podID="9eb75e76-3366-457a-972f-e445bb34328e" containerID="599fc8dd674b900bef4db90a0225f6d0d1fe61f3e52ad17a5a2faa6ae8fcd5ee" exitCode=0
Oct 30 18:42:27.035341 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:27.035311    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c9w86" event={"ID":"9eb75e76-3366-457a-972f-e445bb34328e","Type":"ContainerDied","Data":"599fc8dd674b900bef4db90a0225f6d0d1fe61f3e52ad17a5a2faa6ae8fcd5ee"}
Oct 30 18:42:28.051447 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:28.051410    4525 generic.go:356] "Generic (PLEG): container finished" podID="5e2853c1-7992-446c-a335-2d2e63565ff3" containerID="20e5920d578318aa71eb2a24b693514dcbbaa2b04d786ceb64d0005880b57601" exitCode=0
Oct 30 18:42:28.051873 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:28.051501    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5h8fw" event={"ID":"5e2853c1-7992-446c-a335-2d2e63565ff3","Type":"ContainerDied","Data":"20e5920d578318aa71eb2a24b693514dcbbaa2b04d786ceb64d0005880b57601"}
Oct 30 18:42:28.059127 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:28.058292    4525 generic.go:356] "Generic (PLEG): container finished" podID="555342d6-04c7-43e7-ba14-32e2c3b8fae0" containerID="987e9e06c0631f858f00e48e8f84256d98d63d262f2caa13a4f9770de5ece5f5" exitCode=0
Oct 30 18:42:28.059127 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:28.058395    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8sn8c" event={"ID":"555342d6-04c7-43e7-ba14-32e2c3b8fae0","Type":"ContainerDied","Data":"987e9e06c0631f858f00e48e8f84256d98d63d262f2caa13a4f9770de5ece5f5"}
Oct 30 18:42:28.064832 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:28.064580    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c9w86" event={"ID":"9eb75e76-3366-457a-972f-e445bb34328e","Type":"ContainerStarted","Data":"af84c60bbc4622d5a88b7bba24efa1345e71d88016c637fe6d496a491ea433b0"}
Oct 30 18:42:29.083811 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:29.083764    4525 generic.go:356] "Generic (PLEG): container finished" podID="9eb75e76-3366-457a-972f-e445bb34328e" containerID="af84c60bbc4622d5a88b7bba24efa1345e71d88016c637fe6d496a491ea433b0" exitCode=0
Oct 30 18:42:29.084318 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:29.083856    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c9w86" event={"ID":"9eb75e76-3366-457a-972f-e445bb34328e","Type":"ContainerDied","Data":"af84c60bbc4622d5a88b7bba24efa1345e71d88016c637fe6d496a491ea433b0"}
Oct 30 18:42:33.137105 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:33.137067    4525 generic.go:356] "Generic (PLEG): container finished" podID="dba7dabb-2faf-4472-9b77-786f5c61decb" containerID="bd7d331f0b0619aedd46fc3c517663d468f5813bdb0356eb4b387fcfb899ccd8" exitCode=0
Oct 30 18:42:33.137571 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:33.137140    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644sn5hk" event={"ID":"dba7dabb-2faf-4472-9b77-786f5c61decb","Type":"ContainerDied","Data":"bd7d331f0b0619aedd46fc3c517663d468f5813bdb0356eb4b387fcfb899ccd8"}
Oct 30 18:42:43.251886 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:43.251850    4525 generic.go:356] "Generic (PLEG): container finished" podID="f78d8915-08e0-4559-8cfe-a63505f35f46" containerID="c7f23ef09468bdaf426f9d1ad81d596c936df1ebfddcef200c92d9bf492e0c05" exitCode=0
Oct 30 18:42:43.252333 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:43.251942    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7125bzv" event={"ID":"f78d8915-08e0-4559-8cfe-a63505f35f46","Type":"ContainerDied","Data":"c7f23ef09468bdaf426f9d1ad81d596c936df1ebfddcef200c92d9bf492e0c05"}
Oct 30 18:42:44.462503 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:44.462467    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7125bzv"
Oct 30 18:42:44.577653 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:44.577621    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f78d8915-08e0-4559-8cfe-a63505f35f46-bundle\" (UniqueName: \"kubernetes.io/empty-dir/f78d8915-08e0-4559-8cfe-a63505f35f46-bundle\") pod \"f78d8915-08e0-4559-8cfe-a63505f35f46\" (UID: \"f78d8915-08e0-4559-8cfe-a63505f35f46\") "
Oct 30 18:42:44.577854 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:44.577700    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f78d8915-08e0-4559-8cfe-a63505f35f46-kube-api-access-9pkls\" (UniqueName: \"kubernetes.io/projected/f78d8915-08e0-4559-8cfe-a63505f35f46-kube-api-access-9pkls\") pod \"f78d8915-08e0-4559-8cfe-a63505f35f46\" (UID: \"f78d8915-08e0-4559-8cfe-a63505f35f46\") "
Oct 30 18:42:44.577854 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:44.577786    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f78d8915-08e0-4559-8cfe-a63505f35f46-util\" (UniqueName: \"kubernetes.io/empty-dir/f78d8915-08e0-4559-8cfe-a63505f35f46-util\") pod \"f78d8915-08e0-4559-8cfe-a63505f35f46\" (UID: \"f78d8915-08e0-4559-8cfe-a63505f35f46\") "
Oct 30 18:42:44.578698 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:44.578648    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f78d8915-08e0-4559-8cfe-a63505f35f46-bundle" pod "f78d8915-08e0-4559-8cfe-a63505f35f46" (UID: "f78d8915-08e0-4559-8cfe-a63505f35f46"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:42:44.583200 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:44.583170    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f78d8915-08e0-4559-8cfe-a63505f35f46-kube-api-access-9pkls" pod "f78d8915-08e0-4559-8cfe-a63505f35f46" (UID: "f78d8915-08e0-4559-8cfe-a63505f35f46"). InnerVolumeSpecName "kube-api-access-9pkls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:42:44.584828 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:44.584799    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f78d8915-08e0-4559-8cfe-a63505f35f46-util" pod "f78d8915-08e0-4559-8cfe-a63505f35f46" (UID: "f78d8915-08e0-4559-8cfe-a63505f35f46"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:42:44.678864 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:44.678834    4525 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f78d8915-08e0-4559-8cfe-a63505f35f46-bundle\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:42:44.679000 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:44.678869    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pkls\" (UniqueName: \"kubernetes.io/projected/f78d8915-08e0-4559-8cfe-a63505f35f46-kube-api-access-9pkls\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:42:44.679000 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:44.678884    4525 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f78d8915-08e0-4559-8cfe-a63505f35f46-util\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:42:45.269515 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:45.269481    4525 generic.go:356] "Generic (PLEG): container finished" podID="dba7dabb-2faf-4472-9b77-786f5c61decb" containerID="45d0de6a804022984cedbfdb4c820045ee4687bcb9c86eb8dfcfcd86d25fe1e5" exitCode=0
Oct 30 18:42:45.269956 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:45.269927    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644sn5hk" event={"ID":"dba7dabb-2faf-4472-9b77-786f5c61decb","Type":"ContainerDied","Data":"45d0de6a804022984cedbfdb4c820045ee4687bcb9c86eb8dfcfcd86d25fe1e5"}
Oct 30 18:42:45.272671 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:45.272543    4525 generic.go:356] "Generic (PLEG): container finished" podID="aab475f7-3a68-4c19-930b-b055767366a9" containerID="79b0335992ce1e6e3853901767ad0d609fa5020808b90a98d1b9bf9f65e857aa" exitCode=0
Oct 30 18:42:45.272671 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:45.272649    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lr5h6" event={"ID":"aab475f7-3a68-4c19-930b-b055767366a9","Type":"ContainerDied","Data":"79b0335992ce1e6e3853901767ad0d609fa5020808b90a98d1b9bf9f65e857aa"}
Oct 30 18:42:45.288702 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:45.288494    4525 generic.go:356] "Generic (PLEG): container finished" podID="54116276-13c3-4e07-b8e9-65943636d1e5" containerID="c664a8ac9cd59e54f4990af6a86c5e30acfd738053d8129b79ad6a73c906d47a" exitCode=0
Oct 30 18:42:45.288702 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:45.288587    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chnnzp" event={"ID":"54116276-13c3-4e07-b8e9-65943636d1e5","Type":"ContainerDied","Data":"c664a8ac9cd59e54f4990af6a86c5e30acfd738053d8129b79ad6a73c906d47a"}
Oct 30 18:42:45.291300 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:45.291268    4525 generic.go:356] "Generic (PLEG): container finished" podID="5332c799-1c59-4995-8c40-146f82223d75" containerID="bf4c3371eb0683dc6a76dd0f8cf7c8e861b4176b58612f098aab630e02b5daa4" exitCode=0
Oct 30 18:42:45.291424 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:45.291364    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2gq6q" event={"ID":"5332c799-1c59-4995-8c40-146f82223d75","Type":"ContainerDied","Data":"bf4c3371eb0683dc6a76dd0f8cf7c8e861b4176b58612f098aab630e02b5daa4"}
Oct 30 18:42:45.294062 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:45.294039    4525 generic.go:356] "Generic (PLEG): container finished" podID="d6f9c6ac-4dbd-4080-9b06-4955c441311c" containerID="38f4bc8f4aeedb166a1297780ba84f62295eb6301505bfd3da31bc65157c0a66" exitCode=0
Oct 30 18:42:45.294292 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:45.294169    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71crvs5" event={"ID":"d6f9c6ac-4dbd-4080-9b06-4955c441311c","Type":"ContainerDied","Data":"38f4bc8f4aeedb166a1297780ba84f62295eb6301505bfd3da31bc65157c0a66"}
Oct 30 18:42:45.296605 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:45.296583    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7125bzv" event={"ID":"f78d8915-08e0-4559-8cfe-a63505f35f46","Type":"ContainerDied","Data":"a74851a69b721cf96641286e890e204656631c66d61996cb3d4cbaded9005e17"}
Oct 30 18:42:45.296778 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:45.296751    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7125bzv"
Oct 30 18:42:45.296885 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:45.296752    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a74851a69b721cf96641286e890e204656631c66d61996cb3d4cbaded9005e17"
Oct 30 18:42:46.302669 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:46.302634    4525 generic.go:356] "Generic (PLEG): container finished" podID="9eb75e76-3366-457a-972f-e445bb34328e" containerID="9e61430fd1b907742020fc0a3d1237342bc4401a716f9168c154572937b2c80f" exitCode=0
Oct 30 18:42:46.303069 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:46.302726    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c9w86" event={"ID":"9eb75e76-3366-457a-972f-e445bb34328e","Type":"ContainerDied","Data":"9e61430fd1b907742020fc0a3d1237342bc4401a716f9168c154572937b2c80f"}
Oct 30 18:42:46.304912 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:46.304889    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5h8fw" event={"ID":"5e2853c1-7992-446c-a335-2d2e63565ff3","Type":"ContainerStarted","Data":"d13c713c6b8da2b8f7c5f68a63ae5a9d3b9caf86d9e6e5b82b7cfd6f17958f1d"}
Oct 30 18:42:46.306988 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:46.306965    4525 generic.go:356] "Generic (PLEG): container finished" podID="555342d6-04c7-43e7-ba14-32e2c3b8fae0" containerID="18826343767e45ff8c0f7eb9c77729ea909f428a958e6c53164ce14e85d61717" exitCode=0
Oct 30 18:42:46.307100 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:46.306987    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8sn8c" event={"ID":"555342d6-04c7-43e7-ba14-32e2c3b8fae0","Type":"ContainerDied","Data":"18826343767e45ff8c0f7eb9c77729ea909f428a958e6c53164ce14e85d61717"}
Oct 30 18:42:46.340430 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:46.340361    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5h8fw" podStartSLOduration=3.323982536 podStartE2EDuration="40.340340882s" totalImagesPullingTime="37.016358346s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:42:06 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:42:46.339972605 +0000 UTC m=+388.965642570" watchObservedRunningTime="2026-10-30 18:42:46.340340882 +0000 UTC m=+388.966010847"
Oct 30 18:42:46.481141 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:46.480816    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2gq6q"
Oct 30 18:42:46.485581 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:46.484846    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lr5h6"
Oct 30 18:42:46.522950 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:46.522911    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chnnzp"
Oct 30 18:42:46.526060 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:46.526038    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71crvs5"
Oct 30 18:42:46.529368 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:46.529344    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644sn5hk"
Oct 30 18:42:46.596541 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:46.596515    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5332c799-1c59-4995-8c40-146f82223d75-util\" (UniqueName: \"kubernetes.io/empty-dir/5332c799-1c59-4995-8c40-146f82223d75-util\") pod \"5332c799-1c59-4995-8c40-146f82223d75\" (UID: \"5332c799-1c59-4995-8c40-146f82223d75\") "
Oct 30 18:42:46.596749 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:46.596552    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dba7dabb-2faf-4472-9b77-786f5c61decb-bundle\" (UniqueName: \"kubernetes.io/empty-dir/dba7dabb-2faf-4472-9b77-786f5c61decb-bundle\") pod \"dba7dabb-2faf-4472-9b77-786f5c61decb\" (UID: \"dba7dabb-2faf-4472-9b77-786f5c61decb\") "
Oct 30 18:42:46.596749 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:46.596580    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6f9c6ac-4dbd-4080-9b06-4955c441311c-bundle\" (UniqueName: \"kubernetes.io/empty-dir/d6f9c6ac-4dbd-4080-9b06-4955c441311c-bundle\") pod \"d6f9c6ac-4dbd-4080-9b06-4955c441311c\" (UID: \"d6f9c6ac-4dbd-4080-9b06-4955c441311c\") "
Oct 30 18:42:46.596749 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:46.596613    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5332c799-1c59-4995-8c40-146f82223d75-bundle\" (UniqueName: \"kubernetes.io/empty-dir/5332c799-1c59-4995-8c40-146f82223d75-bundle\") pod \"5332c799-1c59-4995-8c40-146f82223d75\" (UID: \"5332c799-1c59-4995-8c40-146f82223d75\") "
Oct 30 18:42:46.596749 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:46.596636    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aab475f7-3a68-4c19-930b-b055767366a9-kube-api-access-xpqg2\" (UniqueName: \"kubernetes.io/projected/aab475f7-3a68-4c19-930b-b055767366a9-kube-api-access-xpqg2\") pod \"aab475f7-3a68-4c19-930b-b055767366a9\" (UID: \"aab475f7-3a68-4c19-930b-b055767366a9\") "
Oct 30 18:42:46.596749 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:46.596661    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6f9c6ac-4dbd-4080-9b06-4955c441311c-kube-api-access-slrgc\" (UniqueName: \"kubernetes.io/projected/d6f9c6ac-4dbd-4080-9b06-4955c441311c-kube-api-access-slrgc\") pod \"d6f9c6ac-4dbd-4080-9b06-4955c441311c\" (UID: \"d6f9c6ac-4dbd-4080-9b06-4955c441311c\") "
Oct 30 18:42:46.596749 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:46.596696    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5332c799-1c59-4995-8c40-146f82223d75-kube-api-access-6rm9w\" (UniqueName: \"kubernetes.io/projected/5332c799-1c59-4995-8c40-146f82223d75-kube-api-access-6rm9w\") pod \"5332c799-1c59-4995-8c40-146f82223d75\" (UID: \"5332c799-1c59-4995-8c40-146f82223d75\") "
Oct 30 18:42:46.597048 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:46.596999    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dba7dabb-2faf-4472-9b77-786f5c61decb-kube-api-access-5tmdq\" (UniqueName: \"kubernetes.io/projected/dba7dabb-2faf-4472-9b77-786f5c61decb-kube-api-access-5tmdq\") pod \"dba7dabb-2faf-4472-9b77-786f5c61decb\" (UID: \"dba7dabb-2faf-4472-9b77-786f5c61decb\") "
Oct 30 18:42:46.597048 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:46.597042    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dba7dabb-2faf-4472-9b77-786f5c61decb-util\" (UniqueName: \"kubernetes.io/empty-dir/dba7dabb-2faf-4472-9b77-786f5c61decb-util\") pod \"dba7dabb-2faf-4472-9b77-786f5c61decb\" (UID: \"dba7dabb-2faf-4472-9b77-786f5c61decb\") "
Oct 30 18:42:46.597157 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:46.597053    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dba7dabb-2faf-4472-9b77-786f5c61decb-bundle" pod "dba7dabb-2faf-4472-9b77-786f5c61decb" (UID: "dba7dabb-2faf-4472-9b77-786f5c61decb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:42:46.597157 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:46.597070    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54116276-13c3-4e07-b8e9-65943636d1e5-bundle\" (UniqueName: \"kubernetes.io/empty-dir/54116276-13c3-4e07-b8e9-65943636d1e5-bundle\") pod \"54116276-13c3-4e07-b8e9-65943636d1e5\" (UID: \"54116276-13c3-4e07-b8e9-65943636d1e5\") "
Oct 30 18:42:46.597157 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:46.597112    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6f9c6ac-4dbd-4080-9b06-4955c441311c-util\" (UniqueName: \"kubernetes.io/empty-dir/d6f9c6ac-4dbd-4080-9b06-4955c441311c-util\") pod \"d6f9c6ac-4dbd-4080-9b06-4955c441311c\" (UID: \"d6f9c6ac-4dbd-4080-9b06-4955c441311c\") "
Oct 30 18:42:46.597157 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:46.597150    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aab475f7-3a68-4c19-930b-b055767366a9-bundle\" (UniqueName: \"kubernetes.io/empty-dir/aab475f7-3a68-4c19-930b-b055767366a9-bundle\") pod \"aab475f7-3a68-4c19-930b-b055767366a9\" (UID: \"aab475f7-3a68-4c19-930b-b055767366a9\") "
Oct 30 18:42:46.597378 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:46.597182    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aab475f7-3a68-4c19-930b-b055767366a9-util\" (UniqueName: \"kubernetes.io/empty-dir/aab475f7-3a68-4c19-930b-b055767366a9-util\") pod \"aab475f7-3a68-4c19-930b-b055767366a9\" (UID: \"aab475f7-3a68-4c19-930b-b055767366a9\") "
Oct 30 18:42:46.597378 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:46.597219    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54116276-13c3-4e07-b8e9-65943636d1e5-kube-api-access-8nw8w\" (UniqueName: \"kubernetes.io/projected/54116276-13c3-4e07-b8e9-65943636d1e5-kube-api-access-8nw8w\") pod \"54116276-13c3-4e07-b8e9-65943636d1e5\" (UID: \"54116276-13c3-4e07-b8e9-65943636d1e5\") "
Oct 30 18:42:46.597378 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:46.597246    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54116276-13c3-4e07-b8e9-65943636d1e5-util\" (UniqueName: \"kubernetes.io/empty-dir/54116276-13c3-4e07-b8e9-65943636d1e5-util\") pod \"54116276-13c3-4e07-b8e9-65943636d1e5\" (UID: \"54116276-13c3-4e07-b8e9-65943636d1e5\") "
Oct 30 18:42:46.597378 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:46.597252    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6f9c6ac-4dbd-4080-9b06-4955c441311c-bundle" pod "d6f9c6ac-4dbd-4080-9b06-4955c441311c" (UID: "d6f9c6ac-4dbd-4080-9b06-4955c441311c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:42:46.597582 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:46.597565    4525 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dba7dabb-2faf-4472-9b77-786f5c61decb-bundle\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:42:46.597651 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:46.597587    4525 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d6f9c6ac-4dbd-4080-9b06-4955c441311c-bundle\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:42:46.597727 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:46.597696    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5332c799-1c59-4995-8c40-146f82223d75-bundle" pod "5332c799-1c59-4995-8c40-146f82223d75" (UID: "5332c799-1c59-4995-8c40-146f82223d75"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:42:46.598296 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:46.598267    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aab475f7-3a68-4c19-930b-b055767366a9-bundle" pod "aab475f7-3a68-4c19-930b-b055767366a9" (UID: "aab475f7-3a68-4c19-930b-b055767366a9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:42:46.599615 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:46.599586    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dba7dabb-2faf-4472-9b77-786f5c61decb-kube-api-access-5tmdq" pod "dba7dabb-2faf-4472-9b77-786f5c61decb" (UID: "dba7dabb-2faf-4472-9b77-786f5c61decb"). InnerVolumeSpecName "kube-api-access-5tmdq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:42:46.600589 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:46.599991    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6f9c6ac-4dbd-4080-9b06-4955c441311c-kube-api-access-slrgc" pod "d6f9c6ac-4dbd-4080-9b06-4955c441311c" (UID: "d6f9c6ac-4dbd-4080-9b06-4955c441311c"). InnerVolumeSpecName "kube-api-access-slrgc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:42:46.600589 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:46.600053    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aab475f7-3a68-4c19-930b-b055767366a9-kube-api-access-xpqg2" pod "aab475f7-3a68-4c19-930b-b055767366a9" (UID: "aab475f7-3a68-4c19-930b-b055767366a9"). InnerVolumeSpecName "kube-api-access-xpqg2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:42:46.600589 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:46.600168    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54116276-13c3-4e07-b8e9-65943636d1e5-bundle" pod "54116276-13c3-4e07-b8e9-65943636d1e5" (UID: "54116276-13c3-4e07-b8e9-65943636d1e5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:42:46.601322 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:46.601285    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5332c799-1c59-4995-8c40-146f82223d75-kube-api-access-6rm9w" pod "5332c799-1c59-4995-8c40-146f82223d75" (UID: "5332c799-1c59-4995-8c40-146f82223d75"). InnerVolumeSpecName "kube-api-access-6rm9w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:42:46.601845 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:46.601824    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54116276-13c3-4e07-b8e9-65943636d1e5-kube-api-access-8nw8w" pod "54116276-13c3-4e07-b8e9-65943636d1e5" (UID: "54116276-13c3-4e07-b8e9-65943636d1e5"). InnerVolumeSpecName "kube-api-access-8nw8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:42:46.602791 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:46.602768    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5332c799-1c59-4995-8c40-146f82223d75-util" pod "5332c799-1c59-4995-8c40-146f82223d75" (UID: "5332c799-1c59-4995-8c40-146f82223d75"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:42:46.606378 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:46.606357    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6f9c6ac-4dbd-4080-9b06-4955c441311c-util" pod "d6f9c6ac-4dbd-4080-9b06-4955c441311c" (UID: "d6f9c6ac-4dbd-4080-9b06-4955c441311c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:42:46.606632 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:46.606610    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dba7dabb-2faf-4472-9b77-786f5c61decb-util" pod "dba7dabb-2faf-4472-9b77-786f5c61decb" (UID: "dba7dabb-2faf-4472-9b77-786f5c61decb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:42:46.606711 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:46.606659    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aab475f7-3a68-4c19-930b-b055767366a9-util" pod "aab475f7-3a68-4c19-930b-b055767366a9" (UID: "aab475f7-3a68-4c19-930b-b055767366a9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:42:46.608104 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:46.608085    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54116276-13c3-4e07-b8e9-65943636d1e5-util" pod "54116276-13c3-4e07-b8e9-65943636d1e5" (UID: "54116276-13c3-4e07-b8e9-65943636d1e5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:42:46.698773 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:46.698698    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpqg2\" (UniqueName: \"kubernetes.io/projected/aab475f7-3a68-4c19-930b-b055767366a9-kube-api-access-xpqg2\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:42:46.698773 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:46.698724    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slrgc\" (UniqueName: \"kubernetes.io/projected/d6f9c6ac-4dbd-4080-9b06-4955c441311c-kube-api-access-slrgc\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:42:46.698773 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:46.698733    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rm9w\" (UniqueName: \"kubernetes.io/projected/5332c799-1c59-4995-8c40-146f82223d75-kube-api-access-6rm9w\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:42:46.698773 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:46.698742    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tmdq\" (UniqueName: \"kubernetes.io/projected/dba7dabb-2faf-4472-9b77-786f5c61decb-kube-api-access-5tmdq\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:42:46.698773 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:46.698752    4525 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dba7dabb-2faf-4472-9b77-786f5c61decb-util\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:42:46.698773 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:46.698760    4525 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/54116276-13c3-4e07-b8e9-65943636d1e5-bundle\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:42:46.698773 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:46.698767    4525 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d6f9c6ac-4dbd-4080-9b06-4955c441311c-util\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:42:46.698773 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:46.698775    4525 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aab475f7-3a68-4c19-930b-b055767366a9-bundle\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:42:46.699112 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:46.698782    4525 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aab475f7-3a68-4c19-930b-b055767366a9-util\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:42:46.699112 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:46.698791    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nw8w\" (UniqueName: \"kubernetes.io/projected/54116276-13c3-4e07-b8e9-65943636d1e5-kube-api-access-8nw8w\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:42:46.699112 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:46.698799    4525 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/54116276-13c3-4e07-b8e9-65943636d1e5-util\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:42:46.699112 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:46.698808    4525 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5332c799-1c59-4995-8c40-146f82223d75-util\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:42:46.699112 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:46.698815    4525 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5332c799-1c59-4995-8c40-146f82223d75-bundle\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:42:47.313385 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:47.313352    4525 generic.go:356] "Generic (PLEG): container finished" podID="5e2853c1-7992-446c-a335-2d2e63565ff3" containerID="d13c713c6b8da2b8f7c5f68a63ae5a9d3b9caf86d9e6e5b82b7cfd6f17958f1d" exitCode=0
Oct 30 18:42:47.313816 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:47.313448    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5h8fw" event={"ID":"5e2853c1-7992-446c-a335-2d2e63565ff3","Type":"ContainerDied","Data":"d13c713c6b8da2b8f7c5f68a63ae5a9d3b9caf86d9e6e5b82b7cfd6f17958f1d"}
Oct 30 18:42:47.315397 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:47.315375    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644sn5hk"
Oct 30 18:42:47.315522 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:47.315375    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644sn5hk" event={"ID":"dba7dabb-2faf-4472-9b77-786f5c61decb","Type":"ContainerDied","Data":"63b04b979693fc01dc2a73b7efed7488d8216b4e9a551eba95da66bbcb5a4331"}
Oct 30 18:42:47.315522 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:47.315502    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63b04b979693fc01dc2a73b7efed7488d8216b4e9a551eba95da66bbcb5a4331"
Oct 30 18:42:47.317274 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:47.317256    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lr5h6"
Oct 30 18:42:47.317364 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:47.317254    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lr5h6" event={"ID":"aab475f7-3a68-4c19-930b-b055767366a9","Type":"ContainerDied","Data":"b441edd01d0a6af7bb9b755dc9705fc7ed8a3939dfa71a70125af525e44b34a6"}
Oct 30 18:42:47.317418 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:47.317369    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b441edd01d0a6af7bb9b755dc9705fc7ed8a3939dfa71a70125af525e44b34a6"
Oct 30 18:42:47.319289 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:47.319263    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chnnzp" event={"ID":"54116276-13c3-4e07-b8e9-65943636d1e5","Type":"ContainerDied","Data":"0e87b8d9b43f322c38bb6f66caa80558a6e140094b0cfe8e3a50a5687b150d18"}
Oct 30 18:42:47.319386 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:47.319319    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e87b8d9b43f322c38bb6f66caa80558a6e140094b0cfe8e3a50a5687b150d18"
Oct 30 18:42:47.319386 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:47.319268    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chnnzp"
Oct 30 18:42:47.321652 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:47.321435    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2gq6q"
Oct 30 18:42:47.321652 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:47.321443    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2gq6q" event={"ID":"5332c799-1c59-4995-8c40-146f82223d75","Type":"ContainerDied","Data":"637157f9a8e5adbecaefef3f8676f2a7c5f7780ae0cabc55ce4b2e492a921cb5"}
Oct 30 18:42:47.321652 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:47.321476    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="637157f9a8e5adbecaefef3f8676f2a7c5f7780ae0cabc55ce4b2e492a921cb5"
Oct 30 18:42:47.323729 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:47.323696    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71crvs5" event={"ID":"d6f9c6ac-4dbd-4080-9b06-4955c441311c","Type":"ContainerDied","Data":"0b33254e978ab0bf233b9121593a11bd9f01d2c2612416c7d9f48316c769c6ab"}
Oct 30 18:42:47.323831 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:47.323735    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b33254e978ab0bf233b9121593a11bd9f01d2c2612416c7d9f48316c769c6ab"
Oct 30 18:42:47.324390 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:47.324082    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71crvs5"
Oct 30 18:42:47.401885 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:42:47.401852    4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d49l" containerName="extractor"
Oct 30 18:42:47.460971 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:47.460947    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c9w86"
Oct 30 18:42:47.494172 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:47.494148    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8sn8c"
Oct 30 18:42:47.605730 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:47.605706    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/555342d6-04c7-43e7-ba14-32e2c3b8fae0-kube-api-access-qm8p8\" (UniqueName: \"kubernetes.io/projected/555342d6-04c7-43e7-ba14-32e2c3b8fae0-kube-api-access-qm8p8\") pod \"555342d6-04c7-43e7-ba14-32e2c3b8fae0\" (UID: \"555342d6-04c7-43e7-ba14-32e2c3b8fae0\") "
Oct 30 18:42:47.605901 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:47.605751    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/555342d6-04c7-43e7-ba14-32e2c3b8fae0-bundle\" (UniqueName: \"kubernetes.io/empty-dir/555342d6-04c7-43e7-ba14-32e2c3b8fae0-bundle\") pod \"555342d6-04c7-43e7-ba14-32e2c3b8fae0\" (UID: \"555342d6-04c7-43e7-ba14-32e2c3b8fae0\") "
Oct 30 18:42:47.605901 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:47.605794    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9eb75e76-3366-457a-972f-e445bb34328e-bundle\" (UniqueName: \"kubernetes.io/empty-dir/9eb75e76-3366-457a-972f-e445bb34328e-bundle\") pod \"9eb75e76-3366-457a-972f-e445bb34328e\" (UID: \"9eb75e76-3366-457a-972f-e445bb34328e\") "
Oct 30 18:42:47.605901 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:47.605817    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/555342d6-04c7-43e7-ba14-32e2c3b8fae0-util\" (UniqueName: \"kubernetes.io/empty-dir/555342d6-04c7-43e7-ba14-32e2c3b8fae0-util\") pod \"555342d6-04c7-43e7-ba14-32e2c3b8fae0\" (UID: \"555342d6-04c7-43e7-ba14-32e2c3b8fae0\") "
Oct 30 18:42:47.605901 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:47.605842    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9eb75e76-3366-457a-972f-e445bb34328e-util\" (UniqueName: \"kubernetes.io/empty-dir/9eb75e76-3366-457a-972f-e445bb34328e-util\") pod \"9eb75e76-3366-457a-972f-e445bb34328e\" (UID: \"9eb75e76-3366-457a-972f-e445bb34328e\") "
Oct 30 18:42:47.605901 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:47.605874    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9eb75e76-3366-457a-972f-e445bb34328e-kube-api-access-jtpvn\" (UniqueName: \"kubernetes.io/projected/9eb75e76-3366-457a-972f-e445bb34328e-kube-api-access-jtpvn\") pod \"9eb75e76-3366-457a-972f-e445bb34328e\" (UID: \"9eb75e76-3366-457a-972f-e445bb34328e\") "
Oct 30 18:42:47.606468 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:47.606433    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eb75e76-3366-457a-972f-e445bb34328e-bundle" pod "9eb75e76-3366-457a-972f-e445bb34328e" (UID: "9eb75e76-3366-457a-972f-e445bb34328e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:42:47.606636 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:47.606613    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/555342d6-04c7-43e7-ba14-32e2c3b8fae0-bundle" pod "555342d6-04c7-43e7-ba14-32e2c3b8fae0" (UID: "555342d6-04c7-43e7-ba14-32e2c3b8fae0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:42:47.608165 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:47.608140    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/555342d6-04c7-43e7-ba14-32e2c3b8fae0-kube-api-access-qm8p8" pod "555342d6-04c7-43e7-ba14-32e2c3b8fae0" (UID: "555342d6-04c7-43e7-ba14-32e2c3b8fae0"). InnerVolumeSpecName "kube-api-access-qm8p8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:42:47.608238 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:47.608206    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9eb75e76-3366-457a-972f-e445bb34328e-kube-api-access-jtpvn" pod "9eb75e76-3366-457a-972f-e445bb34328e" (UID: "9eb75e76-3366-457a-972f-e445bb34328e"). InnerVolumeSpecName "kube-api-access-jtpvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:42:47.614884 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:47.614860    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/555342d6-04c7-43e7-ba14-32e2c3b8fae0-util" pod "555342d6-04c7-43e7-ba14-32e2c3b8fae0" (UID: "555342d6-04c7-43e7-ba14-32e2c3b8fae0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:42:47.615043 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:47.615028    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eb75e76-3366-457a-972f-e445bb34328e-util" pod "9eb75e76-3366-457a-972f-e445bb34328e" (UID: "9eb75e76-3366-457a-972f-e445bb34328e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:42:47.706701 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:47.706657    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qm8p8\" (UniqueName: \"kubernetes.io/projected/555342d6-04c7-43e7-ba14-32e2c3b8fae0-kube-api-access-qm8p8\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:42:47.706701 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:47.706696    4525 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/555342d6-04c7-43e7-ba14-32e2c3b8fae0-bundle\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:42:47.706882 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:47.706725    4525 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9eb75e76-3366-457a-972f-e445bb34328e-bundle\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:42:47.706882 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:47.706734    4525 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/555342d6-04c7-43e7-ba14-32e2c3b8fae0-util\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:42:47.706882 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:47.706741    4525 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9eb75e76-3366-457a-972f-e445bb34328e-util\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:42:47.706882 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:47.706750    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtpvn\" (UniqueName: \"kubernetes.io/projected/9eb75e76-3366-457a-972f-e445bb34328e-kube-api-access-jtpvn\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:42:48.329848 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:48.329820    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c9w86"
Oct 30 18:42:48.329848 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:48.329835    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c9w86" event={"ID":"9eb75e76-3366-457a-972f-e445bb34328e","Type":"ContainerDied","Data":"14fb4c9862c83f4a5318029de9617565ba81c2dfa1d6bca6cfa3c51ef557a287"}
Oct 30 18:42:48.330492 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:48.329877    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14fb4c9862c83f4a5318029de9617565ba81c2dfa1d6bca6cfa3c51ef557a287"
Oct 30 18:42:48.331737 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:48.331708    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8sn8c" event={"ID":"555342d6-04c7-43e7-ba14-32e2c3b8fae0","Type":"ContainerDied","Data":"78ee8f316b48b0e4dd13516cf4f268aa3ddd49f4d08cdc1c47bf863e0762e2ae"}
Oct 30 18:42:48.331869 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:48.331752    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78ee8f316b48b0e4dd13516cf4f268aa3ddd49f4d08cdc1c47bf863e0762e2ae"
Oct 30 18:42:48.331869 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:48.331754    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8sn8c"
Oct 30 18:42:48.459523 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:48.459496    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5h8fw"
Oct 30 18:42:48.513327 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:48.513297    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e2853c1-7992-446c-a335-2d2e63565ff3-util\" (UniqueName: \"kubernetes.io/empty-dir/5e2853c1-7992-446c-a335-2d2e63565ff3-util\") pod \"5e2853c1-7992-446c-a335-2d2e63565ff3\" (UID: \"5e2853c1-7992-446c-a335-2d2e63565ff3\") "
Oct 30 18:42:48.513495 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:48.513369    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e2853c1-7992-446c-a335-2d2e63565ff3-bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e2853c1-7992-446c-a335-2d2e63565ff3-bundle\") pod \"5e2853c1-7992-446c-a335-2d2e63565ff3\" (UID: \"5e2853c1-7992-446c-a335-2d2e63565ff3\") "
Oct 30 18:42:48.513495 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:48.513403    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e2853c1-7992-446c-a335-2d2e63565ff3-kube-api-access-mxgcs\" (UniqueName: \"kubernetes.io/projected/5e2853c1-7992-446c-a335-2d2e63565ff3-kube-api-access-mxgcs\") pod \"5e2853c1-7992-446c-a335-2d2e63565ff3\" (UID: \"5e2853c1-7992-446c-a335-2d2e63565ff3\") "
Oct 30 18:42:48.514099 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:48.514064    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e2853c1-7992-446c-a335-2d2e63565ff3-bundle" pod "5e2853c1-7992-446c-a335-2d2e63565ff3" (UID: "5e2853c1-7992-446c-a335-2d2e63565ff3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:42:48.515646 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:48.515626    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e2853c1-7992-446c-a335-2d2e63565ff3-kube-api-access-mxgcs" pod "5e2853c1-7992-446c-a335-2d2e63565ff3" (UID: "5e2853c1-7992-446c-a335-2d2e63565ff3"). InnerVolumeSpecName "kube-api-access-mxgcs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:42:48.519269 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:48.519246    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e2853c1-7992-446c-a335-2d2e63565ff3-util" pod "5e2853c1-7992-446c-a335-2d2e63565ff3" (UID: "5e2853c1-7992-446c-a335-2d2e63565ff3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:42:48.614833 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:48.614802    4525 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e2853c1-7992-446c-a335-2d2e63565ff3-util\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:42:48.614833 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:48.614829    4525 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e2853c1-7992-446c-a335-2d2e63565ff3-bundle\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:42:48.614833 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:48.614839    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxgcs\" (UniqueName: \"kubernetes.io/projected/5e2853c1-7992-446c-a335-2d2e63565ff3-kube-api-access-mxgcs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:42:49.337632 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:49.337599    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5h8fw" event={"ID":"5e2853c1-7992-446c-a335-2d2e63565ff3","Type":"ContainerDied","Data":"68036ae21b5d6e24c548ac5ff0870b48838f9540cf9e5a81c6c6064d41b4ceae"}
Oct 30 18:42:49.338047 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:49.337636    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c5h8fw"
Oct 30 18:42:49.338047 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:49.337647    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="68036ae21b5d6e24c548ac5ff0870b48838f9540cf9e5a81c6c6064d41b4ceae"
Oct 30 18:42:49.971768 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:49.971731    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-8ff44d5d-wqtwq"]
Oct 30 18:42:49.971999 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:49.971987    4525 cpu_manager.go:414] "containerMap: removing container" podUID="9eb75e76-3366-457a-972f-e445bb34328e" containerName="util"
Oct 30 18:42:49.972043 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:49.972000    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9eb75e76-3366-457a-972f-e445bb34328e" containerName="util"
Oct 30 18:42:49.972043 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:49.972009    4525 cpu_manager.go:414] "containerMap: removing container" podUID="5e2853c1-7992-446c-a335-2d2e63565ff3" containerName="extract"
Oct 30 18:42:49.972043 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:49.972015    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e2853c1-7992-446c-a335-2d2e63565ff3" containerName="extract"
Oct 30 18:42:49.972043 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:49.972025    4525 cpu_manager.go:414] "containerMap: removing container" podUID="5332c799-1c59-4995-8c40-146f82223d75" containerName="util"
Oct 30 18:42:49.972043 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:49.972030    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5332c799-1c59-4995-8c40-146f82223d75" containerName="util"
Oct 30 18:42:49.972043 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:49.972041    4525 cpu_manager.go:414] "containerMap: removing container" podUID="5e2853c1-7992-446c-a335-2d2e63565ff3" containerName="pull"
Oct 30 18:42:49.972218 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:49.972045    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e2853c1-7992-446c-a335-2d2e63565ff3" containerName="pull"
Oct 30 18:42:49.972218 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:49.972052    4525 cpu_manager.go:414] "containerMap: removing container" podUID="54116276-13c3-4e07-b8e9-65943636d1e5" containerName="pull"
Oct 30 18:42:49.972218 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:49.972056    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54116276-13c3-4e07-b8e9-65943636d1e5" containerName="pull"
Oct 30 18:42:49.972218 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:49.972062    4525 cpu_manager.go:414] "containerMap: removing container" podUID="dba7dabb-2faf-4472-9b77-786f5c61decb" containerName="extract"
Oct 30 18:42:49.972218 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:49.972066    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dba7dabb-2faf-4472-9b77-786f5c61decb" containerName="extract"
Oct 30 18:42:49.972218 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:49.972073    4525 cpu_manager.go:414] "containerMap: removing container" podUID="5332c799-1c59-4995-8c40-146f82223d75" containerName="extract"
Oct 30 18:42:49.972218 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:49.972077    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5332c799-1c59-4995-8c40-146f82223d75" containerName="extract"
Oct 30 18:42:49.972218 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:49.972083    4525 cpu_manager.go:414] "containerMap: removing container" podUID="54116276-13c3-4e07-b8e9-65943636d1e5" containerName="extract"
Oct 30 18:42:49.972218 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:49.972087    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54116276-13c3-4e07-b8e9-65943636d1e5" containerName="extract"
Oct 30 18:42:49.972218 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:49.972092    4525 cpu_manager.go:414] "containerMap: removing container" podUID="5332c799-1c59-4995-8c40-146f82223d75" containerName="pull"
Oct 30 18:42:49.972218 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:49.972096    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5332c799-1c59-4995-8c40-146f82223d75" containerName="pull"
Oct 30 18:42:49.972218 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:49.972101    4525 cpu_manager.go:414] "containerMap: removing container" podUID="aab475f7-3a68-4c19-930b-b055767366a9" containerName="extract"
Oct 30 18:42:49.972218 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:49.972105    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aab475f7-3a68-4c19-930b-b055767366a9" containerName="extract"
Oct 30 18:42:49.972218 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:49.972111    4525 cpu_manager.go:414] "containerMap: removing container" podUID="555342d6-04c7-43e7-ba14-32e2c3b8fae0" containerName="extract"
Oct 30 18:42:49.972218 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:49.972115    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="555342d6-04c7-43e7-ba14-32e2c3b8fae0" containerName="extract"
Oct 30 18:42:49.972218 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:49.972123    4525 cpu_manager.go:414] "containerMap: removing container" podUID="f78d8915-08e0-4559-8cfe-a63505f35f46" containerName="pull"
Oct 30 18:42:49.972218 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:49.972127    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f78d8915-08e0-4559-8cfe-a63505f35f46" containerName="pull"
Oct 30 18:42:49.972218 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:49.972133    4525 cpu_manager.go:414] "containerMap: removing container" podUID="aab475f7-3a68-4c19-930b-b055767366a9" containerName="pull"
Oct 30 18:42:49.972218 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:49.972137    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aab475f7-3a68-4c19-930b-b055767366a9" containerName="pull"
Oct 30 18:42:49.972218 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:49.972143    4525 cpu_manager.go:414] "containerMap: removing container" podUID="5e2853c1-7992-446c-a335-2d2e63565ff3" containerName="util"
Oct 30 18:42:49.972218 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:49.972148    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e2853c1-7992-446c-a335-2d2e63565ff3" containerName="util"
Oct 30 18:42:49.972218 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:49.972154    4525 cpu_manager.go:414] "containerMap: removing container" podUID="d6f9c6ac-4dbd-4080-9b06-4955c441311c" containerName="pull"
Oct 30 18:42:49.972218 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:49.972159    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6f9c6ac-4dbd-4080-9b06-4955c441311c" containerName="pull"
Oct 30 18:42:49.972218 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:49.972164    4525 cpu_manager.go:414] "containerMap: removing container" podUID="d6f9c6ac-4dbd-4080-9b06-4955c441311c" containerName="extract"
Oct 30 18:42:49.972218 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:49.972169    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6f9c6ac-4dbd-4080-9b06-4955c441311c" containerName="extract"
Oct 30 18:42:49.972218 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:49.972175    4525 cpu_manager.go:414] "containerMap: removing container" podUID="aab475f7-3a68-4c19-930b-b055767366a9" containerName="util"
Oct 30 18:42:49.972218 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:49.972179    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aab475f7-3a68-4c19-930b-b055767366a9" containerName="util"
Oct 30 18:42:49.972218 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:49.972184    4525 cpu_manager.go:414] "containerMap: removing container" podUID="9eb75e76-3366-457a-972f-e445bb34328e" containerName="extract"
Oct 30 18:42:49.972218 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:49.972188    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9eb75e76-3366-457a-972f-e445bb34328e" containerName="extract"
Oct 30 18:42:49.972218 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:49.972195    4525 cpu_manager.go:414] "containerMap: removing container" podUID="dba7dabb-2faf-4472-9b77-786f5c61decb" containerName="util"
Oct 30 18:42:49.972218 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:49.972199    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dba7dabb-2faf-4472-9b77-786f5c61decb" containerName="util"
Oct 30 18:42:49.972218 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:49.972205    4525 cpu_manager.go:414] "containerMap: removing container" podUID="9eb75e76-3366-457a-972f-e445bb34328e" containerName="pull"
Oct 30 18:42:49.972218 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:49.972209    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9eb75e76-3366-457a-972f-e445bb34328e" containerName="pull"
Oct 30 18:42:49.972218 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:49.972214    4525 cpu_manager.go:414] "containerMap: removing container" podUID="dba7dabb-2faf-4472-9b77-786f5c61decb" containerName="pull"
Oct 30 18:42:49.972218 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:49.972218    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dba7dabb-2faf-4472-9b77-786f5c61decb" containerName="pull"
Oct 30 18:42:49.972218 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:49.972224    4525 cpu_manager.go:414] "containerMap: removing container" podUID="555342d6-04c7-43e7-ba14-32e2c3b8fae0" containerName="util"
Oct 30 18:42:49.972218 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:49.972228    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="555342d6-04c7-43e7-ba14-32e2c3b8fae0" containerName="util"
Oct 30 18:42:49.972218 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:49.972233    4525 cpu_manager.go:414] "containerMap: removing container" podUID="555342d6-04c7-43e7-ba14-32e2c3b8fae0" containerName="pull"
Oct 30 18:42:49.972218 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:49.972237    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="555342d6-04c7-43e7-ba14-32e2c3b8fae0" containerName="pull"
Oct 30 18:42:49.973547 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:49.972244    4525 cpu_manager.go:414] "containerMap: removing container" podUID="54116276-13c3-4e07-b8e9-65943636d1e5" containerName="util"
Oct 30 18:42:49.973547 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:49.972248    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54116276-13c3-4e07-b8e9-65943636d1e5" containerName="util"
Oct 30 18:42:49.973547 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:49.972253    4525 cpu_manager.go:414] "containerMap: removing container" podUID="f78d8915-08e0-4559-8cfe-a63505f35f46" containerName="extract"
Oct 30 18:42:49.973547 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:49.972256    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f78d8915-08e0-4559-8cfe-a63505f35f46" containerName="extract"
Oct 30 18:42:49.973547 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:49.972265    4525 cpu_manager.go:414] "containerMap: removing container" podUID="f78d8915-08e0-4559-8cfe-a63505f35f46" containerName="util"
Oct 30 18:42:49.973547 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:49.972269    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f78d8915-08e0-4559-8cfe-a63505f35f46" containerName="util"
Oct 30 18:42:49.973547 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:49.972274    4525 cpu_manager.go:414] "containerMap: removing container" podUID="d6f9c6ac-4dbd-4080-9b06-4955c441311c" containerName="util"
Oct 30 18:42:49.973547 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:49.972278    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6f9c6ac-4dbd-4080-9b06-4955c441311c" containerName="util"
Oct 30 18:42:49.973547 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:49.972318    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="9eb75e76-3366-457a-972f-e445bb34328e" containerName="extract"
Oct 30 18:42:49.973547 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:49.972326    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="f78d8915-08e0-4559-8cfe-a63505f35f46" containerName="extract"
Oct 30 18:42:49.973547 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:49.972332    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e2853c1-7992-446c-a335-2d2e63565ff3" containerName="extract"
Oct 30 18:42:49.973547 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:49.972339    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="dba7dabb-2faf-4472-9b77-786f5c61decb" containerName="extract"
Oct 30 18:42:49.973547 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:49.972345    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="5332c799-1c59-4995-8c40-146f82223d75" containerName="extract"
Oct 30 18:42:49.973547 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:49.972350    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="54116276-13c3-4e07-b8e9-65943636d1e5" containerName="extract"
Oct 30 18:42:49.973547 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:49.972354    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="555342d6-04c7-43e7-ba14-32e2c3b8fae0" containerName="extract"
Oct 30 18:42:49.973547 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:49.972359    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="aab475f7-3a68-4c19-930b-b055767366a9" containerName="extract"
Oct 30 18:42:49.973547 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:49.972364    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6f9c6ac-4dbd-4080-9b06-4955c441311c" containerName="extract"
Oct 30 18:42:49.975576 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:49.975560    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff44d5d-wqtwq"
Oct 30 18:42:49.982039 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:49.982016    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 30 18:42:49.982183 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:49.982148    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 30 18:42:49.982386 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:49.982368    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-2vxg7\""
Oct 30 18:42:49.982477 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:49.982409    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 30 18:42:49.982477 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:49.982420    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 30 18:42:49.982477 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:49.982437    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 30 18:42:50.011005 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:50.010974    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-8ff44d5d-wqtwq"]
Oct 30 18:42:50.028016 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:50.027984    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vhq8\" (UniqueName: \"kubernetes.io/projected/8c892580-7428-4992-8677-4b480d24ca12-kube-api-access-4vhq8\") pod \"opendatahub-operator-controller-manager-8ff44d5d-wqtwq\" (UID: \"8c892580-7428-4992-8677-4b480d24ca12\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff44d5d-wqtwq"
Oct 30 18:42:50.028016 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:50.028019    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8c892580-7428-4992-8677-4b480d24ca12-metrics-cert\") pod \"opendatahub-operator-controller-manager-8ff44d5d-wqtwq\" (UID: \"8c892580-7428-4992-8677-4b480d24ca12\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff44d5d-wqtwq"
Oct 30 18:42:50.028228 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:50.028049    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8c892580-7428-4992-8677-4b480d24ca12-manifests\") pod \"opendatahub-operator-controller-manager-8ff44d5d-wqtwq\" (UID: \"8c892580-7428-4992-8677-4b480d24ca12\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff44d5d-wqtwq"
Oct 30 18:42:50.028228 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:50.028120    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8c892580-7428-4992-8677-4b480d24ca12-webhook-cert\") pod \"opendatahub-operator-controller-manager-8ff44d5d-wqtwq\" (UID: \"8c892580-7428-4992-8677-4b480d24ca12\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff44d5d-wqtwq"
Oct 30 18:42:50.028228 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:50.028205    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8c892580-7428-4992-8677-4b480d24ca12-tmp\") pod \"opendatahub-operator-controller-manager-8ff44d5d-wqtwq\" (UID: \"8c892580-7428-4992-8677-4b480d24ca12\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff44d5d-wqtwq"
Oct 30 18:42:50.028228 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:50.028226    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8c892580-7428-4992-8677-4b480d24ca12-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8ff44d5d-wqtwq\" (UID: \"8c892580-7428-4992-8677-4b480d24ca12\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff44d5d-wqtwq"
Oct 30 18:42:50.128767 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:50.128728    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8c892580-7428-4992-8677-4b480d24ca12-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8ff44d5d-wqtwq\" (UID: \"8c892580-7428-4992-8677-4b480d24ca12\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff44d5d-wqtwq"
Oct 30 18:42:50.128987 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:50.128783    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vhq8\" (UniqueName: \"kubernetes.io/projected/8c892580-7428-4992-8677-4b480d24ca12-kube-api-access-4vhq8\") pod \"opendatahub-operator-controller-manager-8ff44d5d-wqtwq\" (UID: \"8c892580-7428-4992-8677-4b480d24ca12\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff44d5d-wqtwq"
Oct 30 18:42:50.128987 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:50.128803    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8c892580-7428-4992-8677-4b480d24ca12-metrics-cert\") pod \"opendatahub-operator-controller-manager-8ff44d5d-wqtwq\" (UID: \"8c892580-7428-4992-8677-4b480d24ca12\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff44d5d-wqtwq"
Oct 30 18:42:50.128987 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:50.128836    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8c892580-7428-4992-8677-4b480d24ca12-manifests\") pod \"opendatahub-operator-controller-manager-8ff44d5d-wqtwq\" (UID: \"8c892580-7428-4992-8677-4b480d24ca12\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff44d5d-wqtwq"
Oct 30 18:42:50.128987 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:50.128852    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8c892580-7428-4992-8677-4b480d24ca12-webhook-cert\") pod \"opendatahub-operator-controller-manager-8ff44d5d-wqtwq\" (UID: \"8c892580-7428-4992-8677-4b480d24ca12\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff44d5d-wqtwq"
Oct 30 18:42:50.128987 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:50.128885    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8c892580-7428-4992-8677-4b480d24ca12-tmp\") pod \"opendatahub-operator-controller-manager-8ff44d5d-wqtwq\" (UID: \"8c892580-7428-4992-8677-4b480d24ca12\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff44d5d-wqtwq"
Oct 30 18:42:50.129303 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:50.129276    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8c892580-7428-4992-8677-4b480d24ca12-manifests\") pod \"opendatahub-operator-controller-manager-8ff44d5d-wqtwq\" (UID: \"8c892580-7428-4992-8677-4b480d24ca12\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff44d5d-wqtwq"
Oct 30 18:42:50.131700 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:50.131649    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8c892580-7428-4992-8677-4b480d24ca12-tmp\") pod \"opendatahub-operator-controller-manager-8ff44d5d-wqtwq\" (UID: \"8c892580-7428-4992-8677-4b480d24ca12\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff44d5d-wqtwq"
Oct 30 18:42:50.131835 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:50.131785    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8c892580-7428-4992-8677-4b480d24ca12-metrics-cert\") pod \"opendatahub-operator-controller-manager-8ff44d5d-wqtwq\" (UID: \"8c892580-7428-4992-8677-4b480d24ca12\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff44d5d-wqtwq"
Oct 30 18:42:50.131960 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:50.131940    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8c892580-7428-4992-8677-4b480d24ca12-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8ff44d5d-wqtwq\" (UID: \"8c892580-7428-4992-8677-4b480d24ca12\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff44d5d-wqtwq"
Oct 30 18:42:50.132107 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:50.132087    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8c892580-7428-4992-8677-4b480d24ca12-webhook-cert\") pod \"opendatahub-operator-controller-manager-8ff44d5d-wqtwq\" (UID: \"8c892580-7428-4992-8677-4b480d24ca12\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff44d5d-wqtwq"
Oct 30 18:42:50.143077 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:50.143057    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vhq8\" (UniqueName: \"kubernetes.io/projected/8c892580-7428-4992-8677-4b480d24ca12-kube-api-access-4vhq8\") pod \"opendatahub-operator-controller-manager-8ff44d5d-wqtwq\" (UID: \"8c892580-7428-4992-8677-4b480d24ca12\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff44d5d-wqtwq"
Oct 30 18:42:50.291158 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:50.291076    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff44d5d-wqtwq"
Oct 30 18:42:50.595889 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:50.595817    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-8ff44d5d-wqtwq"]
Oct 30 18:42:50.598606 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:42:50.598579    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c892580_7428_4992_8677_4b480d24ca12.slice/crio-65f7c88cc545e0e362e482696749e5bce343114f095e48aae631060b14d04082 WatchSource:0}: Error finding container 65f7c88cc545e0e362e482696749e5bce343114f095e48aae631060b14d04082: Status 404 returned error can't find the container with id 65f7c88cc545e0e362e482696749e5bce343114f095e48aae631060b14d04082
Oct 30 18:42:51.354964 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:51.354930    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff44d5d-wqtwq" event={"ID":"8c892580-7428-4992-8677-4b480d24ca12","Type":"ContainerStarted","Data":"65f7c88cc545e0e362e482696749e5bce343114f095e48aae631060b14d04082"}
Oct 30 18:42:54.370811 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:54.370776    4525 generic.go:356] "Generic (PLEG): container finished" podID="8c892580-7428-4992-8677-4b480d24ca12" containerID="ee113a0845c7b01523df7338e18a60d72ffbd2c0f25c3cfa4d0f7d26c6398a79" exitCode=0
Oct 30 18:42:54.371243 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:54.370852    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff44d5d-wqtwq" event={"ID":"8c892580-7428-4992-8677-4b480d24ca12","Type":"ContainerDied","Data":"ee113a0845c7b01523df7338e18a60d72ffbd2c0f25c3cfa4d0f7d26c6398a79"}
Oct 30 18:42:55.377184 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:55.377153    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff44d5d-wqtwq" event={"ID":"8c892580-7428-4992-8677-4b480d24ca12","Type":"ContainerStarted","Data":"eb457f447d81fc4d94346ebfc67437efdd0172c4a794dc23fbb2b20f55fb4e4f"}
Oct 30 18:42:55.377579 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:55.377314    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff44d5d-wqtwq"
Oct 30 18:42:55.426897 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:42:55.426839    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff44d5d-wqtwq" podStartSLOduration=2.725365502 podStartE2EDuration="6.426824614s" totalImagesPullingTime="3.228476762s" totalInitContainerRuntime="472.98235ms" isStatefulPod=false podCreationTimestamp="2026-10-30 18:42:49 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:42:55.425150932 +0000 UTC m=+398.050820898" watchObservedRunningTime="2026-10-30 18:42:55.426824614 +0000 UTC m=+398.052494578"
Oct 30 18:43:05.980014 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:05.979934    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7x8n8"]
Oct 30 18:43:05.983026 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:05.983010    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7x8n8"
Oct 30 18:43:05.989846 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:05.989787    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:43:05.989966 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:05.989947    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Oct 30 18:43:05.990780 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:05.990761    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-khttp\""
Oct 30 18:43:06.002566 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:06.002546    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7x8n8"]
Oct 30 18:43:06.164299 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:06.164271    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx5lk\" (UniqueName: \"kubernetes.io/projected/0bcc112e-bfcd-4cca-a1ba-42476da8bb14-kube-api-access-dx5lk\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-7x8n8\" (UID: \"0bcc112e-bfcd-4cca-a1ba-42476da8bb14\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7x8n8"
Oct 30 18:43:06.164459 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:06.164319    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0bcc112e-bfcd-4cca-a1ba-42476da8bb14-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-7x8n8\" (UID: \"0bcc112e-bfcd-4cca-a1ba-42476da8bb14\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7x8n8"
Oct 30 18:43:06.265085 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:06.265016    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx5lk\" (UniqueName: \"kubernetes.io/projected/0bcc112e-bfcd-4cca-a1ba-42476da8bb14-kube-api-access-dx5lk\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-7x8n8\" (UID: \"0bcc112e-bfcd-4cca-a1ba-42476da8bb14\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7x8n8"
Oct 30 18:43:06.265085 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:06.265061    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0bcc112e-bfcd-4cca-a1ba-42476da8bb14-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-7x8n8\" (UID: \"0bcc112e-bfcd-4cca-a1ba-42476da8bb14\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7x8n8"
Oct 30 18:43:06.265363 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:06.265349    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0bcc112e-bfcd-4cca-a1ba-42476da8bb14-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-7x8n8\" (UID: \"0bcc112e-bfcd-4cca-a1ba-42476da8bb14\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7x8n8"
Oct 30 18:43:06.277777 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:06.277752    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx5lk\" (UniqueName: \"kubernetes.io/projected/0bcc112e-bfcd-4cca-a1ba-42476da8bb14-kube-api-access-dx5lk\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-7x8n8\" (UID: \"0bcc112e-bfcd-4cca-a1ba-42476da8bb14\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7x8n8"
Oct 30 18:43:06.293689 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:06.293658    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7x8n8"
Oct 30 18:43:06.383957 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:06.383926    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8ff44d5d-wqtwq"
Oct 30 18:43:06.571792 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:06.571717    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7x8n8"]
Oct 30 18:43:06.575214 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:43:06.575172    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bcc112e_bfcd_4cca_a1ba_42476da8bb14.slice/crio-43bc4b15b5e33c796882706fec9f5a6fc9f05723561ea826073b3f4ed1979db2 WatchSource:0}: Error finding container 43bc4b15b5e33c796882706fec9f5a6fc9f05723561ea826073b3f4ed1979db2: Status 404 returned error can't find the container with id 43bc4b15b5e33c796882706fec9f5a6fc9f05723561ea826073b3f4ed1979db2
Oct 30 18:43:07.429647 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:07.429608    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7x8n8" event={"ID":"0bcc112e-bfcd-4cca-a1ba-42476da8bb14","Type":"ContainerStarted","Data":"43bc4b15b5e33c796882706fec9f5a6fc9f05723561ea826073b3f4ed1979db2"}
Oct 30 18:43:08.435879 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:08.435847    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7x8n8" event={"ID":"0bcc112e-bfcd-4cca-a1ba-42476da8bb14","Type":"ContainerStarted","Data":"5ac3dcde0b4128e44a48fc4a41d3ba5dca7aeaef0577959dd2e86892f2697a31"}
Oct 30 18:43:08.469777 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:08.469659    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7x8n8" podStartSLOduration=1.8529283539999999 podStartE2EDuration="3.469645138s" totalImagesPullingTime="1.616716784s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:43:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:43:08.469433531 +0000 UTC m=+411.095103496" watchObservedRunningTime="2026-10-30 18:43:08.469645138 +0000 UTC m=+411.095315102"
Oct 30 18:43:10.171163 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:10.171123    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-tftjb"]
Oct 30 18:43:10.174636 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:10.174613    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-tftjb"
Oct 30 18:43:10.183669 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:10.183651    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-l8lft\""
Oct 30 18:43:10.183980 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:10.183959    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 30 18:43:10.183980 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:10.183959    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 30 18:43:10.198578 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:10.198546    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-tftjb"]
Oct 30 18:43:10.305481 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:10.305451    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b05444aa-d6e7-469a-a136-fea31c87110d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-tftjb\" (UID: \"b05444aa-d6e7-469a-a136-fea31c87110d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-tftjb"
Oct 30 18:43:10.305645 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:10.305489    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2v2hv\" (UniqueName: \"kubernetes.io/projected/b05444aa-d6e7-469a-a136-fea31c87110d-kube-api-access-2v2hv\") pod \"kuadrant-operator-controller-manager-cf8469d7b-tftjb\" (UID: \"b05444aa-d6e7-469a-a136-fea31c87110d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-tftjb"
Oct 30 18:43:10.406393 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:10.406355    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b05444aa-d6e7-469a-a136-fea31c87110d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-tftjb\" (UID: \"b05444aa-d6e7-469a-a136-fea31c87110d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-tftjb"
Oct 30 18:43:10.406393 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:10.406396    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2v2hv\" (UniqueName: \"kubernetes.io/projected/b05444aa-d6e7-469a-a136-fea31c87110d-kube-api-access-2v2hv\") pod \"kuadrant-operator-controller-manager-cf8469d7b-tftjb\" (UID: \"b05444aa-d6e7-469a-a136-fea31c87110d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-tftjb"
Oct 30 18:43:10.406775 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:10.406754    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b05444aa-d6e7-469a-a136-fea31c87110d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-tftjb\" (UID: \"b05444aa-d6e7-469a-a136-fea31c87110d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-tftjb"
Oct 30 18:43:10.416869 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:10.416845    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2v2hv\" (UniqueName: \"kubernetes.io/projected/b05444aa-d6e7-469a-a136-fea31c87110d-kube-api-access-2v2hv\") pod \"kuadrant-operator-controller-manager-cf8469d7b-tftjb\" (UID: \"b05444aa-d6e7-469a-a136-fea31c87110d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-tftjb"
Oct 30 18:43:10.487855 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:10.487787    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-tftjb"
Oct 30 18:43:10.768285 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:10.768261    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-tftjb"]
Oct 30 18:43:10.769770 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:43:10.769730    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb05444aa_d6e7_469a_a136_fea31c87110d.slice/crio-cf9770ea80e1158497653e8458a76f2e655dd37eb09d9a338f2ab0683602aaf1 WatchSource:0}: Error finding container cf9770ea80e1158497653e8458a76f2e655dd37eb09d9a338f2ab0683602aaf1: Status 404 returned error can't find the container with id cf9770ea80e1158497653e8458a76f2e655dd37eb09d9a338f2ab0683602aaf1
Oct 30 18:43:11.450274 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:11.450236    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-tftjb" event={"ID":"b05444aa-d6e7-469a-a136-fea31c87110d","Type":"ContainerStarted","Data":"cf9770ea80e1158497653e8458a76f2e655dd37eb09d9a338f2ab0683602aaf1"}
Oct 30 18:43:13.871281 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:13.871240    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-xwr2j"]
Oct 30 18:43:13.874744 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:13.874722    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-xwr2j"
Oct 30 18:43:13.885284 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:13.885256    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-xwr2j"]
Oct 30 18:43:13.935402 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:13.935366    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-4vmbq\""
Oct 30 18:43:14.035100 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:14.035058    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2hj5\" (UniqueName: \"kubernetes.io/projected/c5d9d0df-f991-42dc-8889-1bed5e55ed4e-kube-api-access-w2hj5\") pod \"authorino-operator-7b7bb4bc77-xwr2j\" (UID: \"c5d9d0df-f991-42dc-8889-1bed5e55ed4e\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-xwr2j"
Oct 30 18:43:14.136410 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:14.136326    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2hj5\" (UniqueName: \"kubernetes.io/projected/c5d9d0df-f991-42dc-8889-1bed5e55ed4e-kube-api-access-w2hj5\") pod \"authorino-operator-7b7bb4bc77-xwr2j\" (UID: \"c5d9d0df-f991-42dc-8889-1bed5e55ed4e\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-xwr2j"
Oct 30 18:43:14.145906 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:14.145874    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2hj5\" (UniqueName: \"kubernetes.io/projected/c5d9d0df-f991-42dc-8889-1bed5e55ed4e-kube-api-access-w2hj5\") pod \"authorino-operator-7b7bb4bc77-xwr2j\" (UID: \"c5d9d0df-f991-42dc-8889-1bed5e55ed4e\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-xwr2j"
Oct 30 18:43:14.191810 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:14.191779    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-xwr2j"
Oct 30 18:43:14.482938 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:14.482912    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-xwr2j"]
Oct 30 18:43:14.484570 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:43:14.484544    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5d9d0df_f991_42dc_8889_1bed5e55ed4e.slice/crio-f2704ed4315533278f8606c0f23a82d530e4fdbef6f352ad48171ad7cf6755b4 WatchSource:0}: Error finding container f2704ed4315533278f8606c0f23a82d530e4fdbef6f352ad48171ad7cf6755b4: Status 404 returned error can't find the container with id f2704ed4315533278f8606c0f23a82d530e4fdbef6f352ad48171ad7cf6755b4
Oct 30 18:43:15.470904 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:15.470870    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-tftjb" event={"ID":"b05444aa-d6e7-469a-a136-fea31c87110d","Type":"ContainerStarted","Data":"eee944244ec48936c518585bd722a682da45425c964158d835f1701e76f04c2e"}
Oct 30 18:43:15.471380 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:15.470998    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-tftjb"
Oct 30 18:43:15.472243 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:15.472221    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-xwr2j" event={"ID":"c5d9d0df-f991-42dc-8889-1bed5e55ed4e","Type":"ContainerStarted","Data":"f2704ed4315533278f8606c0f23a82d530e4fdbef6f352ad48171ad7cf6755b4"}
Oct 30 18:43:15.494787 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:15.494745    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-tftjb" podStartSLOduration=1.441743432 podStartE2EDuration="5.4947329s" totalImagesPullingTime="4.052989468s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:43:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:43:15.492742377 +0000 UTC m=+418.118412337" watchObservedRunningTime="2026-10-30 18:43:15.4947329 +0000 UTC m=+418.120402865"
Oct 30 18:43:16.482997 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:16.482894    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-xwr2j" event={"ID":"c5d9d0df-f991-42dc-8889-1bed5e55ed4e","Type":"ContainerStarted","Data":"9ea730826b42175d650a6074aa55afbf914dc928e19bd8129ce9d15882e4df6a"}
Oct 30 18:43:16.503062 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:16.503014    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-xwr2j" podStartSLOduration=1.7850051900000001 podStartE2EDuration="3.503001251s" totalImagesPullingTime="1.717996061s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:43:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:43:16.501887662 +0000 UTC m=+419.127557624" watchObservedRunningTime="2026-10-30 18:43:16.503001251 +0000 UTC m=+419.128671214"
Oct 30 18:43:16.859129 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:16.859085    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-k569b"]
Oct 30 18:43:16.862413 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:16.862395    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-k569b"
Oct 30 18:43:16.869847 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:16.869826    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-v542d\""
Oct 30 18:43:16.885372 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:16.885353    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Oct 30 18:43:16.926649 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:16.926621    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-k569b"]
Oct 30 18:43:16.955637 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:16.955611    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj6qh\" (UniqueName: \"kubernetes.io/projected/6eaca66e-2608-40c2-a989-ea16d6b4eb80-kube-api-access-sj6qh\") pod \"dns-operator-controller-manager-8557cbdb84-k569b\" (UID: \"6eaca66e-2608-40c2-a989-ea16d6b4eb80\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-k569b"
Oct 30 18:43:17.057039 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:17.057005    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj6qh\" (UniqueName: \"kubernetes.io/projected/6eaca66e-2608-40c2-a989-ea16d6b4eb80-kube-api-access-sj6qh\") pod \"dns-operator-controller-manager-8557cbdb84-k569b\" (UID: \"6eaca66e-2608-40c2-a989-ea16d6b4eb80\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-k569b"
Oct 30 18:43:17.073101 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:17.073074    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj6qh\" (UniqueName: \"kubernetes.io/projected/6eaca66e-2608-40c2-a989-ea16d6b4eb80-kube-api-access-sj6qh\") pod \"dns-operator-controller-manager-8557cbdb84-k569b\" (UID: \"6eaca66e-2608-40c2-a989-ea16d6b4eb80\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-k569b"
Oct 30 18:43:17.174793 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:17.174704    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-k569b"
Oct 30 18:43:17.395807 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:43:17.395774    4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d49l" containerName="extractor"
Oct 30 18:43:17.448914 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:17.448891    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-k569b"]
Oct 30 18:43:17.451368 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:43:17.451342    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6eaca66e_2608_40c2_a989_ea16d6b4eb80.slice/crio-10895b821c7294ca9088778f07796190975a6aab767f9e0484f39ced4a1754a1 WatchSource:0}: Error finding container 10895b821c7294ca9088778f07796190975a6aab767f9e0484f39ced4a1754a1: Status 404 returned error can't find the container with id 10895b821c7294ca9088778f07796190975a6aab767f9e0484f39ced4a1754a1
Oct 30 18:43:17.488328 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:17.488298    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-k569b" event={"ID":"6eaca66e-2608-40c2-a989-ea16d6b4eb80","Type":"ContainerStarted","Data":"10895b821c7294ca9088778f07796190975a6aab767f9e0484f39ced4a1754a1"}
Oct 30 18:43:17.488730 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:17.488438    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-xwr2j"
Oct 30 18:43:18.340975 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:18.340944    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7d4ffc574frqzst"]
Oct 30 18:43:18.345797 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:18.345775    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d4ffc574frqzst"
Oct 30 18:43:18.352540 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:18.352204    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-krhfj\""
Oct 30 18:43:18.366211 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:18.366184    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7d4ffc574frqzst"]
Oct 30 18:43:18.367249 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:18.367229    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7b2003d-590d-4a27-a88d-3fb30e1133c4-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7d4ffc574frqzst\" (UID: \"b7b2003d-590d-4a27-a88d-3fb30e1133c4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d4ffc574frqzst"
Oct 30 18:43:18.367363 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:18.367261    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7b2003d-590d-4a27-a88d-3fb30e1133c4-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7d4ffc574frqzst\" (UID: \"b7b2003d-590d-4a27-a88d-3fb30e1133c4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d4ffc574frqzst"
Oct 30 18:43:18.367363 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:18.367290    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7b2003d-590d-4a27-a88d-3fb30e1133c4-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7d4ffc574frqzst\" (UID: \"b7b2003d-590d-4a27-a88d-3fb30e1133c4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d4ffc574frqzst"
Oct 30 18:43:18.367435 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:18.367355    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7b2003d-590d-4a27-a88d-3fb30e1133c4-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7d4ffc574frqzst\" (UID: \"b7b2003d-590d-4a27-a88d-3fb30e1133c4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d4ffc574frqzst"
Oct 30 18:43:18.367470 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:18.367429    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7b2003d-590d-4a27-a88d-3fb30e1133c4-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7d4ffc574frqzst\" (UID: \"b7b2003d-590d-4a27-a88d-3fb30e1133c4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d4ffc574frqzst"
Oct 30 18:43:18.367502 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:18.367469    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7b2003d-590d-4a27-a88d-3fb30e1133c4-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7d4ffc574frqzst\" (UID: \"b7b2003d-590d-4a27-a88d-3fb30e1133c4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d4ffc574frqzst"
Oct 30 18:43:18.367502 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:18.367492    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r69b9\" (UniqueName: \"kubernetes.io/projected/b7b2003d-590d-4a27-a88d-3fb30e1133c4-kube-api-access-r69b9\") pod \"data-science-gateway-data-science-gateway-class-7d4ffc574frqzst\" (UID: \"b7b2003d-590d-4a27-a88d-3fb30e1133c4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d4ffc574frqzst"
Oct 30 18:43:18.367564 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:18.367549    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7b2003d-590d-4a27-a88d-3fb30e1133c4-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7d4ffc574frqzst\" (UID: \"b7b2003d-590d-4a27-a88d-3fb30e1133c4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d4ffc574frqzst"
Oct 30 18:43:18.367597 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:18.367578    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7b2003d-590d-4a27-a88d-3fb30e1133c4-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7d4ffc574frqzst\" (UID: \"b7b2003d-590d-4a27-a88d-3fb30e1133c4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d4ffc574frqzst"
Oct 30 18:43:18.468112 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:18.468081    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7b2003d-590d-4a27-a88d-3fb30e1133c4-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7d4ffc574frqzst\" (UID: \"b7b2003d-590d-4a27-a88d-3fb30e1133c4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d4ffc574frqzst"
Oct 30 18:43:18.468112 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:18.468112    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7b2003d-590d-4a27-a88d-3fb30e1133c4-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7d4ffc574frqzst\" (UID: \"b7b2003d-590d-4a27-a88d-3fb30e1133c4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d4ffc574frqzst"
Oct 30 18:43:18.468326 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:18.468133    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7b2003d-590d-4a27-a88d-3fb30e1133c4-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7d4ffc574frqzst\" (UID: \"b7b2003d-590d-4a27-a88d-3fb30e1133c4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d4ffc574frqzst"
Oct 30 18:43:18.468326 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:18.468154    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7b2003d-590d-4a27-a88d-3fb30e1133c4-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7d4ffc574frqzst\" (UID: \"b7b2003d-590d-4a27-a88d-3fb30e1133c4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d4ffc574frqzst"
Oct 30 18:43:18.468326 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:18.468273    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7b2003d-590d-4a27-a88d-3fb30e1133c4-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7d4ffc574frqzst\" (UID: \"b7b2003d-590d-4a27-a88d-3fb30e1133c4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d4ffc574frqzst"
Oct 30 18:43:18.468462 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:18.468330    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7b2003d-590d-4a27-a88d-3fb30e1133c4-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7d4ffc574frqzst\" (UID: \"b7b2003d-590d-4a27-a88d-3fb30e1133c4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d4ffc574frqzst"
Oct 30 18:43:18.468462 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:18.468358    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r69b9\" (UniqueName: \"kubernetes.io/projected/b7b2003d-590d-4a27-a88d-3fb30e1133c4-kube-api-access-r69b9\") pod \"data-science-gateway-data-science-gateway-class-7d4ffc574frqzst\" (UID: \"b7b2003d-590d-4a27-a88d-3fb30e1133c4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d4ffc574frqzst"
Oct 30 18:43:18.468462 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:18.468449    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7b2003d-590d-4a27-a88d-3fb30e1133c4-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7d4ffc574frqzst\" (UID: \"b7b2003d-590d-4a27-a88d-3fb30e1133c4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d4ffc574frqzst"
Oct 30 18:43:18.468609 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:18.468476    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7b2003d-590d-4a27-a88d-3fb30e1133c4-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7d4ffc574frqzst\" (UID: \"b7b2003d-590d-4a27-a88d-3fb30e1133c4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d4ffc574frqzst"
Oct 30 18:43:18.468666 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:18.468645    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7b2003d-590d-4a27-a88d-3fb30e1133c4-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7d4ffc574frqzst\" (UID: \"b7b2003d-590d-4a27-a88d-3fb30e1133c4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d4ffc574frqzst"
Oct 30 18:43:18.468759 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:18.468661    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7b2003d-590d-4a27-a88d-3fb30e1133c4-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7d4ffc574frqzst\" (UID: \"b7b2003d-590d-4a27-a88d-3fb30e1133c4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d4ffc574frqzst"
Oct 30 18:43:18.468807 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:18.468775    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7b2003d-590d-4a27-a88d-3fb30e1133c4-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7d4ffc574frqzst\" (UID: \"b7b2003d-590d-4a27-a88d-3fb30e1133c4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d4ffc574frqzst"
Oct 30 18:43:18.469025 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:18.469006    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7b2003d-590d-4a27-a88d-3fb30e1133c4-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7d4ffc574frqzst\" (UID: \"b7b2003d-590d-4a27-a88d-3fb30e1133c4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d4ffc574frqzst"
Oct 30 18:43:18.469359 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:18.469338    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7b2003d-590d-4a27-a88d-3fb30e1133c4-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7d4ffc574frqzst\" (UID: \"b7b2003d-590d-4a27-a88d-3fb30e1133c4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d4ffc574frqzst"
Oct 30 18:43:18.470855 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:18.470834    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7b2003d-590d-4a27-a88d-3fb30e1133c4-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7d4ffc574frqzst\" (UID: \"b7b2003d-590d-4a27-a88d-3fb30e1133c4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d4ffc574frqzst"
Oct 30 18:43:18.471622 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:18.471606    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7b2003d-590d-4a27-a88d-3fb30e1133c4-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7d4ffc574frqzst\" (UID: \"b7b2003d-590d-4a27-a88d-3fb30e1133c4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d4ffc574frqzst"
Oct 30 18:43:18.477116 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:18.477070    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7b2003d-590d-4a27-a88d-3fb30e1133c4-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7d4ffc574frqzst\" (UID: \"b7b2003d-590d-4a27-a88d-3fb30e1133c4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d4ffc574frqzst"
Oct 30 18:43:18.477270 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:18.477251    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r69b9\" (UniqueName: \"kubernetes.io/projected/b7b2003d-590d-4a27-a88d-3fb30e1133c4-kube-api-access-r69b9\") pod \"data-science-gateway-data-science-gateway-class-7d4ffc574frqzst\" (UID: \"b7b2003d-590d-4a27-a88d-3fb30e1133c4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d4ffc574frqzst"
Oct 30 18:43:18.658594 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:18.658566    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d4ffc574frqzst"
Oct 30 18:43:18.945309 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:18.945284    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7d4ffc574frqzst"]
Oct 30 18:43:18.945594 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:43:18.945575    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7b2003d_590d_4a27_a88d_3fb30e1133c4.slice/crio-acb25f708565932ac07a453b146e0a7d9da0f399eaf1a60aaac422f918bce6a6 WatchSource:0}: Error finding container acb25f708565932ac07a453b146e0a7d9da0f399eaf1a60aaac422f918bce6a6: Status 404 returned error can't find the container with id acb25f708565932ac07a453b146e0a7d9da0f399eaf1a60aaac422f918bce6a6
Oct 30 18:43:18.947884 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:18.947853    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:43:18.947969 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:18.947924    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:43:18.947969 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:18.947956    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:43:19.500347 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:19.500312    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d4ffc574frqzst" event={"ID":"b7b2003d-590d-4a27-a88d-3fb30e1133c4","Type":"ContainerStarted","Data":"c5bd38e5c33fa4ab7f2a78e82da6ec85d8be755d555edb63bd4abb454b7795a5"}
Oct 30 18:43:19.500625 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:19.500362    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d4ffc574frqzst" event={"ID":"b7b2003d-590d-4a27-a88d-3fb30e1133c4","Type":"ContainerStarted","Data":"acb25f708565932ac07a453b146e0a7d9da0f399eaf1a60aaac422f918bce6a6"}
Oct 30 18:43:19.536255 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:19.536194    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d4ffc574frqzst" podStartSLOduration=1.536179315 podStartE2EDuration="1.536179315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:43:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:43:19.534446178 +0000 UTC m=+422.160116187" watchObservedRunningTime="2026-10-30 18:43:19.536179315 +0000 UTC m=+422.161849278"
Oct 30 18:43:19.659602 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:19.659518    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d4ffc574frqzst"
Oct 30 18:43:19.665887 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:19.665858    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d4ffc574frqzst"
Oct 30 18:43:20.239343 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:20.239322    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Oct 30 18:43:20.334549 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:20.334516    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-ls657"]
Oct 30 18:43:20.338778 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:20.338749    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ls657"
Oct 30 18:43:20.342414 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:20.342393    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-98m5n\""
Oct 30 18:43:20.349018 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:20.348983    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-ls657"]
Oct 30 18:43:20.371241 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:20.371218    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 18:43:20.371385 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:20.371354    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 18:43:20.374688 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:20.374660    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\""
Oct 30 18:43:20.390942 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:20.390916    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0c027763-f795-4d38-8c7f-f366f8ae47c6-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-ls657\" (UID: \"0c027763-f795-4d38-8c7f-f366f8ae47c6\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ls657"
Oct 30 18:43:20.391060 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:20.390965    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0c027763-f795-4d38-8c7f-f366f8ae47c6-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-ls657\" (UID: \"0c027763-f795-4d38-8c7f-f366f8ae47c6\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ls657"
Oct 30 18:43:20.391060 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:20.390990    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0c027763-f795-4d38-8c7f-f366f8ae47c6-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-ls657\" (UID: \"0c027763-f795-4d38-8c7f-f366f8ae47c6\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ls657"
Oct 30 18:43:20.391060 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:20.391046    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tffr7\" (UniqueName: \"kubernetes.io/projected/0c027763-f795-4d38-8c7f-f366f8ae47c6-kube-api-access-tffr7\") pod \"kserve-module-controller-manager-7566cf5fc8-ls657\" (UID: \"0c027763-f795-4d38-8c7f-f366f8ae47c6\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ls657"
Oct 30 18:43:20.492308 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:20.492240    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0c027763-f795-4d38-8c7f-f366f8ae47c6-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-ls657\" (UID: \"0c027763-f795-4d38-8c7f-f366f8ae47c6\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ls657"
Oct 30 18:43:20.492432 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:20.492304    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0c027763-f795-4d38-8c7f-f366f8ae47c6-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-ls657\" (UID: \"0c027763-f795-4d38-8c7f-f366f8ae47c6\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ls657"
Oct 30 18:43:20.492432 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:20.492352    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0c027763-f795-4d38-8c7f-f366f8ae47c6-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-ls657\" (UID: \"0c027763-f795-4d38-8c7f-f366f8ae47c6\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ls657"
Oct 30 18:43:20.492432 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:20.492391    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tffr7\" (UniqueName: \"kubernetes.io/projected/0c027763-f795-4d38-8c7f-f366f8ae47c6-kube-api-access-tffr7\") pod \"kserve-module-controller-manager-7566cf5fc8-ls657\" (UID: \"0c027763-f795-4d38-8c7f-f366f8ae47c6\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ls657"
Oct 30 18:43:20.492672 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:20.492656    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0c027763-f795-4d38-8c7f-f366f8ae47c6-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-ls657\" (UID: \"0c027763-f795-4d38-8c7f-f366f8ae47c6\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ls657"
Oct 30 18:43:20.492729 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:20.492703    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0c027763-f795-4d38-8c7f-f366f8ae47c6-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-ls657\" (UID: \"0c027763-f795-4d38-8c7f-f366f8ae47c6\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ls657"
Oct 30 18:43:20.493337 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:20.493320    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0c027763-f795-4d38-8c7f-f366f8ae47c6-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-ls657\" (UID: \"0c027763-f795-4d38-8c7f-f366f8ae47c6\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ls657"
Oct 30 18:43:20.507138 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:20.507103    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-k569b" event={"ID":"6eaca66e-2608-40c2-a989-ea16d6b4eb80","Type":"ContainerStarted","Data":"72f263c2f0da1ac6066a71f4a24b6a3deb95c71c645b95bc083011b527ae9cd3"}
Oct 30 18:43:20.507302 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:20.507204    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-k569b"
Oct 30 18:43:20.507569 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:20.507554    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d4ffc574frqzst"
Oct 30 18:43:20.508489 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:20.508474    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d4ffc574frqzst"
Oct 30 18:43:20.508868 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:20.508848    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tffr7\" (UniqueName: \"kubernetes.io/projected/0c027763-f795-4d38-8c7f-f366f8ae47c6-kube-api-access-tffr7\") pod \"kserve-module-controller-manager-7566cf5fc8-ls657\" (UID: \"0c027763-f795-4d38-8c7f-f366f8ae47c6\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ls657"
Oct 30 18:43:20.532451 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:20.532408    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-k569b" podStartSLOduration=1.749718192 podStartE2EDuration="4.532395998s" totalImagesPullingTime="2.782677806s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:43:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:43:20.530235798 +0000 UTC m=+423.155905763" watchObservedRunningTime="2026-10-30 18:43:20.532395998 +0000 UTC m=+423.158065961"
Oct 30 18:43:20.651922 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:20.651891    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ls657"
Oct 30 18:43:20.721040 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:20.721016    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-w68l9"]
Oct 30 18:43:20.726922 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:20.726898    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-w68l9"
Oct 30 18:43:20.740155 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:20.740114    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-w68l9"]
Oct 30 18:43:20.763195 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:20.763136    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-pttlx\""
Oct 30 18:43:20.794253 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:20.794220    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs5vg\" (UniqueName: \"kubernetes.io/projected/ef0c72de-55ce-4860-89a0-e4bad58e11bb-kube-api-access-vs5vg\") pod \"limitador-operator-controller-manager-5778bb4449-w68l9\" (UID: \"ef0c72de-55ce-4860-89a0-e4bad58e11bb\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-w68l9"
Oct 30 18:43:20.894571 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:20.894532    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs5vg\" (UniqueName: \"kubernetes.io/projected/ef0c72de-55ce-4860-89a0-e4bad58e11bb-kube-api-access-vs5vg\") pod \"limitador-operator-controller-manager-5778bb4449-w68l9\" (UID: \"ef0c72de-55ce-4860-89a0-e4bad58e11bb\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-w68l9"
Oct 30 18:43:20.911502 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:20.911478    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs5vg\" (UniqueName: \"kubernetes.io/projected/ef0c72de-55ce-4860-89a0-e4bad58e11bb-kube-api-access-vs5vg\") pod \"limitador-operator-controller-manager-5778bb4449-w68l9\" (UID: \"ef0c72de-55ce-4860-89a0-e4bad58e11bb\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-w68l9"
Oct 30 18:43:20.939035 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:20.939015    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-ls657"]
Oct 30 18:43:20.940844 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:43:20.940820    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c027763_f795_4d38_8c7f_f366f8ae47c6.slice/crio-d158ec0ad26604c3bd111d53703a620f604ca40f181049f08caccb71af45b5fd WatchSource:0}: Error finding container d158ec0ad26604c3bd111d53703a620f604ca40f181049f08caccb71af45b5fd: Status 404 returned error can't find the container with id d158ec0ad26604c3bd111d53703a620f604ca40f181049f08caccb71af45b5fd
Oct 30 18:43:21.041171 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:21.041091    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-w68l9"
Oct 30 18:43:21.322401 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:21.322370    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-w68l9"]
Oct 30 18:43:21.323976 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:43:21.323950    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef0c72de_55ce_4860_89a0_e4bad58e11bb.slice/crio-2cd988dde935e514ace61453fdae57b44a7a1d8f72b5bfd51799e20337253a50 WatchSource:0}: Error finding container 2cd988dde935e514ace61453fdae57b44a7a1d8f72b5bfd51799e20337253a50: Status 404 returned error can't find the container with id 2cd988dde935e514ace61453fdae57b44a7a1d8f72b5bfd51799e20337253a50
Oct 30 18:43:21.530900 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:21.530862    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-w68l9" event={"ID":"ef0c72de-55ce-4860-89a0-e4bad58e11bb","Type":"ContainerStarted","Data":"2cd988dde935e514ace61453fdae57b44a7a1d8f72b5bfd51799e20337253a50"}
Oct 30 18:43:21.534131 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:21.533076    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ls657" event={"ID":"0c027763-f795-4d38-8c7f-f366f8ae47c6","Type":"ContainerStarted","Data":"d158ec0ad26604c3bd111d53703a620f604ca40f181049f08caccb71af45b5fd"}
Oct 30 18:43:22.542886 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:22.542857    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-7thx5"]
Oct 30 18:43:22.553252 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:22.553208    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-7thx5"
Oct 30 18:43:22.555983 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:22.555933    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-7thx5"]
Oct 30 18:43:22.564460 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:22.563854    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Oct 30 18:43:22.564460 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:22.564176    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Oct 30 18:43:22.564460 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:22.564290    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-bn9v4\""
Oct 30 18:43:22.610667 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:22.610632    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/85996be7-be72-43a3-b9a4-1d6b03f74b17-bound-sa-token\") pod \"cert-manager-597b95cb84-7thx5\" (UID: \"85996be7-be72-43a3-b9a4-1d6b03f74b17\") " pod="cert-manager/cert-manager-597b95cb84-7thx5"
Oct 30 18:43:22.611150 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:22.611123    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7fxb\" (UniqueName: \"kubernetes.io/projected/85996be7-be72-43a3-b9a4-1d6b03f74b17-kube-api-access-l7fxb\") pod \"cert-manager-597b95cb84-7thx5\" (UID: \"85996be7-be72-43a3-b9a4-1d6b03f74b17\") " pod="cert-manager/cert-manager-597b95cb84-7thx5"
Oct 30 18:43:22.712206 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:22.712173    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/85996be7-be72-43a3-b9a4-1d6b03f74b17-bound-sa-token\") pod \"cert-manager-597b95cb84-7thx5\" (UID: \"85996be7-be72-43a3-b9a4-1d6b03f74b17\") " pod="cert-manager/cert-manager-597b95cb84-7thx5"
Oct 30 18:43:22.712389 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:22.712253    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7fxb\" (UniqueName: \"kubernetes.io/projected/85996be7-be72-43a3-b9a4-1d6b03f74b17-kube-api-access-l7fxb\") pod \"cert-manager-597b95cb84-7thx5\" (UID: \"85996be7-be72-43a3-b9a4-1d6b03f74b17\") " pod="cert-manager/cert-manager-597b95cb84-7thx5"
Oct 30 18:43:22.722121 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:22.722090    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/85996be7-be72-43a3-b9a4-1d6b03f74b17-bound-sa-token\") pod \"cert-manager-597b95cb84-7thx5\" (UID: \"85996be7-be72-43a3-b9a4-1d6b03f74b17\") " pod="cert-manager/cert-manager-597b95cb84-7thx5"
Oct 30 18:43:22.722121 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:22.722112    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7fxb\" (UniqueName: \"kubernetes.io/projected/85996be7-be72-43a3-b9a4-1d6b03f74b17-kube-api-access-l7fxb\") pod \"cert-manager-597b95cb84-7thx5\" (UID: \"85996be7-be72-43a3-b9a4-1d6b03f74b17\") " pod="cert-manager/cert-manager-597b95cb84-7thx5"
Oct 30 18:43:22.878925 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:22.878889    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-7thx5"
Oct 30 18:43:23.172343 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:23.172314    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-7thx5"]
Oct 30 18:43:23.174628 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:43:23.174599    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod85996be7_be72_43a3_b9a4_1d6b03f74b17.slice/crio-eeeac47312a427752e210e94c68218a57864d6d883b03340c58ed64fe5e801bc WatchSource:0}: Error finding container eeeac47312a427752e210e94c68218a57864d6d883b03340c58ed64fe5e801bc: Status 404 returned error can't find the container with id eeeac47312a427752e210e94c68218a57864d6d883b03340c58ed64fe5e801bc
Oct 30 18:43:23.544483 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:23.544451    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-w68l9" event={"ID":"ef0c72de-55ce-4860-89a0-e4bad58e11bb","Type":"ContainerStarted","Data":"4b70b62ff2f28a744440122baee9abe92682031a505bd7c94b82186a27491b82"}
Oct 30 18:43:23.544942 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:23.544587    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-w68l9"
Oct 30 18:43:23.545697 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:23.545664    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-7thx5" event={"ID":"85996be7-be72-43a3-b9a4-1d6b03f74b17","Type":"ContainerStarted","Data":"eeeac47312a427752e210e94c68218a57864d6d883b03340c58ed64fe5e801bc"}
Oct 30 18:43:23.547010 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:23.546990    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ls657" event={"ID":"0c027763-f795-4d38-8c7f-f366f8ae47c6","Type":"ContainerStarted","Data":"956642362f39edc8a0fe72bbf4f247be2503eedfcc9f537d9e1daba16a0fc295"}
Oct 30 18:43:23.547132 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:23.547118    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ls657"
Oct 30 18:43:23.562188 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:23.562147    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-w68l9" podStartSLOduration=1.533500438 podStartE2EDuration="3.562135002s" totalImagesPullingTime="2.028634564s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:43:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:43:23.561370095 +0000 UTC m=+426.187040059" watchObservedRunningTime="2026-10-30 18:43:23.562135002 +0000 UTC m=+426.187804966"
Oct 30 18:43:23.581083 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:23.581034    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ls657" podStartSLOduration=1.118947946 podStartE2EDuration="3.581019466s" totalImagesPullingTime="2.46207152s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:43:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:43:23.579649559 +0000 UTC m=+426.205319553" watchObservedRunningTime="2026-10-30 18:43:23.581019466 +0000 UTC m=+426.206689458"
Oct 30 18:43:26.485443 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:26.485411    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-tftjb"
Oct 30 18:43:26.563901 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:26.563858    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-7thx5" event={"ID":"85996be7-be72-43a3-b9a4-1d6b03f74b17","Type":"ContainerStarted","Data":"ba47fdcd57bf7329f20136567b60fa05cac1fa119020db58e6f0b63b8f409d22"}
Oct 30 18:43:28.495337 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:28.495308    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-xwr2j"
Oct 30 18:43:28.515607 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:28.515558    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-7thx5" podStartSLOduration=3.698130161 podStartE2EDuration="6.515546216s" totalImagesPullingTime="2.817416055s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:43:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:43:26.588989784 +0000 UTC m=+429.214659748" watchObservedRunningTime="2026-10-30 18:43:28.515546216 +0000 UTC m=+431.141216183"
Oct 30 18:43:31.535937 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:31.535906    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-k569b"
Oct 30 18:43:34.555065 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:34.555036    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-w68l9"
Oct 30 18:43:43.260656 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:43.260615    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-tftjb"]
Oct 30 18:43:43.261165 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:43.260865    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-tftjb" podUID="b05444aa-d6e7-469a-a136-fea31c87110d" containerName="manager" containerID="cri-o://eee944244ec48936c518585bd722a682da45425c964158d835f1701e76f04c2e" gracePeriod=10
Oct 30 18:43:43.523672 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:43.523588    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-tftjb"
Oct 30 18:43:43.595442 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:43.595406    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b05444aa-d6e7-469a-a136-fea31c87110d-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b05444aa-d6e7-469a-a136-fea31c87110d-extensions-socket-volume\") pod \"b05444aa-d6e7-469a-a136-fea31c87110d\" (UID: \"b05444aa-d6e7-469a-a136-fea31c87110d\") "
Oct 30 18:43:43.595625 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:43.595554    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b05444aa-d6e7-469a-a136-fea31c87110d-kube-api-access-2v2hv\" (UniqueName: \"kubernetes.io/projected/b05444aa-d6e7-469a-a136-fea31c87110d-kube-api-access-2v2hv\") pod \"b05444aa-d6e7-469a-a136-fea31c87110d\" (UID: \"b05444aa-d6e7-469a-a136-fea31c87110d\") "
Oct 30 18:43:43.595824 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:43.595799    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b05444aa-d6e7-469a-a136-fea31c87110d-extensions-socket-volume" pod "b05444aa-d6e7-469a-a136-fea31c87110d" (UID: "b05444aa-d6e7-469a-a136-fea31c87110d"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:43:43.597959 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:43.597934    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b05444aa-d6e7-469a-a136-fea31c87110d-kube-api-access-2v2hv" pod "b05444aa-d6e7-469a-a136-fea31c87110d" (UID: "b05444aa-d6e7-469a-a136-fea31c87110d"). InnerVolumeSpecName "kube-api-access-2v2hv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:43:43.645290 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:43.645257    4525 generic.go:356] "Generic (PLEG): container finished" podID="b05444aa-d6e7-469a-a136-fea31c87110d" containerID="eee944244ec48936c518585bd722a682da45425c964158d835f1701e76f04c2e" exitCode=0
Oct 30 18:43:43.645495 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:43.645303    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-tftjb"
Oct 30 18:43:43.645495 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:43.645346    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-tftjb" event={"ID":"b05444aa-d6e7-469a-a136-fea31c87110d","Type":"ContainerDied","Data":"eee944244ec48936c518585bd722a682da45425c964158d835f1701e76f04c2e"}
Oct 30 18:43:43.645495 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:43.645406    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-tftjb" event={"ID":"b05444aa-d6e7-469a-a136-fea31c87110d","Type":"ContainerDied","Data":"cf9770ea80e1158497653e8458a76f2e655dd37eb09d9a338f2ab0683602aaf1"}
Oct 30 18:43:43.645495 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:43.645437    4525 scope.go:122] "RemoveContainer" containerID="eee944244ec48936c518585bd722a682da45425c964158d835f1701e76f04c2e"
Oct 30 18:43:43.654466 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:43.654441    4525 scope.go:122] "RemoveContainer" containerID="eee944244ec48936c518585bd722a682da45425c964158d835f1701e76f04c2e"
Oct 30 18:43:43.654831 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:43:43.654805    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eee944244ec48936c518585bd722a682da45425c964158d835f1701e76f04c2e\": container with ID starting with eee944244ec48936c518585bd722a682da45425c964158d835f1701e76f04c2e not found: ID does not exist" containerID="eee944244ec48936c518585bd722a682da45425c964158d835f1701e76f04c2e"
Oct 30 18:43:43.654890 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:43.654843    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eee944244ec48936c518585bd722a682da45425c964158d835f1701e76f04c2e"} err="failed to get container status \"eee944244ec48936c518585bd722a682da45425c964158d835f1701e76f04c2e\": rpc error: code = NotFound desc = could not find container \"eee944244ec48936c518585bd722a682da45425c964158d835f1701e76f04c2e\": container with ID starting with eee944244ec48936c518585bd722a682da45425c964158d835f1701e76f04c2e not found: ID does not exist"
Oct 30 18:43:43.682298 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:43.682258    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-tftjb"]
Oct 30 18:43:43.689093 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:43.689064    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-tftjb"]
Oct 30 18:43:43.696256 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:43.696223    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2v2hv\" (UniqueName: \"kubernetes.io/projected/b05444aa-d6e7-469a-a136-fea31c87110d-kube-api-access-2v2hv\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:43:43.696256 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:43.696251    4525 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b05444aa-d6e7-469a-a136-fea31c87110d-extensions-socket-volume\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:43:44.553330 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:44.553298    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ls657"
Oct 30 18:43:45.542950 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:45.542914    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b05444aa-d6e7-469a-a136-fea31c87110d" path="/var/lib/kubelet/pods/b05444aa-d6e7-469a-a136-fea31c87110d/volumes"
Oct 30 18:43:47.396128 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:43:47.396088    4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d49l" containerName="extractor"
Oct 30 18:43:59.277037 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:59.276989    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-2j9vp"]
Oct 30 18:43:59.277651 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:59.277342    4525 cpu_manager.go:414] "containerMap: removing container" podUID="b05444aa-d6e7-469a-a136-fea31c87110d" containerName="manager"
Oct 30 18:43:59.277651 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:59.277352    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b05444aa-d6e7-469a-a136-fea31c87110d" containerName="manager"
Oct 30 18:43:59.277651 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:59.277402    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="b05444aa-d6e7-469a-a136-fea31c87110d" containerName="manager"
Oct 30 18:43:59.280280 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:59.280256    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-2j9vp"
Oct 30 18:43:59.286639 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:59.286610    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-fzh8z\""
Oct 30 18:43:59.289752 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:59.289723    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-2j9vp"]
Oct 30 18:43:59.293877 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:59.293856    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Oct 30 18:43:59.388244 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:59.388206    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-2j9vp"]
Oct 30 18:43:59.432788 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:59.432745    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/de63be4b-4390-4692-a698-61a3c6bcc10f-config-file\") pod \"limitador-limitador-74dfcfb894-2j9vp\" (UID: \"de63be4b-4390-4692-a698-61a3c6bcc10f\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-2j9vp"
Oct 30 18:43:59.432962 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:59.432867    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8nm9\" (UniqueName: \"kubernetes.io/projected/de63be4b-4390-4692-a698-61a3c6bcc10f-kube-api-access-x8nm9\") pod \"limitador-limitador-74dfcfb894-2j9vp\" (UID: \"de63be4b-4390-4692-a698-61a3c6bcc10f\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-2j9vp"
Oct 30 18:43:59.533326 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:59.533232    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8nm9\" (UniqueName: \"kubernetes.io/projected/de63be4b-4390-4692-a698-61a3c6bcc10f-kube-api-access-x8nm9\") pod \"limitador-limitador-74dfcfb894-2j9vp\" (UID: \"de63be4b-4390-4692-a698-61a3c6bcc10f\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-2j9vp"
Oct 30 18:43:59.533326 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:59.533297    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/de63be4b-4390-4692-a698-61a3c6bcc10f-config-file\") pod \"limitador-limitador-74dfcfb894-2j9vp\" (UID: \"de63be4b-4390-4692-a698-61a3c6bcc10f\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-2j9vp"
Oct 30 18:43:59.534009 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:59.533984    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/de63be4b-4390-4692-a698-61a3c6bcc10f-config-file\") pod \"limitador-limitador-74dfcfb894-2j9vp\" (UID: \"de63be4b-4390-4692-a698-61a3c6bcc10f\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-2j9vp"
Oct 30 18:43:59.541958 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:59.541932    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8nm9\" (UniqueName: \"kubernetes.io/projected/de63be4b-4390-4692-a698-61a3c6bcc10f-kube-api-access-x8nm9\") pod \"limitador-limitador-74dfcfb894-2j9vp\" (UID: \"de63be4b-4390-4692-a698-61a3c6bcc10f\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-2j9vp"
Oct 30 18:43:59.594839 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:59.594802    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-2j9vp"
Oct 30 18:43:59.871575 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:43:59.871548    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-2j9vp"]
Oct 30 18:43:59.872812 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:43:59.872782    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde63be4b_4390_4692_a698_61a3c6bcc10f.slice/crio-959f99dfad1de7295ee46767073c55967735391634ed764e636b9510f9b35873 WatchSource:0}: Error finding container 959f99dfad1de7295ee46767073c55967735391634ed764e636b9510f9b35873: Status 404 returned error can't find the container with id 959f99dfad1de7295ee46767073c55967735391634ed764e636b9510f9b35873
Oct 30 18:44:00.727086 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:00.727040    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-2j9vp" event={"ID":"de63be4b-4390-4692-a698-61a3c6bcc10f","Type":"ContainerStarted","Data":"959f99dfad1de7295ee46767073c55967735391634ed764e636b9510f9b35873"}
Oct 30 18:44:01.098817 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:01.098668    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-zt6bv"]
Oct 30 18:44:01.101566 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:01.101545    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-zt6bv"
Oct 30 18:44:01.109428 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:01.109387    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-99gx9\""
Oct 30 18:44:01.110749 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:01.110729    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-zt6bv"]
Oct 30 18:44:01.249266 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:01.249230    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzq9w\" (UniqueName: \"kubernetes.io/projected/98374c90-0ab3-45ef-8a11-115ff09da150-kube-api-access-hzq9w\") pod \"authorino-57f7f8659f-zt6bv\" (UID: \"98374c90-0ab3-45ef-8a11-115ff09da150\") " pod="kuadrant-system/authorino-57f7f8659f-zt6bv"
Oct 30 18:44:01.350181 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:01.350101    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzq9w\" (UniqueName: \"kubernetes.io/projected/98374c90-0ab3-45ef-8a11-115ff09da150-kube-api-access-hzq9w\") pod \"authorino-57f7f8659f-zt6bv\" (UID: \"98374c90-0ab3-45ef-8a11-115ff09da150\") " pod="kuadrant-system/authorino-57f7f8659f-zt6bv"
Oct 30 18:44:01.359430 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:01.359400    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzq9w\" (UniqueName: \"kubernetes.io/projected/98374c90-0ab3-45ef-8a11-115ff09da150-kube-api-access-hzq9w\") pod \"authorino-57f7f8659f-zt6bv\" (UID: \"98374c90-0ab3-45ef-8a11-115ff09da150\") " pod="kuadrant-system/authorino-57f7f8659f-zt6bv"
Oct 30 18:44:01.416120 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:01.416078    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-zt6bv"
Oct 30 18:44:01.695852 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:01.695824    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-zt6bv"]
Oct 30 18:44:01.697497 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:44:01.697456    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod98374c90_0ab3_45ef_8a11_115ff09da150.slice/crio-5f0a7016207ac1e38d4ca8115ca4d632a9bc9eba3af34220640ed0cb1febea65 WatchSource:0}: Error finding container 5f0a7016207ac1e38d4ca8115ca4d632a9bc9eba3af34220640ed0cb1febea65: Status 404 returned error can't find the container with id 5f0a7016207ac1e38d4ca8115ca4d632a9bc9eba3af34220640ed0cb1febea65
Oct 30 18:44:01.733065 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:01.733033    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-2j9vp" event={"ID":"de63be4b-4390-4692-a698-61a3c6bcc10f","Type":"ContainerStarted","Data":"0393765a13453e5f16fbb7eee1c25fe5aa2e3674a95132450205da94dc12896d"}
Oct 30 18:44:01.733555 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:01.733137    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-2j9vp"
Oct 30 18:44:01.734278 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:01.734251    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-zt6bv" event={"ID":"98374c90-0ab3-45ef-8a11-115ff09da150","Type":"ContainerStarted","Data":"5f0a7016207ac1e38d4ca8115ca4d632a9bc9eba3af34220640ed0cb1febea65"}
Oct 30 18:44:01.750940 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:01.750893    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-2j9vp" podStartSLOduration=1.018977269 podStartE2EDuration="2.750881188s" totalImagesPullingTime="1.731903919s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:43:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:44:01.749379985 +0000 UTC m=+464.375049949" watchObservedRunningTime="2026-10-30 18:44:01.750881188 +0000 UTC m=+464.376551152"
Oct 30 18:44:03.747491 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:03.747448    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-zt6bv" event={"ID":"98374c90-0ab3-45ef-8a11-115ff09da150","Type":"ContainerStarted","Data":"05a4124a8ce6ee1a1b1c627b41dfc1e67689b34fa267ddf1b2ba6b391fc0a45e"}
Oct 30 18:44:03.763347 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:03.763280    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-zt6bv" podStartSLOduration=0.809875871 podStartE2EDuration="2.763262249s" totalImagesPullingTime="1.953386378s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:44:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:44:03.762430017 +0000 UTC m=+466.388099985" watchObservedRunningTime="2026-10-30 18:44:03.763262249 +0000 UTC m=+466.388932214"
Oct 30 18:44:04.706318 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:04.706151    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-zt6bv"]
Oct 30 18:44:05.757260 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:05.757217    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-zt6bv" podUID="98374c90-0ab3-45ef-8a11-115ff09da150" containerName="authorino" containerID="cri-o://05a4124a8ce6ee1a1b1c627b41dfc1e67689b34fa267ddf1b2ba6b391fc0a45e" gracePeriod=30
Oct 30 18:44:06.016032 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:06.015966    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-zt6bv"
Oct 30 18:44:06.102925 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:06.102890    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98374c90-0ab3-45ef-8a11-115ff09da150-kube-api-access-hzq9w\" (UniqueName: \"kubernetes.io/projected/98374c90-0ab3-45ef-8a11-115ff09da150-kube-api-access-hzq9w\") pod \"98374c90-0ab3-45ef-8a11-115ff09da150\" (UID: \"98374c90-0ab3-45ef-8a11-115ff09da150\") "
Oct 30 18:44:06.105116 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:06.105087    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98374c90-0ab3-45ef-8a11-115ff09da150-kube-api-access-hzq9w" pod "98374c90-0ab3-45ef-8a11-115ff09da150" (UID: "98374c90-0ab3-45ef-8a11-115ff09da150"). InnerVolumeSpecName "kube-api-access-hzq9w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:44:06.203488 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:06.203446    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzq9w\" (UniqueName: \"kubernetes.io/projected/98374c90-0ab3-45ef-8a11-115ff09da150-kube-api-access-hzq9w\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:44:06.763255 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:06.763214    4525 generic.go:356] "Generic (PLEG): container finished" podID="98374c90-0ab3-45ef-8a11-115ff09da150" containerID="05a4124a8ce6ee1a1b1c627b41dfc1e67689b34fa267ddf1b2ba6b391fc0a45e" exitCode=0
Oct 30 18:44:06.763255 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:06.763258    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-zt6bv"
Oct 30 18:44:06.763752 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:06.763305    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-zt6bv" event={"ID":"98374c90-0ab3-45ef-8a11-115ff09da150","Type":"ContainerDied","Data":"05a4124a8ce6ee1a1b1c627b41dfc1e67689b34fa267ddf1b2ba6b391fc0a45e"}
Oct 30 18:44:06.763752 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:06.763345    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-zt6bv" event={"ID":"98374c90-0ab3-45ef-8a11-115ff09da150","Type":"ContainerDied","Data":"5f0a7016207ac1e38d4ca8115ca4d632a9bc9eba3af34220640ed0cb1febea65"}
Oct 30 18:44:06.763752 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:06.763362    4525 scope.go:122] "RemoveContainer" containerID="05a4124a8ce6ee1a1b1c627b41dfc1e67689b34fa267ddf1b2ba6b391fc0a45e"
Oct 30 18:44:06.772852 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:06.772832    4525 scope.go:122] "RemoveContainer" containerID="05a4124a8ce6ee1a1b1c627b41dfc1e67689b34fa267ddf1b2ba6b391fc0a45e"
Oct 30 18:44:06.773190 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:44:06.773171    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05a4124a8ce6ee1a1b1c627b41dfc1e67689b34fa267ddf1b2ba6b391fc0a45e\": container with ID starting with 05a4124a8ce6ee1a1b1c627b41dfc1e67689b34fa267ddf1b2ba6b391fc0a45e not found: ID does not exist" containerID="05a4124a8ce6ee1a1b1c627b41dfc1e67689b34fa267ddf1b2ba6b391fc0a45e"
Oct 30 18:44:06.773257 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:06.773203    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05a4124a8ce6ee1a1b1c627b41dfc1e67689b34fa267ddf1b2ba6b391fc0a45e"} err="failed to get container status \"05a4124a8ce6ee1a1b1c627b41dfc1e67689b34fa267ddf1b2ba6b391fc0a45e\": rpc error: code = NotFound desc = could not find container \"05a4124a8ce6ee1a1b1c627b41dfc1e67689b34fa267ddf1b2ba6b391fc0a45e\": container with ID starting with 05a4124a8ce6ee1a1b1c627b41dfc1e67689b34fa267ddf1b2ba6b391fc0a45e not found: ID does not exist"
Oct 30 18:44:06.784583 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:06.784549    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-zt6bv"]
Oct 30 18:44:06.788109 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:06.788084    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-zt6bv"]
Oct 30 18:44:07.542273 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:07.542244    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98374c90-0ab3-45ef-8a11-115ff09da150" path="/var/lib/kubelet/pods/98374c90-0ab3-45ef-8a11-115ff09da150/volumes"
Oct 30 18:44:12.741424 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:12.741395    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-2j9vp"
Oct 30 18:44:14.104555 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:14.104519    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-qtvgz"]
Oct 30 18:44:14.105082 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:14.105067    4525 cpu_manager.go:414] "containerMap: removing container" podUID="98374c90-0ab3-45ef-8a11-115ff09da150" containerName="authorino"
Oct 30 18:44:14.105127 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:14.105085    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98374c90-0ab3-45ef-8a11-115ff09da150" containerName="authorino"
Oct 30 18:44:14.105166 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:14.105160    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="98374c90-0ab3-45ef-8a11-115ff09da150" containerName="authorino"
Oct 30 18:44:14.111907 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:14.111886    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-qtvgz"
Oct 30 18:44:14.121168 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:14.121142    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-qtvgz"]
Oct 30 18:44:14.135765 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:14.135511    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-stf9v\""
Oct 30 18:44:14.135765 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:14.135601    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Oct 30 18:44:14.181794 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:14.181763    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-lxnnq"]
Oct 30 18:44:14.185381 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:14.185356    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-lxnnq"
Oct 30 18:44:14.195141 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:14.195111    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-lxnnq"]
Oct 30 18:44:14.195665 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:14.195643    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\""
Oct 30 18:44:14.196528 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:14.196511    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-grndv\""
Oct 30 18:44:14.196722 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:14.196674    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Oct 30 18:44:14.269983 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:14.269952    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxrs6\" (UniqueName: \"kubernetes.io/projected/1e8fda7d-c348-4818-92bb-89fb11836b50-kube-api-access-gxrs6\") pod \"kserve-controller-manager-776cc66d47-qtvgz\" (UID: \"1e8fda7d-c348-4818-92bb-89fb11836b50\") " pod="opendatahub/kserve-controller-manager-776cc66d47-qtvgz"
Oct 30 18:44:14.269983 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:14.269988    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6a9d3618-5a4c-40aa-839e-002d218bd69b-cert\") pod \"odh-model-controller-769875db56-lxnnq\" (UID: \"6a9d3618-5a4c-40aa-839e-002d218bd69b\") " pod="opendatahub/odh-model-controller-769875db56-lxnnq"
Oct 30 18:44:14.270244 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:14.270012    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a9d3618-5a4c-40aa-839e-002d218bd69b-metrics-certs\") pod \"odh-model-controller-769875db56-lxnnq\" (UID: \"6a9d3618-5a4c-40aa-839e-002d218bd69b\") " pod="opendatahub/odh-model-controller-769875db56-lxnnq"
Oct 30 18:44:14.270244 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:14.270080    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1e8fda7d-c348-4818-92bb-89fb11836b50-cert\") pod \"kserve-controller-manager-776cc66d47-qtvgz\" (UID: \"1e8fda7d-c348-4818-92bb-89fb11836b50\") " pod="opendatahub/kserve-controller-manager-776cc66d47-qtvgz"
Oct 30 18:44:14.270244 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:14.270161    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pvkb\" (UniqueName: \"kubernetes.io/projected/6a9d3618-5a4c-40aa-839e-002d218bd69b-kube-api-access-8pvkb\") pod \"odh-model-controller-769875db56-lxnnq\" (UID: \"6a9d3618-5a4c-40aa-839e-002d218bd69b\") " pod="opendatahub/odh-model-controller-769875db56-lxnnq"
Oct 30 18:44:14.371173 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:14.371083    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxrs6\" (UniqueName: \"kubernetes.io/projected/1e8fda7d-c348-4818-92bb-89fb11836b50-kube-api-access-gxrs6\") pod \"kserve-controller-manager-776cc66d47-qtvgz\" (UID: \"1e8fda7d-c348-4818-92bb-89fb11836b50\") " pod="opendatahub/kserve-controller-manager-776cc66d47-qtvgz"
Oct 30 18:44:14.371173 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:14.371124    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6a9d3618-5a4c-40aa-839e-002d218bd69b-cert\") pod \"odh-model-controller-769875db56-lxnnq\" (UID: \"6a9d3618-5a4c-40aa-839e-002d218bd69b\") " pod="opendatahub/odh-model-controller-769875db56-lxnnq"
Oct 30 18:44:14.371173 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:14.371156    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a9d3618-5a4c-40aa-839e-002d218bd69b-metrics-certs\") pod \"odh-model-controller-769875db56-lxnnq\" (UID: \"6a9d3618-5a4c-40aa-839e-002d218bd69b\") " pod="opendatahub/odh-model-controller-769875db56-lxnnq"
Oct 30 18:44:14.371443 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:14.371300    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1e8fda7d-c348-4818-92bb-89fb11836b50-cert\") pod \"kserve-controller-manager-776cc66d47-qtvgz\" (UID: \"1e8fda7d-c348-4818-92bb-89fb11836b50\") " pod="opendatahub/kserve-controller-manager-776cc66d47-qtvgz"
Oct 30 18:44:14.371794 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:14.371754    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pvkb\" (UniqueName: \"kubernetes.io/projected/6a9d3618-5a4c-40aa-839e-002d218bd69b-kube-api-access-8pvkb\") pod \"odh-model-controller-769875db56-lxnnq\" (UID: \"6a9d3618-5a4c-40aa-839e-002d218bd69b\") " pod="opendatahub/odh-model-controller-769875db56-lxnnq"
Oct 30 18:44:14.374180 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:14.374156    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a9d3618-5a4c-40aa-839e-002d218bd69b-metrics-certs\") pod \"odh-model-controller-769875db56-lxnnq\" (UID: \"6a9d3618-5a4c-40aa-839e-002d218bd69b\") " pod="opendatahub/odh-model-controller-769875db56-lxnnq"
Oct 30 18:44:14.374304 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:14.374161    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6a9d3618-5a4c-40aa-839e-002d218bd69b-cert\") pod \"odh-model-controller-769875db56-lxnnq\" (UID: \"6a9d3618-5a4c-40aa-839e-002d218bd69b\") " pod="opendatahub/odh-model-controller-769875db56-lxnnq"
Oct 30 18:44:14.375181 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:14.375158    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1e8fda7d-c348-4818-92bb-89fb11836b50-cert\") pod \"kserve-controller-manager-776cc66d47-qtvgz\" (UID: \"1e8fda7d-c348-4818-92bb-89fb11836b50\") " pod="opendatahub/kserve-controller-manager-776cc66d47-qtvgz"
Oct 30 18:44:14.381547 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:14.381095    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pvkb\" (UniqueName: \"kubernetes.io/projected/6a9d3618-5a4c-40aa-839e-002d218bd69b-kube-api-access-8pvkb\") pod \"odh-model-controller-769875db56-lxnnq\" (UID: \"6a9d3618-5a4c-40aa-839e-002d218bd69b\") " pod="opendatahub/odh-model-controller-769875db56-lxnnq"
Oct 30 18:44:14.381547 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:14.381326    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxrs6\" (UniqueName: \"kubernetes.io/projected/1e8fda7d-c348-4818-92bb-89fb11836b50-kube-api-access-gxrs6\") pod \"kserve-controller-manager-776cc66d47-qtvgz\" (UID: \"1e8fda7d-c348-4818-92bb-89fb11836b50\") " pod="opendatahub/kserve-controller-manager-776cc66d47-qtvgz"
Oct 30 18:44:14.435074 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:14.435041    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-qtvgz"
Oct 30 18:44:14.511052 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:14.511023    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-lxnnq"
Oct 30 18:44:14.711713 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:14.711643    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-qtvgz"]
Oct 30 18:44:14.714056 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:44:14.714025    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e8fda7d_c348_4818_92bb_89fb11836b50.slice/crio-feddd5f81ffd1e91c135485c8c42c87fb55093dd2f28eeff76299690a5db3102 WatchSource:0}: Error finding container feddd5f81ffd1e91c135485c8c42c87fb55093dd2f28eeff76299690a5db3102: Status 404 returned error can't find the container with id feddd5f81ffd1e91c135485c8c42c87fb55093dd2f28eeff76299690a5db3102
Oct 30 18:44:14.719096 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:14.719074    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-2j9vp"]
Oct 30 18:44:14.719263 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:14.719245    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-2j9vp" podUID="de63be4b-4390-4692-a698-61a3c6bcc10f" containerName="limitador" containerID="cri-o://0393765a13453e5f16fbb7eee1c25fe5aa2e3674a95132450205da94dc12896d" gracePeriod=30
Oct 30 18:44:14.809154 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:14.809114    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-qtvgz" event={"ID":"1e8fda7d-c348-4818-92bb-89fb11836b50","Type":"ContainerStarted","Data":"feddd5f81ffd1e91c135485c8c42c87fb55093dd2f28eeff76299690a5db3102"}
Oct 30 18:44:14.821159 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:14.821135    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-lxnnq"]
Oct 30 18:44:14.822873 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:44:14.822846    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a9d3618_5a4c_40aa_839e_002d218bd69b.slice/crio-b681c6fe06205edd30e6f995cbadb78caae179f3c444e4b9d846d89f5d13ce8e WatchSource:0}: Error finding container b681c6fe06205edd30e6f995cbadb78caae179f3c444e4b9d846d89f5d13ce8e: Status 404 returned error can't find the container with id b681c6fe06205edd30e6f995cbadb78caae179f3c444e4b9d846d89f5d13ce8e
Oct 30 18:44:15.256232 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:15.256201    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-2j9vp"
Oct 30 18:44:15.395658 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:15.395608    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de63be4b-4390-4692-a698-61a3c6bcc10f-config-file\" (UniqueName: \"kubernetes.io/configmap/de63be4b-4390-4692-a698-61a3c6bcc10f-config-file\") pod \"de63be4b-4390-4692-a698-61a3c6bcc10f\" (UID: \"de63be4b-4390-4692-a698-61a3c6bcc10f\") "
Oct 30 18:44:15.395838 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:15.395731    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de63be4b-4390-4692-a698-61a3c6bcc10f-kube-api-access-x8nm9\" (UniqueName: \"kubernetes.io/projected/de63be4b-4390-4692-a698-61a3c6bcc10f-kube-api-access-x8nm9\") pod \"de63be4b-4390-4692-a698-61a3c6bcc10f\" (UID: \"de63be4b-4390-4692-a698-61a3c6bcc10f\") "
Oct 30 18:44:15.396278 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:15.396254    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de63be4b-4390-4692-a698-61a3c6bcc10f-config-file" pod "de63be4b-4390-4692-a698-61a3c6bcc10f" (UID: "de63be4b-4390-4692-a698-61a3c6bcc10f"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:44:15.398779 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:15.398750    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de63be4b-4390-4692-a698-61a3c6bcc10f-kube-api-access-x8nm9" pod "de63be4b-4390-4692-a698-61a3c6bcc10f" (UID: "de63be4b-4390-4692-a698-61a3c6bcc10f"). InnerVolumeSpecName "kube-api-access-x8nm9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:44:15.496964 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:15.496924    4525 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/de63be4b-4390-4692-a698-61a3c6bcc10f-config-file\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:44:15.496964 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:15.496967    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8nm9\" (UniqueName: \"kubernetes.io/projected/de63be4b-4390-4692-a698-61a3c6bcc10f-kube-api-access-x8nm9\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:44:15.818347 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:15.818166    4525 generic.go:356] "Generic (PLEG): container finished" podID="de63be4b-4390-4692-a698-61a3c6bcc10f" containerID="0393765a13453e5f16fbb7eee1c25fe5aa2e3674a95132450205da94dc12896d" exitCode=0
Oct 30 18:44:15.818520 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:15.818361    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-2j9vp"
Oct 30 18:44:15.819025 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:15.818807    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-2j9vp" event={"ID":"de63be4b-4390-4692-a698-61a3c6bcc10f","Type":"ContainerDied","Data":"0393765a13453e5f16fbb7eee1c25fe5aa2e3674a95132450205da94dc12896d"}
Oct 30 18:44:15.819025 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:15.818862    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-2j9vp" event={"ID":"de63be4b-4390-4692-a698-61a3c6bcc10f","Type":"ContainerDied","Data":"959f99dfad1de7295ee46767073c55967735391634ed764e636b9510f9b35873"}
Oct 30 18:44:15.819025 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:15.818890    4525 scope.go:122] "RemoveContainer" containerID="0393765a13453e5f16fbb7eee1c25fe5aa2e3674a95132450205da94dc12896d"
Oct 30 18:44:15.821107 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:15.821080    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-lxnnq" event={"ID":"6a9d3618-5a4c-40aa-839e-002d218bd69b","Type":"ContainerStarted","Data":"b681c6fe06205edd30e6f995cbadb78caae179f3c444e4b9d846d89f5d13ce8e"}
Oct 30 18:44:15.833999 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:15.833978    4525 scope.go:122] "RemoveContainer" containerID="0393765a13453e5f16fbb7eee1c25fe5aa2e3674a95132450205da94dc12896d"
Oct 30 18:44:15.834818 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:44:15.834788    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0393765a13453e5f16fbb7eee1c25fe5aa2e3674a95132450205da94dc12896d\": container with ID starting with 0393765a13453e5f16fbb7eee1c25fe5aa2e3674a95132450205da94dc12896d not found: ID does not exist" containerID="0393765a13453e5f16fbb7eee1c25fe5aa2e3674a95132450205da94dc12896d"
Oct 30 18:44:15.834923 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:15.834830    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0393765a13453e5f16fbb7eee1c25fe5aa2e3674a95132450205da94dc12896d"} err="failed to get container status \"0393765a13453e5f16fbb7eee1c25fe5aa2e3674a95132450205da94dc12896d\": rpc error: code = NotFound desc = could not find container \"0393765a13453e5f16fbb7eee1c25fe5aa2e3674a95132450205da94dc12896d\": container with ID starting with 0393765a13453e5f16fbb7eee1c25fe5aa2e3674a95132450205da94dc12896d not found: ID does not exist"
Oct 30 18:44:15.840001 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:15.839951    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-2j9vp"]
Oct 30 18:44:15.844079 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:15.844054    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-2j9vp"]
Oct 30 18:44:17.289313 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:17.289275    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-vw6gn"]
Oct 30 18:44:17.289790 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:17.289772    4525 cpu_manager.go:414] "containerMap: removing container" podUID="de63be4b-4390-4692-a698-61a3c6bcc10f" containerName="limitador"
Oct 30 18:44:17.289840 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:17.289792    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de63be4b-4390-4692-a698-61a3c6bcc10f" containerName="limitador"
Oct 30 18:44:17.289905 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:17.289893    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="de63be4b-4390-4692-a698-61a3c6bcc10f" containerName="limitador"
Oct 30 18:44:17.293230 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:17.293206    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-vw6gn"
Oct 30 18:44:17.300083 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:17.300018    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Oct 30 18:44:17.300224 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:17.300083    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 18:44:17.302080 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:17.300567    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 18:44:17.302080 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:17.300579    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-vkg7r\""
Oct 30 18:44:17.303951 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:17.303929    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-vw6gn"]
Oct 30 18:44:17.315553 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:17.315523    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szkzz\" (UniqueName: \"kubernetes.io/projected/834bfbdb-e855-429d-9699-60f3b7cf82c9-kube-api-access-szkzz\") pod \"postgres-7cdbf6dc44-vw6gn\" (UID: \"834bfbdb-e855-429d-9699-60f3b7cf82c9\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-vw6gn"
Oct 30 18:44:17.315720 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:17.315585    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/834bfbdb-e855-429d-9699-60f3b7cf82c9-data\") pod \"postgres-7cdbf6dc44-vw6gn\" (UID: \"834bfbdb-e855-429d-9699-60f3b7cf82c9\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-vw6gn"
Oct 30 18:44:17.400082 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:44:17.400044    4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d49l" containerName="extractor"
Oct 30 18:44:17.416669 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:17.416630    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szkzz\" (UniqueName: \"kubernetes.io/projected/834bfbdb-e855-429d-9699-60f3b7cf82c9-kube-api-access-szkzz\") pod \"postgres-7cdbf6dc44-vw6gn\" (UID: \"834bfbdb-e855-429d-9699-60f3b7cf82c9\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-vw6gn"
Oct 30 18:44:17.416861 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:17.416716    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/834bfbdb-e855-429d-9699-60f3b7cf82c9-data\") pod \"postgres-7cdbf6dc44-vw6gn\" (UID: \"834bfbdb-e855-429d-9699-60f3b7cf82c9\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-vw6gn"
Oct 30 18:44:17.417096 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:17.417078    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/834bfbdb-e855-429d-9699-60f3b7cf82c9-data\") pod \"postgres-7cdbf6dc44-vw6gn\" (UID: \"834bfbdb-e855-429d-9699-60f3b7cf82c9\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-vw6gn"
Oct 30 18:44:17.429447 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:17.429383    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szkzz\" (UniqueName: \"kubernetes.io/projected/834bfbdb-e855-429d-9699-60f3b7cf82c9-kube-api-access-szkzz\") pod \"postgres-7cdbf6dc44-vw6gn\" (UID: \"834bfbdb-e855-429d-9699-60f3b7cf82c9\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-vw6gn"
Oct 30 18:44:17.545500 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:17.545467    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de63be4b-4390-4692-a698-61a3c6bcc10f" path="/var/lib/kubelet/pods/de63be4b-4390-4692-a698-61a3c6bcc10f/volumes"
Oct 30 18:44:17.613793 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:17.613769    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-vkg7r\""
Oct 30 18:44:17.622128 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:17.622106    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-vw6gn"
Oct 30 18:44:17.837633 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:17.837595    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-qtvgz" event={"ID":"1e8fda7d-c348-4818-92bb-89fb11836b50","Type":"ContainerStarted","Data":"bc0abfbdeeb29bebbcf85ca4c2852536de2de45eb2153e7f91517da9e8f58a73"}
Oct 30 18:44:17.838339 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:17.838323    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-qtvgz"
Oct 30 18:44:17.840007 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:17.839974    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-lxnnq" event={"ID":"6a9d3618-5a4c-40aa-839e-002d218bd69b","Type":"ContainerStarted","Data":"10db9c35685200b0f555efd0de38594796b7ef772bc0ee419fa0a56471f27ff2"}
Oct 30 18:44:17.840127 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:17.840029    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-lxnnq"
Oct 30 18:44:17.855498 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:17.855451    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-qtvgz" podStartSLOduration=1.5012310100000001 podStartE2EDuration="3.855437045s" totalImagesPullingTime="2.354206035s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:44:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:44:17.854428864 +0000 UTC m=+480.480098827" watchObservedRunningTime="2026-10-30 18:44:17.855437045 +0000 UTC m=+480.481107008"
Oct 30 18:44:17.873663 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:17.873610    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-lxnnq" podStartSLOduration=1.051645248 podStartE2EDuration="3.873596947s" totalImagesPullingTime="2.821951699s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:44:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:44:17.872127966 +0000 UTC m=+480.497797932" watchObservedRunningTime="2026-10-30 18:44:17.873596947 +0000 UTC m=+480.499266910"
Oct 30 18:44:17.913829 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:17.913771    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-vw6gn"]
Oct 30 18:44:17.915136 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:44:17.915073    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod834bfbdb_e855_429d_9699_60f3b7cf82c9.slice/crio-65691b9442c9b31f90fb777fd710a29ea3e897f7000ee38be7bb0deccf9b8126 WatchSource:0}: Error finding container 65691b9442c9b31f90fb777fd710a29ea3e897f7000ee38be7bb0deccf9b8126: Status 404 returned error can't find the container with id 65691b9442c9b31f90fb777fd710a29ea3e897f7000ee38be7bb0deccf9b8126
Oct 30 18:44:18.845758 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:18.845716    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-vw6gn" event={"ID":"834bfbdb-e855-429d-9699-60f3b7cf82c9","Type":"ContainerStarted","Data":"65691b9442c9b31f90fb777fd710a29ea3e897f7000ee38be7bb0deccf9b8126"}
Oct 30 18:44:23.512846 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:23.512824    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Oct 30 18:44:23.873026 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:23.872992    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-vw6gn" event={"ID":"834bfbdb-e855-429d-9699-60f3b7cf82c9","Type":"ContainerStarted","Data":"b905c41c817b97bf15d69aed2f818daddee53aa5c7fadb622fa239ab9716968a"}
Oct 30 18:44:23.873180 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:23.873131    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-vw6gn"
Oct 30 18:44:23.890907 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:23.890863    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-vw6gn" podStartSLOduration=1.2979162 podStartE2EDuration="6.890848025s" totalImagesPullingTime="5.592931825s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:44:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:44:23.889018886 +0000 UTC m=+486.514688850" watchObservedRunningTime="2026-10-30 18:44:23.890848025 +0000 UTC m=+486.516517981"
Oct 30 18:44:25.533955 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:25.533916    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-757857f55b-rbcqw"]
Oct 30 18:44:25.538958 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:25.538935    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-757857f55b-rbcqw"
Oct 30 18:44:25.544284 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:25.544167    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\""
Oct 30 18:44:25.544284 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:25.544169    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:44:25.544284 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:25.544203    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Oct 30 18:44:25.544284 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:25.544261    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-xx2d6\""
Oct 30 18:44:25.544284 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:25.544263    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\""
Oct 30 18:44:25.544604 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:25.544434    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\""
Oct 30 18:44:25.547107 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:25.547081    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-757857f55b-rbcqw"]
Oct 30 18:44:25.588107 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:25.588071    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1fda7a86-a0aa-4fb3-be52-2e3ee2ffdd3a-cert\") pod \"jobset-controller-manager-757857f55b-rbcqw\" (UID: \"1fda7a86-a0aa-4fb3-be52-2e3ee2ffdd3a\") " pod="openshift-jobset-operator/jobset-controller-manager-757857f55b-rbcqw"
Oct 30 18:44:25.588261 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:25.588195    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fda7a86-a0aa-4fb3-be52-2e3ee2ffdd3a-metrics-certs\") pod \"jobset-controller-manager-757857f55b-rbcqw\" (UID: \"1fda7a86-a0aa-4fb3-be52-2e3ee2ffdd3a\") " pod="openshift-jobset-operator/jobset-controller-manager-757857f55b-rbcqw"
Oct 30 18:44:25.588303 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:25.588257    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1fda7a86-a0aa-4fb3-be52-2e3ee2ffdd3a-manager-config\") pod \"jobset-controller-manager-757857f55b-rbcqw\" (UID: \"1fda7a86-a0aa-4fb3-be52-2e3ee2ffdd3a\") " pod="openshift-jobset-operator/jobset-controller-manager-757857f55b-rbcqw"
Oct 30 18:44:25.588340 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:25.588322    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-299cz\" (UniqueName: \"kubernetes.io/projected/1fda7a86-a0aa-4fb3-be52-2e3ee2ffdd3a-kube-api-access-299cz\") pod \"jobset-controller-manager-757857f55b-rbcqw\" (UID: \"1fda7a86-a0aa-4fb3-be52-2e3ee2ffdd3a\") " pod="openshift-jobset-operator/jobset-controller-manager-757857f55b-rbcqw"
Oct 30 18:44:25.689356 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:25.689313    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fda7a86-a0aa-4fb3-be52-2e3ee2ffdd3a-metrics-certs\") pod \"jobset-controller-manager-757857f55b-rbcqw\" (UID: \"1fda7a86-a0aa-4fb3-be52-2e3ee2ffdd3a\") " pod="openshift-jobset-operator/jobset-controller-manager-757857f55b-rbcqw"
Oct 30 18:44:25.689356 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:25.689354    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1fda7a86-a0aa-4fb3-be52-2e3ee2ffdd3a-manager-config\") pod \"jobset-controller-manager-757857f55b-rbcqw\" (UID: \"1fda7a86-a0aa-4fb3-be52-2e3ee2ffdd3a\") " pod="openshift-jobset-operator/jobset-controller-manager-757857f55b-rbcqw"
Oct 30 18:44:25.689584 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:25.689396    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-299cz\" (UniqueName: \"kubernetes.io/projected/1fda7a86-a0aa-4fb3-be52-2e3ee2ffdd3a-kube-api-access-299cz\") pod \"jobset-controller-manager-757857f55b-rbcqw\" (UID: \"1fda7a86-a0aa-4fb3-be52-2e3ee2ffdd3a\") " pod="openshift-jobset-operator/jobset-controller-manager-757857f55b-rbcqw"
Oct 30 18:44:25.689645 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:25.689594    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1fda7a86-a0aa-4fb3-be52-2e3ee2ffdd3a-cert\") pod \"jobset-controller-manager-757857f55b-rbcqw\" (UID: \"1fda7a86-a0aa-4fb3-be52-2e3ee2ffdd3a\") " pod="openshift-jobset-operator/jobset-controller-manager-757857f55b-rbcqw"
Oct 30 18:44:25.690168 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:25.690137    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1fda7a86-a0aa-4fb3-be52-2e3ee2ffdd3a-manager-config\") pod \"jobset-controller-manager-757857f55b-rbcqw\" (UID: \"1fda7a86-a0aa-4fb3-be52-2e3ee2ffdd3a\") " pod="openshift-jobset-operator/jobset-controller-manager-757857f55b-rbcqw"
Oct 30 18:44:25.692127 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:25.692101    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1fda7a86-a0aa-4fb3-be52-2e3ee2ffdd3a-cert\") pod \"jobset-controller-manager-757857f55b-rbcqw\" (UID: \"1fda7a86-a0aa-4fb3-be52-2e3ee2ffdd3a\") " pod="openshift-jobset-operator/jobset-controller-manager-757857f55b-rbcqw"
Oct 30 18:44:25.692343 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:25.692319    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fda7a86-a0aa-4fb3-be52-2e3ee2ffdd3a-metrics-certs\") pod \"jobset-controller-manager-757857f55b-rbcqw\" (UID: \"1fda7a86-a0aa-4fb3-be52-2e3ee2ffdd3a\") " pod="openshift-jobset-operator/jobset-controller-manager-757857f55b-rbcqw"
Oct 30 18:44:25.699598 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:25.699567    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-299cz\" (UniqueName: \"kubernetes.io/projected/1fda7a86-a0aa-4fb3-be52-2e3ee2ffdd3a-kube-api-access-299cz\") pod \"jobset-controller-manager-757857f55b-rbcqw\" (UID: \"1fda7a86-a0aa-4fb3-be52-2e3ee2ffdd3a\") " pod="openshift-jobset-operator/jobset-controller-manager-757857f55b-rbcqw"
Oct 30 18:44:25.852292 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:25.852195    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-757857f55b-rbcqw"
Oct 30 18:44:26.127084 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:26.127063    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-757857f55b-rbcqw"]
Oct 30 18:44:26.128953 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:44:26.128908    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fda7a86_a0aa_4fb3_be52_2e3ee2ffdd3a.slice/crio-a203f044d50f9e01cc68d8af5f5cb68f07d4ff752c4971a80c5a535d3ca97864 WatchSource:0}: Error finding container a203f044d50f9e01cc68d8af5f5cb68f07d4ff752c4971a80c5a535d3ca97864: Status 404 returned error can't find the container with id a203f044d50f9e01cc68d8af5f5cb68f07d4ff752c4971a80c5a535d3ca97864
Oct 30 18:44:26.889984 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:26.889949    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-757857f55b-rbcqw" event={"ID":"1fda7a86-a0aa-4fb3-be52-2e3ee2ffdd3a","Type":"ContainerStarted","Data":"a203f044d50f9e01cc68d8af5f5cb68f07d4ff752c4971a80c5a535d3ca97864"}
Oct 30 18:44:28.848634 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:28.848561    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-lxnnq"
Oct 30 18:44:29.909199 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:29.909162    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-vw6gn"
Oct 30 18:44:29.909671 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:29.909592    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-757857f55b-rbcqw" event={"ID":"1fda7a86-a0aa-4fb3-be52-2e3ee2ffdd3a","Type":"ContainerStarted","Data":"5716e7feb40f432c0bad1117ca09864f23747a868182a8c52be2e3d3c222da54"}
Oct 30 18:44:29.909671 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:29.909641    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-757857f55b-rbcqw"
Oct 30 18:44:29.945406 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:29.945355    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-757857f55b-rbcqw" podStartSLOduration=2.026722869 podStartE2EDuration="4.945341663s" totalImagesPullingTime="2.918618794s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:44:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:44:29.944758509 +0000 UTC m=+492.570428476" watchObservedRunningTime="2026-10-30 18:44:29.945341663 +0000 UTC m=+492.571011626"
Oct 30 18:44:40.920383 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:40.920354    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-757857f55b-rbcqw"
Oct 30 18:44:47.396400 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:44:47.396355    4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d49l" containerName="extractor"
Oct 30 18:44:49.856776 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:44:49.856748    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-qtvgz"
Oct 30 18:45:17.424350 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:45:17.424296    4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d49l" containerName="extractor"
Oct 30 18:45:24.174215 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:45:24.174180    4525 generic.go:356] "Generic (PLEG): container finished" podID="85ce0a7f-b7fb-4909-bfd6-affca6da6d49" containerID="6cf9b5c37b7c71638afc520527f10f9d8a5b3bc1d32b025d3f3507f70f0c8b44" exitCode=0
Oct 30 18:45:24.174789 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:45:24.174248    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-sgtll" event={"ID":"85ce0a7f-b7fb-4909-bfd6-affca6da6d49","Type":"ContainerDied","Data":"6cf9b5c37b7c71638afc520527f10f9d8a5b3bc1d32b025d3f3507f70f0c8b44"}
Oct 30 18:45:24.174789 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:45:24.174657    4525 scope.go:122] "RemoveContainer" containerID="6cf9b5c37b7c71638afc520527f10f9d8a5b3bc1d32b025d3f3507f70f0c8b44"
Oct 30 18:45:24.175829 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:45:24.175802    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:45:24.175894 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:45:24.175872    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:45:24.175936 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:45:24.175905    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:45:24.806758 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:45:24.806716    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6dd76875b5-l9ndj"]
Oct 30 18:45:24.809932 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:45:24.809910    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6dd76875b5-l9ndj"
Oct 30 18:45:24.816160 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:45:24.816129    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6dd76875b5-l9ndj"]
Oct 30 18:45:24.819744 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:45:24.819712    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Oct 30 18:45:24.819744 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:45:24.819736    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-99gx9\""
Oct 30 18:45:24.854568 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:45:24.854537    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6dd76875b5-l9ndj"]
Oct 30 18:45:24.854815 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:45:24.854788    4525 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-xbsrn tls-cert], unattached volumes=[], failed to process volumes=[kube-api-access-xbsrn tls-cert]: context canceled" pod="kuadrant-system/authorino-6dd76875b5-l9ndj" podUID="da7f1d33-a53d-409d-ab2f-d7a5834afb47"
Oct 30 18:45:24.906663 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:45:24.906266    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/da7f1d33-a53d-409d-ab2f-d7a5834afb47-tls-cert\") pod \"authorino-6dd76875b5-l9ndj\" (UID: \"da7f1d33-a53d-409d-ab2f-d7a5834afb47\") " pod="kuadrant-system/authorino-6dd76875b5-l9ndj"
Oct 30 18:45:24.906663 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:45:24.906310    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbsrn\" (UniqueName: \"kubernetes.io/projected/da7f1d33-a53d-409d-ab2f-d7a5834afb47-kube-api-access-xbsrn\") pod \"authorino-6dd76875b5-l9ndj\" (UID: \"da7f1d33-a53d-409d-ab2f-d7a5834afb47\") " pod="kuadrant-system/authorino-6dd76875b5-l9ndj"
Oct 30 18:45:25.001003 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:45:25.000968    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-sgtll"
Oct 30 18:45:25.007550 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:45:25.007518    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/da7f1d33-a53d-409d-ab2f-d7a5834afb47-tls-cert\") pod \"authorino-6dd76875b5-l9ndj\" (UID: \"da7f1d33-a53d-409d-ab2f-d7a5834afb47\") " pod="kuadrant-system/authorino-6dd76875b5-l9ndj"
Oct 30 18:45:25.007550 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:45:25.007556    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbsrn\" (UniqueName: \"kubernetes.io/projected/da7f1d33-a53d-409d-ab2f-d7a5834afb47-kube-api-access-xbsrn\") pod \"authorino-6dd76875b5-l9ndj\" (UID: \"da7f1d33-a53d-409d-ab2f-d7a5834afb47\") " pod="kuadrant-system/authorino-6dd76875b5-l9ndj"
Oct 30 18:45:25.010297 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:45:25.010275    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/da7f1d33-a53d-409d-ab2f-d7a5834afb47-tls-cert\") pod \"authorino-6dd76875b5-l9ndj\" (UID: \"da7f1d33-a53d-409d-ab2f-d7a5834afb47\") " pod="kuadrant-system/authorino-6dd76875b5-l9ndj"
Oct 30 18:45:25.020136 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:45:25.020113    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbsrn\" (UniqueName: \"kubernetes.io/projected/da7f1d33-a53d-409d-ab2f-d7a5834afb47-kube-api-access-xbsrn\") pod \"authorino-6dd76875b5-l9ndj\" (UID: \"da7f1d33-a53d-409d-ab2f-d7a5834afb47\") " pod="kuadrant-system/authorino-6dd76875b5-l9ndj"
Oct 30 18:45:25.205823 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:45:25.205785    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-sgtll" event={"ID":"85ce0a7f-b7fb-4909-bfd6-affca6da6d49","Type":"ContainerStarted","Data":"3d3545a95e12f94856895bb6c21a8fa3396344e6a0d7065fdc68383c8671abe1"}
Oct 30 18:45:25.206289 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:45:25.205839    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6dd76875b5-l9ndj"
Oct 30 18:45:25.212819 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:45:25.212797    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6dd76875b5-l9ndj"
Oct 30 18:45:25.310933 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:45:25.310896    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da7f1d33-a53d-409d-ab2f-d7a5834afb47-tls-cert\" (UniqueName: \"kubernetes.io/secret/da7f1d33-a53d-409d-ab2f-d7a5834afb47-tls-cert\") pod \"da7f1d33-a53d-409d-ab2f-d7a5834afb47\" (UID: \"da7f1d33-a53d-409d-ab2f-d7a5834afb47\") "
Oct 30 18:45:25.311104 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:45:25.311038    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da7f1d33-a53d-409d-ab2f-d7a5834afb47-kube-api-access-xbsrn\" (UniqueName: \"kubernetes.io/projected/da7f1d33-a53d-409d-ab2f-d7a5834afb47-kube-api-access-xbsrn\") pod \"da7f1d33-a53d-409d-ab2f-d7a5834afb47\" (UID: \"da7f1d33-a53d-409d-ab2f-d7a5834afb47\") "
Oct 30 18:45:25.313263 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:45:25.313238    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da7f1d33-a53d-409d-ab2f-d7a5834afb47-tls-cert" pod "da7f1d33-a53d-409d-ab2f-d7a5834afb47" (UID: "da7f1d33-a53d-409d-ab2f-d7a5834afb47"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:45:25.313407 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:45:25.313270    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da7f1d33-a53d-409d-ab2f-d7a5834afb47-kube-api-access-xbsrn" pod "da7f1d33-a53d-409d-ab2f-d7a5834afb47" (UID: "da7f1d33-a53d-409d-ab2f-d7a5834afb47"). InnerVolumeSpecName "kube-api-access-xbsrn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:45:25.412485 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:45:25.412450    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbsrn\" (UniqueName: \"kubernetes.io/projected/da7f1d33-a53d-409d-ab2f-d7a5834afb47-kube-api-access-xbsrn\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:45:25.412485 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:45:25.412478    4525 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/da7f1d33-a53d-409d-ab2f-d7a5834afb47-tls-cert\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:45:25.691545 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:45:25.691078    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-sgtll"
Oct 30 18:45:25.691545 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:45:25.691200    4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-68cb9b4b95-sgtll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" start-of-body=
Oct 30 18:45:25.691545 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:45:25.691246    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-sgtll" podUID="85ce0a7f-b7fb-4909-bfd6-affca6da6d49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused"
Oct 30 18:45:26.211362 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:45:26.211326    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6dd76875b5-l9ndj"
Oct 30 18:45:26.242822 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:45:26.242790    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6dd76875b5-l9ndj"]
Oct 30 18:45:26.248714 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:45:26.248670    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6dd76875b5-l9ndj"]
Oct 30 18:45:26.690127 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:45:26.690087    4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-68cb9b4b95-sgtll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" start-of-body=
Oct 30 18:45:26.690428 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:45:26.690398    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-sgtll" podUID="85ce0a7f-b7fb-4909-bfd6-affca6da6d49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused"
Oct 30 18:45:27.543919 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:45:27.543886    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da7f1d33-a53d-409d-ab2f-d7a5834afb47" path="/var/lib/kubelet/pods/da7f1d33-a53d-409d-ab2f-d7a5834afb47/volumes"
Oct 30 18:45:27.689716 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:45:27.687065    4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-68cb9b4b95-sgtll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" start-of-body=
Oct 30 18:45:27.689716 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:45:27.687130    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-sgtll" podUID="85ce0a7f-b7fb-4909-bfd6-affca6da6d49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused"
Oct 30 18:45:28.701154 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:45:28.701117    4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-68cb9b4b95-sgtll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" start-of-body=
Oct 30 18:45:28.701777 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:45:28.701743    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-sgtll" podUID="85ce0a7f-b7fb-4909-bfd6-affca6da6d49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused"
Oct 30 18:45:29.690708 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:45:29.689051    4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-68cb9b4b95-sgtll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" start-of-body=
Oct 30 18:45:29.690708 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:45:29.689112    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-sgtll" podUID="85ce0a7f-b7fb-4909-bfd6-affca6da6d49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused"
Oct 30 18:45:30.690546 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:45:30.690507    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-sgtll"
Oct 30 18:45:30.691024 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:45:30.690812    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-sgtll"
Oct 30 18:45:30.691480 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:45:30.691464    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-sgtll"
Oct 30 18:45:47.396119 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:45:47.396085    4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d49l" containerName="extractor"
Oct 30 18:46:17.397250 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:46:17.397210    4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d49l" containerName="extractor"
Oct 30 18:46:38.192032 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:38.192000    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-zgw99"]
Oct 30 18:46:38.195657 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:38.195637    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zgw99"
Oct 30 18:46:38.204792 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:38.204771    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-nz76v\""
Oct 30 18:46:38.204792 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:38.204775    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 30 18:46:38.210894 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:38.210872    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-zgw99"]
Oct 30 18:46:38.233024 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:38.232990    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72bab614-8d2f-4640-9d16-b54f05759572-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-zgw99\" (UID: \"72bab614-8d2f-4640-9d16-b54f05759572\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zgw99"
Oct 30 18:46:38.233159 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:38.233076    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72bab614-8d2f-4640-9d16-b54f05759572-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-zgw99\" (UID: \"72bab614-8d2f-4640-9d16-b54f05759572\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zgw99"
Oct 30 18:46:38.233159 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:38.233117    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72bab614-8d2f-4640-9d16-b54f05759572-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-zgw99\" (UID: \"72bab614-8d2f-4640-9d16-b54f05759572\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zgw99"
Oct 30 18:46:38.233296 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:38.233203    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fljh5\" (UniqueName: \"kubernetes.io/projected/72bab614-8d2f-4640-9d16-b54f05759572-kube-api-access-fljh5\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-zgw99\" (UID: \"72bab614-8d2f-4640-9d16-b54f05759572\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zgw99"
Oct 30 18:46:38.233296 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:38.233262    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72bab614-8d2f-4640-9d16-b54f05759572-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-zgw99\" (UID: \"72bab614-8d2f-4640-9d16-b54f05759572\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zgw99"
Oct 30 18:46:38.233399 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:38.233296    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72bab614-8d2f-4640-9d16-b54f05759572-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-zgw99\" (UID: \"72bab614-8d2f-4640-9d16-b54f05759572\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zgw99"
Oct 30 18:46:38.304396 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:38.304372    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 30 18:46:38.304540 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:38.304500    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Oct 30 18:46:38.333844 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:38.333808    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72bab614-8d2f-4640-9d16-b54f05759572-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-zgw99\" (UID: \"72bab614-8d2f-4640-9d16-b54f05759572\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zgw99"
Oct 30 18:46:38.333844 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:38.333844    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72bab614-8d2f-4640-9d16-b54f05759572-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-zgw99\" (UID: \"72bab614-8d2f-4640-9d16-b54f05759572\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zgw99"
Oct 30 18:46:38.334088 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:38.333866    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72bab614-8d2f-4640-9d16-b54f05759572-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-zgw99\" (UID: \"72bab614-8d2f-4640-9d16-b54f05759572\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zgw99"
Oct 30 18:46:38.334088 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:38.333892    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fljh5\" (UniqueName: \"kubernetes.io/projected/72bab614-8d2f-4640-9d16-b54f05759572-kube-api-access-fljh5\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-zgw99\" (UID: \"72bab614-8d2f-4640-9d16-b54f05759572\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zgw99"
Oct 30 18:46:38.334088 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:38.334017    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72bab614-8d2f-4640-9d16-b54f05759572-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-zgw99\" (UID: \"72bab614-8d2f-4640-9d16-b54f05759572\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zgw99"
Oct 30 18:46:38.334088 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:38.334067    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72bab614-8d2f-4640-9d16-b54f05759572-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-zgw99\" (UID: \"72bab614-8d2f-4640-9d16-b54f05759572\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zgw99"
Oct 30 18:46:38.334290 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:38.334271    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72bab614-8d2f-4640-9d16-b54f05759572-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-zgw99\" (UID: \"72bab614-8d2f-4640-9d16-b54f05759572\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zgw99"
Oct 30 18:46:38.334333 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:38.334304    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72bab614-8d2f-4640-9d16-b54f05759572-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-zgw99\" (UID: \"72bab614-8d2f-4640-9d16-b54f05759572\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zgw99"
Oct 30 18:46:38.334439 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:38.334423    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72bab614-8d2f-4640-9d16-b54f05759572-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-zgw99\" (UID: \"72bab614-8d2f-4640-9d16-b54f05759572\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zgw99"
Oct 30 18:46:38.336297 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:38.336278    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72bab614-8d2f-4640-9d16-b54f05759572-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-zgw99\" (UID: \"72bab614-8d2f-4640-9d16-b54f05759572\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zgw99"
Oct 30 18:46:38.336521 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:38.336504    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72bab614-8d2f-4640-9d16-b54f05759572-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-zgw99\" (UID: \"72bab614-8d2f-4640-9d16-b54f05759572\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zgw99"
Oct 30 18:46:38.345105 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:38.345085    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fljh5\" (UniqueName: \"kubernetes.io/projected/72bab614-8d2f-4640-9d16-b54f05759572-kube-api-access-fljh5\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-zgw99\" (UID: \"72bab614-8d2f-4640-9d16-b54f05759572\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zgw99"
Oct 30 18:46:38.508948 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:38.508871    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zgw99"
Oct 30 18:46:38.785143 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:38.785111    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-zgw99"]
Oct 30 18:46:38.786826 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:46:38.786795    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72bab614_8d2f_4640_9d16_b54f05759572.slice/crio-ea609cfd8cc4af4f03aa423e2c3c92db69c2accb022e4d62d339a7a90d6964da WatchSource:0}: Error finding container ea609cfd8cc4af4f03aa423e2c3c92db69c2accb022e4d62d339a7a90d6964da: Status 404 returned error can't find the container with id ea609cfd8cc4af4f03aa423e2c3c92db69c2accb022e4d62d339a7a90d6964da
Oct 30 18:46:39.572474 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:39.572434    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zgw99" event={"ID":"72bab614-8d2f-4640-9d16-b54f05759572","Type":"ContainerStarted","Data":"ea609cfd8cc4af4f03aa423e2c3c92db69c2accb022e4d62d339a7a90d6964da"}
Oct 30 18:46:40.579170 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:40.579135    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zgw99" event={"ID":"72bab614-8d2f-4640-9d16-b54f05759572","Type":"ContainerStarted","Data":"d143ea9dba1b38b8b5d4324e1a18d9f899064db2a5cf845337d6158550b320d9"}
Oct 30 18:46:40.600824 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:40.600772    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zgw99" podStartSLOduration=1.329980785 podStartE2EDuration="2.600757893s" totalImagesPullingTime="1.270777108s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:46:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:46:40.599224404 +0000 UTC m=+623.224894368" watchObservedRunningTime="2026-10-30 18:46:40.600757893 +0000 UTC m=+623.226427856"
Oct 30 18:46:41.509944 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:41.509912    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zgw99"
Oct 30 18:46:42.933725 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:42.933658    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh28d"]
Oct 30 18:46:42.937413 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:42.937386    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh28d"
Oct 30 18:46:42.948457 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:42.948431    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Oct 30 18:46:42.949973 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:42.949932    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh28d"]
Oct 30 18:46:43.075517 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:43.075485    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e396973d-d477-43a8-a781-4d3b9d2044ad-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh28d\" (UID: \"e396973d-d477-43a8-a781-4d3b9d2044ad\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh28d"
Oct 30 18:46:43.075517 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:43.075522    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e396973d-d477-43a8-a781-4d3b9d2044ad-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh28d\" (UID: \"e396973d-d477-43a8-a781-4d3b9d2044ad\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh28d"
Oct 30 18:46:43.075783 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:43.075542    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2l79h\" (UniqueName: \"kubernetes.io/projected/e396973d-d477-43a8-a781-4d3b9d2044ad-kube-api-access-2l79h\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh28d\" (UID: \"e396973d-d477-43a8-a781-4d3b9d2044ad\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh28d"
Oct 30 18:46:43.075783 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:43.075624    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e396973d-d477-43a8-a781-4d3b9d2044ad-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh28d\" (UID: \"e396973d-d477-43a8-a781-4d3b9d2044ad\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh28d"
Oct 30 18:46:43.075783 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:43.075661    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e396973d-d477-43a8-a781-4d3b9d2044ad-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh28d\" (UID: \"e396973d-d477-43a8-a781-4d3b9d2044ad\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh28d"
Oct 30 18:46:43.075783 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:43.075776    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e396973d-d477-43a8-a781-4d3b9d2044ad-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh28d\" (UID: \"e396973d-d477-43a8-a781-4d3b9d2044ad\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh28d"
Oct 30 18:46:43.176356 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:43.176322    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e396973d-d477-43a8-a781-4d3b9d2044ad-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh28d\" (UID: \"e396973d-d477-43a8-a781-4d3b9d2044ad\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh28d"
Oct 30 18:46:43.176356 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:43.176360    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e396973d-d477-43a8-a781-4d3b9d2044ad-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh28d\" (UID: \"e396973d-d477-43a8-a781-4d3b9d2044ad\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh28d"
Oct 30 18:46:43.176626 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:43.176379    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l79h\" (UniqueName: \"kubernetes.io/projected/e396973d-d477-43a8-a781-4d3b9d2044ad-kube-api-access-2l79h\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh28d\" (UID: \"e396973d-d477-43a8-a781-4d3b9d2044ad\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh28d"
Oct 30 18:46:43.176626 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:43.176407    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e396973d-d477-43a8-a781-4d3b9d2044ad-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh28d\" (UID: \"e396973d-d477-43a8-a781-4d3b9d2044ad\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh28d"
Oct 30 18:46:43.176626 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:43.176427    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e396973d-d477-43a8-a781-4d3b9d2044ad-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh28d\" (UID: \"e396973d-d477-43a8-a781-4d3b9d2044ad\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh28d"
Oct 30 18:46:43.176626 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:43.176494    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e396973d-d477-43a8-a781-4d3b9d2044ad-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh28d\" (UID: \"e396973d-d477-43a8-a781-4d3b9d2044ad\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh28d"
Oct 30 18:46:43.176888 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:43.176837    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e396973d-d477-43a8-a781-4d3b9d2044ad-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh28d\" (UID: \"e396973d-d477-43a8-a781-4d3b9d2044ad\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh28d"
Oct 30 18:46:43.177005 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:43.176984    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e396973d-d477-43a8-a781-4d3b9d2044ad-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh28d\" (UID: \"e396973d-d477-43a8-a781-4d3b9d2044ad\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh28d"
Oct 30 18:46:43.177046 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:43.177001    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e396973d-d477-43a8-a781-4d3b9d2044ad-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh28d\" (UID: \"e396973d-d477-43a8-a781-4d3b9d2044ad\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh28d"
Oct 30 18:46:43.179057 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:43.179024    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e396973d-d477-43a8-a781-4d3b9d2044ad-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh28d\" (UID: \"e396973d-d477-43a8-a781-4d3b9d2044ad\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh28d"
Oct 30 18:46:43.179203 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:43.179181    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e396973d-d477-43a8-a781-4d3b9d2044ad-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh28d\" (UID: \"e396973d-d477-43a8-a781-4d3b9d2044ad\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh28d"
Oct 30 18:46:43.185034 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:43.184978    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2l79h\" (UniqueName: \"kubernetes.io/projected/e396973d-d477-43a8-a781-4d3b9d2044ad-kube-api-access-2l79h\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh28d\" (UID: \"e396973d-d477-43a8-a781-4d3b9d2044ad\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh28d"
Oct 30 18:46:43.252770 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:43.252737    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh28d"
Oct 30 18:46:43.533359 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:43.533328    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh28d"]
Oct 30 18:46:43.534752 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:46:43.534723    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode396973d_d477_43a8_a781_4d3b9d2044ad.slice/crio-8b22595e43cbd3c32276228f1ccda2dcc243f298ff2987bfc87182e596eb57b6 WatchSource:0}: Error finding container 8b22595e43cbd3c32276228f1ccda2dcc243f298ff2987bfc87182e596eb57b6: Status 404 returned error can't find the container with id 8b22595e43cbd3c32276228f1ccda2dcc243f298ff2987bfc87182e596eb57b6
Oct 30 18:46:43.596285 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:43.596252    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh28d" event={"ID":"e396973d-d477-43a8-a781-4d3b9d2044ad","Type":"ContainerStarted","Data":"8b22595e43cbd3c32276228f1ccda2dcc243f298ff2987bfc87182e596eb57b6"}
Oct 30 18:46:44.605265 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:44.605228    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh28d" event={"ID":"e396973d-d477-43a8-a781-4d3b9d2044ad","Type":"ContainerStarted","Data":"35ed34e8de4f8af7ec0e66b37856d603f32059615dd3675609b2ba2371a748e3"}
Oct 30 18:46:44.627105 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:44.627027    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh28d" podStartSLOduration=2.321240009 podStartE2EDuration="2.627008609s" totalImagesPullingTime="305.7686ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:46:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:46:44.625561211 +0000 UTC m=+627.251231174" watchObservedRunningTime="2026-10-30 18:46:44.627008609 +0000 UTC m=+627.252678573"
Oct 30 18:46:45.055103 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:45.055017    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nmscq"]
Oct 30 18:46:45.060121 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:45.060094    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nmscq"
Oct 30 18:46:45.067279 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:45.067240    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Oct 30 18:46:45.068625 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:45.068585    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nmscq"]
Oct 30 18:46:45.195416 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:45.195385    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/787150b8-33c2-41a6-b54c-e35fd0d27bef-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nmscq\" (UID: \"787150b8-33c2-41a6-b54c-e35fd0d27bef\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nmscq"
Oct 30 18:46:45.195625 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:45.195453    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/787150b8-33c2-41a6-b54c-e35fd0d27bef-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nmscq\" (UID: \"787150b8-33c2-41a6-b54c-e35fd0d27bef\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nmscq"
Oct 30 18:46:45.195625 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:45.195563    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/787150b8-33c2-41a6-b54c-e35fd0d27bef-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nmscq\" (UID: \"787150b8-33c2-41a6-b54c-e35fd0d27bef\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nmscq"
Oct 30 18:46:45.195625 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:45.195596    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/787150b8-33c2-41a6-b54c-e35fd0d27bef-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nmscq\" (UID: \"787150b8-33c2-41a6-b54c-e35fd0d27bef\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nmscq"
Oct 30 18:46:45.195832 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:45.195633    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/787150b8-33c2-41a6-b54c-e35fd0d27bef-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nmscq\" (UID: \"787150b8-33c2-41a6-b54c-e35fd0d27bef\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nmscq"
Oct 30 18:46:45.195832 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:45.195753    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6dxr\" (UniqueName: \"kubernetes.io/projected/787150b8-33c2-41a6-b54c-e35fd0d27bef-kube-api-access-p6dxr\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nmscq\" (UID: \"787150b8-33c2-41a6-b54c-e35fd0d27bef\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nmscq"
Oct 30 18:46:45.253404 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:45.253371    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh28d"
Oct 30 18:46:45.296849 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:45.296819    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/787150b8-33c2-41a6-b54c-e35fd0d27bef-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nmscq\" (UID: \"787150b8-33c2-41a6-b54c-e35fd0d27bef\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nmscq"
Oct 30 18:46:45.297056 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:45.296880    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/787150b8-33c2-41a6-b54c-e35fd0d27bef-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nmscq\" (UID: \"787150b8-33c2-41a6-b54c-e35fd0d27bef\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nmscq"
Oct 30 18:46:45.297056 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:45.296909    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/787150b8-33c2-41a6-b54c-e35fd0d27bef-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nmscq\" (UID: \"787150b8-33c2-41a6-b54c-e35fd0d27bef\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nmscq"
Oct 30 18:46:45.297056 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:45.297040    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/787150b8-33c2-41a6-b54c-e35fd0d27bef-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nmscq\" (UID: \"787150b8-33c2-41a6-b54c-e35fd0d27bef\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nmscq"
Oct 30 18:46:45.297211 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:45.297119    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/787150b8-33c2-41a6-b54c-e35fd0d27bef-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nmscq\" (UID: \"787150b8-33c2-41a6-b54c-e35fd0d27bef\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nmscq"
Oct 30 18:46:45.297211 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:45.297148    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6dxr\" (UniqueName: \"kubernetes.io/projected/787150b8-33c2-41a6-b54c-e35fd0d27bef-kube-api-access-p6dxr\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nmscq\" (UID: \"787150b8-33c2-41a6-b54c-e35fd0d27bef\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nmscq"
Oct 30 18:46:45.297364 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:45.297343    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/787150b8-33c2-41a6-b54c-e35fd0d27bef-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nmscq\" (UID: \"787150b8-33c2-41a6-b54c-e35fd0d27bef\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nmscq"
Oct 30 18:46:45.297427 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:45.297365    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/787150b8-33c2-41a6-b54c-e35fd0d27bef-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nmscq\" (UID: \"787150b8-33c2-41a6-b54c-e35fd0d27bef\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nmscq"
Oct 30 18:46:45.297473 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:45.297424    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/787150b8-33c2-41a6-b54c-e35fd0d27bef-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nmscq\" (UID: \"787150b8-33c2-41a6-b54c-e35fd0d27bef\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nmscq"
Oct 30 18:46:45.299493 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:45.299471    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/787150b8-33c2-41a6-b54c-e35fd0d27bef-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nmscq\" (UID: \"787150b8-33c2-41a6-b54c-e35fd0d27bef\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nmscq"
Oct 30 18:46:45.299798 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:45.299778    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/787150b8-33c2-41a6-b54c-e35fd0d27bef-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nmscq\" (UID: \"787150b8-33c2-41a6-b54c-e35fd0d27bef\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nmscq"
Oct 30 18:46:45.306182 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:45.306131    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6dxr\" (UniqueName: \"kubernetes.io/projected/787150b8-33c2-41a6-b54c-e35fd0d27bef-kube-api-access-p6dxr\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nmscq\" (UID: \"787150b8-33c2-41a6-b54c-e35fd0d27bef\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nmscq"
Oct 30 18:46:45.375834 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:45.375803    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nmscq"
Oct 30 18:46:45.604612 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:45.604588    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb"]
Oct 30 18:46:45.609460 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:45.609434    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb"
Oct 30 18:46:45.614023 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:45.613993    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Oct 30 18:46:45.622030 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:45.622005    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb"]
Oct 30 18:46:45.654913 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:45.654737    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nmscq"]
Oct 30 18:46:45.657620 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:46:45.657580    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod787150b8_33c2_41a6_b54c_e35fd0d27bef.slice/crio-2fda2595c97bb342f189f899184eb408511e36d6fadb044724bad5c3b1f282de WatchSource:0}: Error finding container 2fda2595c97bb342f189f899184eb408511e36d6fadb044724bad5c3b1f282de: Status 404 returned error can't find the container with id 2fda2595c97bb342f189f899184eb408511e36d6fadb044724bad5c3b1f282de
Oct 30 18:46:45.701409 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:45.701371    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12ebc8b9-c794-47b5-b87b-a63c4bd993e6-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb\" (UID: \"12ebc8b9-c794-47b5-b87b-a63c4bd993e6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb"
Oct 30 18:46:45.701596 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:45.701434    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12ebc8b9-c794-47b5-b87b-a63c4bd993e6-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb\" (UID: \"12ebc8b9-c794-47b5-b87b-a63c4bd993e6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb"
Oct 30 18:46:45.701596 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:45.701488    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bnhw\" (UniqueName: \"kubernetes.io/projected/12ebc8b9-c794-47b5-b87b-a63c4bd993e6-kube-api-access-6bnhw\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb\" (UID: \"12ebc8b9-c794-47b5-b87b-a63c4bd993e6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb"
Oct 30 18:46:45.701728 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:45.701624    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12ebc8b9-c794-47b5-b87b-a63c4bd993e6-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb\" (UID: \"12ebc8b9-c794-47b5-b87b-a63c4bd993e6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb"
Oct 30 18:46:45.701728 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:45.701705    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12ebc8b9-c794-47b5-b87b-a63c4bd993e6-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb\" (UID: \"12ebc8b9-c794-47b5-b87b-a63c4bd993e6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb"
Oct 30 18:46:45.701848 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:45.701833    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12ebc8b9-c794-47b5-b87b-a63c4bd993e6-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb\" (UID: \"12ebc8b9-c794-47b5-b87b-a63c4bd993e6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb"
Oct 30 18:46:45.802725 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:45.802663    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12ebc8b9-c794-47b5-b87b-a63c4bd993e6-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb\" (UID: \"12ebc8b9-c794-47b5-b87b-a63c4bd993e6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb"
Oct 30 18:46:45.802879 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:45.802752    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12ebc8b9-c794-47b5-b87b-a63c4bd993e6-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb\" (UID: \"12ebc8b9-c794-47b5-b87b-a63c4bd993e6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb"
Oct 30 18:46:45.802879 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:45.802777    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bnhw\" (UniqueName: \"kubernetes.io/projected/12ebc8b9-c794-47b5-b87b-a63c4bd993e6-kube-api-access-6bnhw\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb\" (UID: \"12ebc8b9-c794-47b5-b87b-a63c4bd993e6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb"
Oct 30 18:46:45.802879 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:45.802824    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12ebc8b9-c794-47b5-b87b-a63c4bd993e6-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb\" (UID: \"12ebc8b9-c794-47b5-b87b-a63c4bd993e6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb"
Oct 30 18:46:45.802879 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:45.802853    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12ebc8b9-c794-47b5-b87b-a63c4bd993e6-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb\" (UID: \"12ebc8b9-c794-47b5-b87b-a63c4bd993e6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb"
Oct 30 18:46:45.803106 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:45.802891    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12ebc8b9-c794-47b5-b87b-a63c4bd993e6-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb\" (UID: \"12ebc8b9-c794-47b5-b87b-a63c4bd993e6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb"
Oct 30 18:46:45.803163 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:45.803129    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12ebc8b9-c794-47b5-b87b-a63c4bd993e6-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb\" (UID: \"12ebc8b9-c794-47b5-b87b-a63c4bd993e6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb"
Oct 30 18:46:45.803220 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:45.803202    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12ebc8b9-c794-47b5-b87b-a63c4bd993e6-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb\" (UID: \"12ebc8b9-c794-47b5-b87b-a63c4bd993e6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb"
Oct 30 18:46:45.803267 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:45.803232    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12ebc8b9-c794-47b5-b87b-a63c4bd993e6-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb\" (UID: \"12ebc8b9-c794-47b5-b87b-a63c4bd993e6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb"
Oct 30 18:46:45.805192 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:45.805170    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12ebc8b9-c794-47b5-b87b-a63c4bd993e6-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb\" (UID: \"12ebc8b9-c794-47b5-b87b-a63c4bd993e6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb"
Oct 30 18:46:45.805426 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:45.805409    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12ebc8b9-c794-47b5-b87b-a63c4bd993e6-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb\" (UID: \"12ebc8b9-c794-47b5-b87b-a63c4bd993e6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb"
Oct 30 18:46:45.810928 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:45.810903    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bnhw\" (UniqueName: \"kubernetes.io/projected/12ebc8b9-c794-47b5-b87b-a63c4bd993e6-kube-api-access-6bnhw\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb\" (UID: \"12ebc8b9-c794-47b5-b87b-a63c4bd993e6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb"
Oct 30 18:46:45.926155 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:45.926058    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb"
Oct 30 18:46:46.204546 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:46.204519    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb"]
Oct 30 18:46:46.206409 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:46:46.206381    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod12ebc8b9_c794_47b5_b87b_a63c4bd993e6.slice/crio-71da8eb12234304b76a6a4f22fc7be591870a0edf44482a9f1d4bfb5feaf0e61 WatchSource:0}: Error finding container 71da8eb12234304b76a6a4f22fc7be591870a0edf44482a9f1d4bfb5feaf0e61: Status 404 returned error can't find the container with id 71da8eb12234304b76a6a4f22fc7be591870a0edf44482a9f1d4bfb5feaf0e61
Oct 30 18:46:46.617996 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:46.617961    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nmscq" event={"ID":"787150b8-33c2-41a6-b54c-e35fd0d27bef","Type":"ContainerStarted","Data":"ac65846e93ac86bf5be4e085b8e49b3d9ec705852d7fe2733a47256210602e27"}
Oct 30 18:46:46.618429 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:46.618017    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nmscq" event={"ID":"787150b8-33c2-41a6-b54c-e35fd0d27bef","Type":"ContainerStarted","Data":"2fda2595c97bb342f189f899184eb408511e36d6fadb044724bad5c3b1f282de"}
Oct 30 18:46:46.619714 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:46.619664    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb" event={"ID":"12ebc8b9-c794-47b5-b87b-a63c4bd993e6","Type":"ContainerStarted","Data":"20c1c6097e84d9ebcdb20795d8bfb8d78991b36536af953cb82dc5bef5b9a022"}
Oct 30 18:46:46.619833 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:46.619723    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb" event={"ID":"12ebc8b9-c794-47b5-b87b-a63c4bd993e6","Type":"ContainerStarted","Data":"71da8eb12234304b76a6a4f22fc7be591870a0edf44482a9f1d4bfb5feaf0e61"}
Oct 30 18:46:46.640415 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:46.640363    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nmscq" podStartSLOduration=1.327666107 podStartE2EDuration="1.640349018s" totalImagesPullingTime="312.682911ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:46:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:46:46.637916832 +0000 UTC m=+629.263586797" watchObservedRunningTime="2026-10-30 18:46:46.640349018 +0000 UTC m=+629.266018982"
Oct 30 18:46:46.657663 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:46.657608    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb" podStartSLOduration=1.344527359 podStartE2EDuration="1.657594552s" totalImagesPullingTime="313.067193ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:46:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:46:46.655409266 +0000 UTC m=+629.281079231" watchObservedRunningTime="2026-10-30 18:46:46.657594552 +0000 UTC m=+629.283264517"
Oct 30 18:46:46.927237 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:46.927197    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb"
Oct 30 18:46:47.376697 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:47.376659    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nmscq"
Oct 30 18:46:47.396442 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:46:47.396401    4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d49l" containerName="extractor"
Oct 30 18:46:48.509372 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:48.509331    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zgw99"
Oct 30 18:46:48.528557 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:48.528528    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zgw99"
Oct 30 18:46:48.643121 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:48.643086    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zgw99"
Oct 30 18:46:53.253235 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:53.253191    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh28d"
Oct 30 18:46:53.265856 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:53.265832    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh28d"
Oct 30 18:46:53.666118 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:53.666092    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh28d"
Oct 30 18:46:55.376760 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:55.376721    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nmscq"
Oct 30 18:46:55.389552 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:55.389528    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nmscq"
Oct 30 18:46:55.676902 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:55.676827    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nmscq"
Oct 30 18:46:55.927356 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:55.927261    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb"
Oct 30 18:46:55.939728 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:55.939672    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb"
Oct 30 18:46:56.682311 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:56.682285    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb"
Oct 30 18:46:56.818996 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:56.818956    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-dhclt"]
Oct 30 18:46:56.822625 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:56.822605    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-dhclt"
Oct 30 18:46:56.839520 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:56.839499    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-dhclt"]
Oct 30 18:46:56.900945 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:56.900912    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldkb5\" (UniqueName: \"kubernetes.io/projected/0e0636e7-6f61-4843-844d-fb9d7bb527da-kube-api-access-ldkb5\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-dhclt\" (UID: \"0e0636e7-6f61-4843-844d-fb9d7bb527da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-dhclt"
Oct 30 18:46:56.901107 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:56.900954    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e0636e7-6f61-4843-844d-fb9d7bb527da-istio-podinfo\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-dhclt\" (UID: \"0e0636e7-6f61-4843-844d-fb9d7bb527da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-dhclt"
Oct 30 18:46:56.901107 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:56.901039    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e0636e7-6f61-4843-844d-fb9d7bb527da-istio-envoy\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-dhclt\" (UID: \"0e0636e7-6f61-4843-844d-fb9d7bb527da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-dhclt"
Oct 30 18:46:56.901107 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:56.901081    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e0636e7-6f61-4843-844d-fb9d7bb527da-credential-socket\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-dhclt\" (UID: \"0e0636e7-6f61-4843-844d-fb9d7bb527da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-dhclt"
Oct 30 18:46:56.901107 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:56.901100    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e0636e7-6f61-4843-844d-fb9d7bb527da-istio-data\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-dhclt\" (UID: \"0e0636e7-6f61-4843-844d-fb9d7bb527da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-dhclt"
Oct 30 18:46:56.901249 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:56.901182    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e0636e7-6f61-4843-844d-fb9d7bb527da-workload-certs\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-dhclt\" (UID: \"0e0636e7-6f61-4843-844d-fb9d7bb527da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-dhclt"
Oct 30 18:46:56.901249 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:56.901208    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e0636e7-6f61-4843-844d-fb9d7bb527da-workload-socket\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-dhclt\" (UID: \"0e0636e7-6f61-4843-844d-fb9d7bb527da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-dhclt"
Oct 30 18:46:56.901313 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:56.901249    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e0636e7-6f61-4843-844d-fb9d7bb527da-istio-token\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-dhclt\" (UID: \"0e0636e7-6f61-4843-844d-fb9d7bb527da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-dhclt"
Oct 30 18:46:56.901313 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:56.901281    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e0636e7-6f61-4843-844d-fb9d7bb527da-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-dhclt\" (UID: \"0e0636e7-6f61-4843-844d-fb9d7bb527da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-dhclt"
Oct 30 18:46:57.001905 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:57.001838    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e0636e7-6f61-4843-844d-fb9d7bb527da-credential-socket\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-dhclt\" (UID: \"0e0636e7-6f61-4843-844d-fb9d7bb527da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-dhclt"
Oct 30 18:46:57.001905 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:57.001868    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e0636e7-6f61-4843-844d-fb9d7bb527da-istio-data\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-dhclt\" (UID: \"0e0636e7-6f61-4843-844d-fb9d7bb527da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-dhclt"
Oct 30 18:46:57.001905 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:57.001889    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e0636e7-6f61-4843-844d-fb9d7bb527da-workload-certs\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-dhclt\" (UID: \"0e0636e7-6f61-4843-844d-fb9d7bb527da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-dhclt"
Oct 30 18:46:57.001905 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:57.001906    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e0636e7-6f61-4843-844d-fb9d7bb527da-workload-socket\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-dhclt\" (UID: \"0e0636e7-6f61-4843-844d-fb9d7bb527da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-dhclt"
Oct 30 18:46:57.002200 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:57.001953    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e0636e7-6f61-4843-844d-fb9d7bb527da-istio-token\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-dhclt\" (UID: \"0e0636e7-6f61-4843-844d-fb9d7bb527da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-dhclt"
Oct 30 18:46:57.002200 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:57.001970    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e0636e7-6f61-4843-844d-fb9d7bb527da-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-dhclt\" (UID: \"0e0636e7-6f61-4843-844d-fb9d7bb527da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-dhclt"
Oct 30 18:46:57.002200 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:57.002005    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldkb5\" (UniqueName: \"kubernetes.io/projected/0e0636e7-6f61-4843-844d-fb9d7bb527da-kube-api-access-ldkb5\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-dhclt\" (UID: \"0e0636e7-6f61-4843-844d-fb9d7bb527da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-dhclt"
Oct 30 18:46:57.002200 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:57.002046    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e0636e7-6f61-4843-844d-fb9d7bb527da-istio-podinfo\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-dhclt\" (UID: \"0e0636e7-6f61-4843-844d-fb9d7bb527da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-dhclt"
Oct 30 18:46:57.002397 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:57.002305    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e0636e7-6f61-4843-844d-fb9d7bb527da-workload-certs\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-dhclt\" (UID: \"0e0636e7-6f61-4843-844d-fb9d7bb527da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-dhclt"
Oct 30 18:46:57.002397 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:57.002313    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e0636e7-6f61-4843-844d-fb9d7bb527da-istio-envoy\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-dhclt\" (UID: \"0e0636e7-6f61-4843-844d-fb9d7bb527da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-dhclt"
Oct 30 18:46:57.002397 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:57.002369    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e0636e7-6f61-4843-844d-fb9d7bb527da-istio-data\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-dhclt\" (UID: \"0e0636e7-6f61-4843-844d-fb9d7bb527da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-dhclt"
Oct 30 18:46:57.002549 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:57.002376    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e0636e7-6f61-4843-844d-fb9d7bb527da-credential-socket\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-dhclt\" (UID: \"0e0636e7-6f61-4843-844d-fb9d7bb527da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-dhclt"
Oct 30 18:46:57.002720 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:57.002665    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e0636e7-6f61-4843-844d-fb9d7bb527da-workload-socket\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-dhclt\" (UID: \"0e0636e7-6f61-4843-844d-fb9d7bb527da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-dhclt"
Oct 30 18:46:57.002805 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:57.002710    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e0636e7-6f61-4843-844d-fb9d7bb527da-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-dhclt\" (UID: \"0e0636e7-6f61-4843-844d-fb9d7bb527da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-dhclt"
Oct 30 18:46:57.004528 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:57.004503    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e0636e7-6f61-4843-844d-fb9d7bb527da-istio-envoy\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-dhclt\" (UID: \"0e0636e7-6f61-4843-844d-fb9d7bb527da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-dhclt"
Oct 30 18:46:57.004746 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:57.004730    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e0636e7-6f61-4843-844d-fb9d7bb527da-istio-podinfo\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-dhclt\" (UID: \"0e0636e7-6f61-4843-844d-fb9d7bb527da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-dhclt"
Oct 30 18:46:57.010542 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:57.010519    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldkb5\" (UniqueName: \"kubernetes.io/projected/0e0636e7-6f61-4843-844d-fb9d7bb527da-kube-api-access-ldkb5\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-dhclt\" (UID: \"0e0636e7-6f61-4843-844d-fb9d7bb527da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-dhclt"
Oct 30 18:46:57.010629 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:57.010564    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e0636e7-6f61-4843-844d-fb9d7bb527da-istio-token\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-dhclt\" (UID: \"0e0636e7-6f61-4843-844d-fb9d7bb527da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-dhclt"
Oct 30 18:46:57.136669 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:57.136623    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-dhclt"
Oct 30 18:46:57.422517 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:57.422494    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-dhclt"]
Oct 30 18:46:57.424036 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:46:57.424007    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e0636e7_6f61_4843_844d_fb9d7bb527da.slice/crio-2d010365bfca6d534b7527883645987a3349dc0870c581235140c207df2266dd WatchSource:0}: Error finding container 2d010365bfca6d534b7527883645987a3349dc0870c581235140c207df2266dd: Status 404 returned error can't find the container with id 2d010365bfca6d534b7527883645987a3349dc0870c581235140c207df2266dd
Oct 30 18:46:57.426228 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:57.426190    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:46:57.426309 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:57.426282    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:46:57.426347 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:57.426331    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:46:57.680053 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:57.679956    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-dhclt" event={"ID":"0e0636e7-6f61-4843-844d-fb9d7bb527da","Type":"ContainerStarted","Data":"744818fd672a41f9671e8a439aa5cfef93a5f459b0ed208ee19d37e71d823797"}
Oct 30 18:46:57.680053 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:57.680022    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-dhclt" event={"ID":"0e0636e7-6f61-4843-844d-fb9d7bb527da","Type":"ContainerStarted","Data":"2d010365bfca6d534b7527883645987a3349dc0870c581235140c207df2266dd"}
Oct 30 18:46:57.706125 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:57.706061    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-dhclt" podStartSLOduration=1.706048497 podStartE2EDuration="1.706048497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:46:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:46:57.703831391 +0000 UTC m=+640.329501366" watchObservedRunningTime="2026-10-30 18:46:57.706048497 +0000 UTC m=+640.331718461"
Oct 30 18:46:58.137167 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:58.137119    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-dhclt"
Oct 30 18:46:58.138774 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:58.138749    4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-68cb9b4b95-dhclt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Oct 30 18:46:58.138913 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:58.138797    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-dhclt" podUID="0e0636e7-6f61-4843-844d-fb9d7bb527da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Oct 30 18:46:58.827122 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:58.827091    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58966b5867-lsc7q"]
Oct 30 18:46:58.830605 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:58.830585    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58966b5867-lsc7q"
Oct 30 18:46:58.837039 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:58.836993    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Oct 30 18:46:58.838388 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:58.838072    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-99gx9\""
Oct 30 18:46:58.841419 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:58.841393    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58966b5867-lsc7q"]
Oct 30 18:46:58.923028 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:58.922987    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/30bd1d87-7569-4d72-9898-13990914ca4a-tls-cert\") pod \"authorino-58966b5867-lsc7q\" (UID: \"30bd1d87-7569-4d72-9898-13990914ca4a\") " pod="kuadrant-system/authorino-58966b5867-lsc7q"
Oct 30 18:46:58.923187 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:58.923036    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qp4l\" (UniqueName: \"kubernetes.io/projected/30bd1d87-7569-4d72-9898-13990914ca4a-kube-api-access-9qp4l\") pod \"authorino-58966b5867-lsc7q\" (UID: \"30bd1d87-7569-4d72-9898-13990914ca4a\") " pod="kuadrant-system/authorino-58966b5867-lsc7q"
Oct 30 18:46:59.023929 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:59.023895    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/30bd1d87-7569-4d72-9898-13990914ca4a-tls-cert\") pod \"authorino-58966b5867-lsc7q\" (UID: \"30bd1d87-7569-4d72-9898-13990914ca4a\") " pod="kuadrant-system/authorino-58966b5867-lsc7q"
Oct 30 18:46:59.024092 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:59.023934    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qp4l\" (UniqueName: \"kubernetes.io/projected/30bd1d87-7569-4d72-9898-13990914ca4a-kube-api-access-9qp4l\") pod \"authorino-58966b5867-lsc7q\" (UID: \"30bd1d87-7569-4d72-9898-13990914ca4a\") " pod="kuadrant-system/authorino-58966b5867-lsc7q"
Oct 30 18:46:59.026574 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:59.026556    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/30bd1d87-7569-4d72-9898-13990914ca4a-tls-cert\") pod \"authorino-58966b5867-lsc7q\" (UID: \"30bd1d87-7569-4d72-9898-13990914ca4a\") " pod="kuadrant-system/authorino-58966b5867-lsc7q"
Oct 30 18:46:59.032524 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:59.032500    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qp4l\" (UniqueName: \"kubernetes.io/projected/30bd1d87-7569-4d72-9898-13990914ca4a-kube-api-access-9qp4l\") pod \"authorino-58966b5867-lsc7q\" (UID: \"30bd1d87-7569-4d72-9898-13990914ca4a\") " pod="kuadrant-system/authorino-58966b5867-lsc7q"
Oct 30 18:46:59.137978 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:59.137948    4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-68cb9b4b95-dhclt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Oct 30 18:46:59.138129 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:59.137999    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-dhclt" podUID="0e0636e7-6f61-4843-844d-fb9d7bb527da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Oct 30 18:46:59.144122 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:59.144103    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58966b5867-lsc7q"
Oct 30 18:46:59.437101 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:59.436753    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58966b5867-lsc7q"]
Oct 30 18:46:59.439322 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:46:59.439290    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod30bd1d87_7569_4d72_9898_13990914ca4a.slice/crio-7baaef741cb8e5d3843d71a5ddac8bc79f9434ba42c3cedc325a6b9043b291ba WatchSource:0}: Error finding container 7baaef741cb8e5d3843d71a5ddac8bc79f9434ba42c3cedc325a6b9043b291ba: Status 404 returned error can't find the container with id 7baaef741cb8e5d3843d71a5ddac8bc79f9434ba42c3cedc325a6b9043b291ba
Oct 30 18:46:59.711218 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:46:59.711137    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58966b5867-lsc7q" event={"ID":"30bd1d87-7569-4d72-9898-13990914ca4a","Type":"ContainerStarted","Data":"7baaef741cb8e5d3843d71a5ddac8bc79f9434ba42c3cedc325a6b9043b291ba"}
Oct 30 18:47:00.137266 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:00.137234    4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-68cb9b4b95-dhclt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Oct 30 18:47:00.137640 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:00.137287    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-dhclt" podUID="0e0636e7-6f61-4843-844d-fb9d7bb527da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Oct 30 18:47:00.720501 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:00.720417    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58966b5867-lsc7q" event={"ID":"30bd1d87-7569-4d72-9898-13990914ca4a","Type":"ContainerStarted","Data":"083a71a7d6c3d254bc047c8e4550c29d127e172602ab16b8ddcbd2a57f497965"}
Oct 30 18:47:00.740322 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:00.740266    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-58966b5867-lsc7q" podStartSLOduration=1.721185279 podStartE2EDuration="2.740248167s" totalImagesPullingTime="1.019062888s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:46:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:00.737089595 +0000 UTC m=+643.362759565" watchObservedRunningTime="2026-10-30 18:47:00.740248167 +0000 UTC m=+643.365918131"
Oct 30 18:47:01.137264 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:01.137225    4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-68cb9b4b95-dhclt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Oct 30 18:47:01.137891 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:01.137859    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-dhclt" podUID="0e0636e7-6f61-4843-844d-fb9d7bb527da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Oct 30 18:47:02.137088 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:02.137042    4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-68cb9b4b95-dhclt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Oct 30 18:47:02.137266 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:02.137118    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-dhclt" podUID="0e0636e7-6f61-4843-844d-fb9d7bb527da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Oct 30 18:47:03.138113 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:03.138069    4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-68cb9b4b95-dhclt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Oct 30 18:47:03.138721 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:03.138665    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-dhclt" podUID="0e0636e7-6f61-4843-844d-fb9d7bb527da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Oct 30 18:47:04.137035 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:04.136989    4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-68cb9b4b95-dhclt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Oct 30 18:47:04.137242 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:04.137048    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-dhclt" podUID="0e0636e7-6f61-4843-844d-fb9d7bb527da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Oct 30 18:47:05.824182 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:05.824140    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-dhclt"
Oct 30 18:47:05.825122 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:05.825096    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-dhclt"
Oct 30 18:47:05.825982 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:05.825962    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-dhclt"
Oct 30 18:47:17.397096 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:47:17.397051    4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d49l" containerName="extractor"
Oct 30 18:47:38.015573 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:38.015489    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-6156d5d6-55fcbb8767-4w5g4"]
Oct 30 18:47:38.019115 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:38.019098    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6156d5d6-55fcbb8767-4w5g4"
Oct 30 18:47:38.025550 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:38.025526    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-6156d5d6-dockercfg-2pkrx\""
Oct 30 18:47:38.035816 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:38.035789    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-6156d5d6-55fcbb8767-4w5g4"]
Oct 30 18:47:38.109153 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:38.109113    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-6156d5d6\""
Oct 30 18:47:38.138291 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:38.138250    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cec3994-12ce-4931-9d31-1b620974dd99-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-6156d5d6-55fcbb8767-4w5g4\" (UID: \"1cec3994-12ce-4931-9d31-1b620974dd99\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-6156d5d6-55fcbb8767-4w5g4"
Oct 30 18:47:38.138569 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:38.138545    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvmck\" (UniqueName: \"kubernetes.io/projected/1cec3994-12ce-4931-9d31-1b620974dd99-kube-api-access-rvmck\") pod \"payload-pre-processing-e2e-migrate-6156d5d6-55fcbb8767-4w5g4\" (UID: \"1cec3994-12ce-4931-9d31-1b620974dd99\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-6156d5d6-55fcbb8767-4w5g4"
Oct 30 18:47:38.240156 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:38.240122    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvmck\" (UniqueName: \"kubernetes.io/projected/1cec3994-12ce-4931-9d31-1b620974dd99-kube-api-access-rvmck\") pod \"payload-pre-processing-e2e-migrate-6156d5d6-55fcbb8767-4w5g4\" (UID: \"1cec3994-12ce-4931-9d31-1b620974dd99\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-6156d5d6-55fcbb8767-4w5g4"
Oct 30 18:47:38.240363 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:38.240243    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cec3994-12ce-4931-9d31-1b620974dd99-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-6156d5d6-55fcbb8767-4w5g4\" (UID: \"1cec3994-12ce-4931-9d31-1b620974dd99\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-6156d5d6-55fcbb8767-4w5g4"
Oct 30 18:47:38.240975 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:38.240950    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cec3994-12ce-4931-9d31-1b620974dd99-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-6156d5d6-55fcbb8767-4w5g4\" (UID: \"1cec3994-12ce-4931-9d31-1b620974dd99\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-6156d5d6-55fcbb8767-4w5g4"
Oct 30 18:47:38.256131 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:38.256100    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvmck\" (UniqueName: \"kubernetes.io/projected/1cec3994-12ce-4931-9d31-1b620974dd99-kube-api-access-rvmck\") pod \"payload-pre-processing-e2e-migrate-6156d5d6-55fcbb8767-4w5g4\" (UID: \"1cec3994-12ce-4931-9d31-1b620974dd99\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-6156d5d6-55fcbb8767-4w5g4"
Oct 30 18:47:38.331987 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:38.331907    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6156d5d6-55fcbb8767-4w5g4"
Oct 30 18:47:38.610637 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:38.610612    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-6156d5d6-55fcbb8767-4w5g4"]
Oct 30 18:47:38.612021 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:47:38.611988    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cec3994_12ce_4931_9d31_1b620974dd99.slice/crio-97eb60d2dac0fd337eda1fe3f0bd8bd0a619f8ab9232a202d3ced7e55266cd42 WatchSource:0}: Error finding container 97eb60d2dac0fd337eda1fe3f0bd8bd0a619f8ab9232a202d3ced7e55266cd42: Status 404 returned error can't find the container with id 97eb60d2dac0fd337eda1fe3f0bd8bd0a619f8ab9232a202d3ced7e55266cd42
Oct 30 18:47:38.614511 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:38.614492    4525 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 18:47:38.925335 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:38.925256    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6156d5d6-55fcbb8767-4w5g4" event={"ID":"1cec3994-12ce-4931-9d31-1b620974dd99","Type":"ContainerStarted","Data":"97eb60d2dac0fd337eda1fe3f0bd8bd0a619f8ab9232a202d3ced7e55266cd42"}
Oct 30 18:47:39.013627 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:39.013591    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-226ecbcb-69b85788bc-8xszf"]
Oct 30 18:47:39.018387 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:39.018370    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-226ecbcb-69b85788bc-8xszf"
Oct 30 18:47:39.029065 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:39.029043    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-226ecbcb\""
Oct 30 18:47:39.030179 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:39.030156    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-226ecbcb-69b85788bc-8xszf"]
Oct 30 18:47:39.039985 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:39.039961    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-226ecbcb-dockercfg-88fzc\""
Oct 30 18:47:39.049218 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:39.049190    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-226ecbcb-5d69cc78dd-rjzsb"]
Oct 30 18:47:39.052791 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:39.052772    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-226ecbcb-5d69cc78dd-rjzsb"
Oct 30 18:47:39.070593 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:39.070570    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-226ecbcb-5d69cc78dd-rjzsb"]
Oct 30 18:47:39.149605 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:39.149562    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6bb3a0f-8779-436c-b404-a26849ff7943-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-226ecbcb-69b85788bc-8xszf\" (UID: \"f6bb3a0f-8779-436c-b404-a26849ff7943\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-226ecbcb-69b85788bc-8xszf"
Oct 30 18:47:39.149824 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:39.149643    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcxh2\" (UniqueName: \"kubernetes.io/projected/f6bb3a0f-8779-436c-b404-a26849ff7943-kube-api-access-zcxh2\") pod \"payload-pre-processing-e2e-ait-226ecbcb-69b85788bc-8xszf\" (UID: \"f6bb3a0f-8779-436c-b404-a26849ff7943\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-226ecbcb-69b85788bc-8xszf"
Oct 30 18:47:39.149824 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:39.149751    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p99gf\" (UniqueName: \"kubernetes.io/projected/52ca0496-3987-481c-8efa-28c13e432616-kube-api-access-p99gf\") pod \"payload-processing-e2e-ait-226ecbcb-5d69cc78dd-rjzsb\" (UID: \"52ca0496-3987-481c-8efa-28c13e432616\") " pod="openshift-ingress/payload-processing-e2e-ait-226ecbcb-5d69cc78dd-rjzsb"
Oct 30 18:47:39.149824 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:39.149801    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52ca0496-3987-481c-8efa-28c13e432616-plugins-config-volume\") pod \"payload-processing-e2e-ait-226ecbcb-5d69cc78dd-rjzsb\" (UID: \"52ca0496-3987-481c-8efa-28c13e432616\") " pod="openshift-ingress/payload-processing-e2e-ait-226ecbcb-5d69cc78dd-rjzsb"
Oct 30 18:47:39.250931 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:39.250856    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52ca0496-3987-481c-8efa-28c13e432616-plugins-config-volume\") pod \"payload-processing-e2e-ait-226ecbcb-5d69cc78dd-rjzsb\" (UID: \"52ca0496-3987-481c-8efa-28c13e432616\") " pod="openshift-ingress/payload-processing-e2e-ait-226ecbcb-5d69cc78dd-rjzsb"
Oct 30 18:47:39.251097 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:39.250942    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6bb3a0f-8779-436c-b404-a26849ff7943-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-226ecbcb-69b85788bc-8xszf\" (UID: \"f6bb3a0f-8779-436c-b404-a26849ff7943\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-226ecbcb-69b85788bc-8xszf"
Oct 30 18:47:39.251097 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:39.250998    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcxh2\" (UniqueName: \"kubernetes.io/projected/f6bb3a0f-8779-436c-b404-a26849ff7943-kube-api-access-zcxh2\") pod \"payload-pre-processing-e2e-ait-226ecbcb-69b85788bc-8xszf\" (UID: \"f6bb3a0f-8779-436c-b404-a26849ff7943\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-226ecbcb-69b85788bc-8xszf"
Oct 30 18:47:39.251097 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:39.251051    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p99gf\" (UniqueName: \"kubernetes.io/projected/52ca0496-3987-481c-8efa-28c13e432616-kube-api-access-p99gf\") pod \"payload-processing-e2e-ait-226ecbcb-5d69cc78dd-rjzsb\" (UID: \"52ca0496-3987-481c-8efa-28c13e432616\") " pod="openshift-ingress/payload-processing-e2e-ait-226ecbcb-5d69cc78dd-rjzsb"
Oct 30 18:47:39.251747 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:39.251719    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6bb3a0f-8779-436c-b404-a26849ff7943-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-226ecbcb-69b85788bc-8xszf\" (UID: \"f6bb3a0f-8779-436c-b404-a26849ff7943\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-226ecbcb-69b85788bc-8xszf"
Oct 30 18:47:39.251880 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:39.251764    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52ca0496-3987-481c-8efa-28c13e432616-plugins-config-volume\") pod \"payload-processing-e2e-ait-226ecbcb-5d69cc78dd-rjzsb\" (UID: \"52ca0496-3987-481c-8efa-28c13e432616\") " pod="openshift-ingress/payload-processing-e2e-ait-226ecbcb-5d69cc78dd-rjzsb"
Oct 30 18:47:39.265791 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:39.265756    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p99gf\" (UniqueName: \"kubernetes.io/projected/52ca0496-3987-481c-8efa-28c13e432616-kube-api-access-p99gf\") pod \"payload-processing-e2e-ait-226ecbcb-5d69cc78dd-rjzsb\" (UID: \"52ca0496-3987-481c-8efa-28c13e432616\") " pod="openshift-ingress/payload-processing-e2e-ait-226ecbcb-5d69cc78dd-rjzsb"
Oct 30 18:47:39.266845 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:39.266820    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcxh2\" (UniqueName: \"kubernetes.io/projected/f6bb3a0f-8779-436c-b404-a26849ff7943-kube-api-access-zcxh2\") pod \"payload-pre-processing-e2e-ait-226ecbcb-69b85788bc-8xszf\" (UID: \"f6bb3a0f-8779-436c-b404-a26849ff7943\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-226ecbcb-69b85788bc-8xszf"
Oct 30 18:47:39.333539 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:39.333504    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-226ecbcb-69b85788bc-8xszf"
Oct 30 18:47:39.364843 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:39.364813    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-226ecbcb-5d69cc78dd-rjzsb"
Oct 30 18:47:39.583589 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:39.583504    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e4242163-d74b5477f-fwc2h"]
Oct 30 18:47:39.587226 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:39.587201    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e4242163-d74b5477f-fwc2h"
Oct 30 18:47:39.598545 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:39.598521    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e4242163-6fddb59d56-jvvtf"]
Oct 30 18:47:39.602267 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:39.602242    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e4242163-6fddb59d56-jvvtf"
Oct 30 18:47:39.606195 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:39.606170    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-e4242163\""
Oct 30 18:47:39.606330 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:39.606230    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-e4242163-dockercfg-sv645\""
Oct 30 18:47:39.616242 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:39.616218    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e4242163-d74b5477f-fwc2h"]
Oct 30 18:47:39.619425 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:39.619402    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e4242163-6fddb59d56-jvvtf"]
Oct 30 18:47:39.660650 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:39.660607    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-226ecbcb-69b85788bc-8xszf"]
Oct 30 18:47:39.664564 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:47:39.664539    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6bb3a0f_8779_436c_b404_a26849ff7943.slice/crio-8deb25233a75babfd1f6f84c0365d2edc76bcc2a3b83c3e75fa6a0270f18777a WatchSource:0}: Error finding container 8deb25233a75babfd1f6f84c0365d2edc76bcc2a3b83c3e75fa6a0270f18777a: Status 404 returned error can't find the container with id 8deb25233a75babfd1f6f84c0365d2edc76bcc2a3b83c3e75fa6a0270f18777a
Oct 30 18:47:39.666018 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:39.665991    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-226ecbcb-5d69cc78dd-rjzsb"]
Oct 30 18:47:39.666444 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:47:39.666146    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52ca0496_3987_481c_8efa_28c13e432616.slice/crio-ff9c1252eac0b002ec98ee54816df1a21492ae1d5958bfc2d88ad011211ecf4e WatchSource:0}: Error finding container ff9c1252eac0b002ec98ee54816df1a21492ae1d5958bfc2d88ad011211ecf4e: Status 404 returned error can't find the container with id ff9c1252eac0b002ec98ee54816df1a21492ae1d5958bfc2d88ad011211ecf4e
Oct 30 18:47:39.757072 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:39.757035    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58554b4f-e880-4ac7-ac7a-d1d4cf33851a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e4242163-d74b5477f-fwc2h\" (UID: \"58554b4f-e880-4ac7-ac7a-d1d4cf33851a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e4242163-d74b5477f-fwc2h"
Oct 30 18:47:39.757251 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:39.757165    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb4vb\" (UniqueName: \"kubernetes.io/projected/58554b4f-e880-4ac7-ac7a-d1d4cf33851a-kube-api-access-hb4vb\") pod \"payload-pre-processing-e2e-ait-e4242163-d74b5477f-fwc2h\" (UID: \"58554b4f-e880-4ac7-ac7a-d1d4cf33851a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e4242163-d74b5477f-fwc2h"
Oct 30 18:47:39.757251 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:39.757238    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4wg6\" (UniqueName: \"kubernetes.io/projected/17744dc5-6240-4081-bc2b-768529e1ed13-kube-api-access-c4wg6\") pod \"payload-processing-e2e-ait-e4242163-6fddb59d56-jvvtf\" (UID: \"17744dc5-6240-4081-bc2b-768529e1ed13\") " pod="openshift-ingress/payload-processing-e2e-ait-e4242163-6fddb59d56-jvvtf"
Oct 30 18:47:39.757393 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:39.757268    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17744dc5-6240-4081-bc2b-768529e1ed13-plugins-config-volume\") pod \"payload-processing-e2e-ait-e4242163-6fddb59d56-jvvtf\" (UID: \"17744dc5-6240-4081-bc2b-768529e1ed13\") " pod="openshift-ingress/payload-processing-e2e-ait-e4242163-6fddb59d56-jvvtf"
Oct 30 18:47:39.857990 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:39.857953    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb4vb\" (UniqueName: \"kubernetes.io/projected/58554b4f-e880-4ac7-ac7a-d1d4cf33851a-kube-api-access-hb4vb\") pod \"payload-pre-processing-e2e-ait-e4242163-d74b5477f-fwc2h\" (UID: \"58554b4f-e880-4ac7-ac7a-d1d4cf33851a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e4242163-d74b5477f-fwc2h"
Oct 30 18:47:39.858184 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:39.858034    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4wg6\" (UniqueName: \"kubernetes.io/projected/17744dc5-6240-4081-bc2b-768529e1ed13-kube-api-access-c4wg6\") pod \"payload-processing-e2e-ait-e4242163-6fddb59d56-jvvtf\" (UID: \"17744dc5-6240-4081-bc2b-768529e1ed13\") " pod="openshift-ingress/payload-processing-e2e-ait-e4242163-6fddb59d56-jvvtf"
Oct 30 18:47:39.858184 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:39.858078    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17744dc5-6240-4081-bc2b-768529e1ed13-plugins-config-volume\") pod \"payload-processing-e2e-ait-e4242163-6fddb59d56-jvvtf\" (UID: \"17744dc5-6240-4081-bc2b-768529e1ed13\") " pod="openshift-ingress/payload-processing-e2e-ait-e4242163-6fddb59d56-jvvtf"
Oct 30 18:47:39.858184 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:39.858165    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58554b4f-e880-4ac7-ac7a-d1d4cf33851a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e4242163-d74b5477f-fwc2h\" (UID: \"58554b4f-e880-4ac7-ac7a-d1d4cf33851a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e4242163-d74b5477f-fwc2h"
Oct 30 18:47:39.859434 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:39.859401    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58554b4f-e880-4ac7-ac7a-d1d4cf33851a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e4242163-d74b5477f-fwc2h\" (UID: \"58554b4f-e880-4ac7-ac7a-d1d4cf33851a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e4242163-d74b5477f-fwc2h"
Oct 30 18:47:39.861574 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:39.861545    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17744dc5-6240-4081-bc2b-768529e1ed13-plugins-config-volume\") pod \"payload-processing-e2e-ait-e4242163-6fddb59d56-jvvtf\" (UID: \"17744dc5-6240-4081-bc2b-768529e1ed13\") " pod="openshift-ingress/payload-processing-e2e-ait-e4242163-6fddb59d56-jvvtf"
Oct 30 18:47:39.872549 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:39.872523    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4wg6\" (UniqueName: \"kubernetes.io/projected/17744dc5-6240-4081-bc2b-768529e1ed13-kube-api-access-c4wg6\") pod \"payload-processing-e2e-ait-e4242163-6fddb59d56-jvvtf\" (UID: \"17744dc5-6240-4081-bc2b-768529e1ed13\") " pod="openshift-ingress/payload-processing-e2e-ait-e4242163-6fddb59d56-jvvtf"
Oct 30 18:47:39.872549 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:39.872547    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb4vb\" (UniqueName: \"kubernetes.io/projected/58554b4f-e880-4ac7-ac7a-d1d4cf33851a-kube-api-access-hb4vb\") pod \"payload-pre-processing-e2e-ait-e4242163-d74b5477f-fwc2h\" (UID: \"58554b4f-e880-4ac7-ac7a-d1d4cf33851a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e4242163-d74b5477f-fwc2h"
Oct 30 18:47:39.906864 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:39.906831    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e4242163-d74b5477f-fwc2h"
Oct 30 18:47:39.922909 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:39.922888    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e4242163-6fddb59d56-jvvtf"
Oct 30 18:47:39.933995 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:39.933951    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-226ecbcb-5d69cc78dd-rjzsb" event={"ID":"52ca0496-3987-481c-8efa-28c13e432616","Type":"ContainerStarted","Data":"ff9c1252eac0b002ec98ee54816df1a21492ae1d5958bfc2d88ad011211ecf4e"}
Oct 30 18:47:39.935480 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:39.935448    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-226ecbcb-69b85788bc-8xszf" event={"ID":"f6bb3a0f-8779-436c-b404-a26849ff7943","Type":"ContainerStarted","Data":"8deb25233a75babfd1f6f84c0365d2edc76bcc2a3b83c3e75fa6a0270f18777a"}
Oct 30 18:47:40.140958 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:40.140868    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ee87bf29-5dbc87966c-x9gwt"]
Oct 30 18:47:40.146531 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:40.146507    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ee87bf29-5dbc87966c-x9gwt"
Oct 30 18:47:40.159461 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:40.159431    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-ee87bf29-dockercfg-ggkl7\""
Oct 30 18:47:40.161950 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:40.160979    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ee87bf29-5dbc87966c-x9gwt"]
Oct 30 18:47:40.224725 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:40.224447    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e4242163-d74b5477f-fwc2h"]
Oct 30 18:47:40.224840 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:40.224756    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-ee87bf29\""
Oct 30 18:47:40.237286 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:40.237258    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e4242163-6fddb59d56-jvvtf"]
Oct 30 18:47:40.243706 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:47:40.242236    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17744dc5_6240_4081_bc2b_768529e1ed13.slice/crio-a78e4f2835ceb32e5786593771b76cb2b523b9592bc37c404d4e7241f5b4c0a6 WatchSource:0}: Error finding container a78e4f2835ceb32e5786593771b76cb2b523b9592bc37c404d4e7241f5b4c0a6: Status 404 returned error can't find the container with id a78e4f2835ceb32e5786593771b76cb2b523b9592bc37c404d4e7241f5b4c0a6
Oct 30 18:47:40.262044 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:40.262010    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-286rw\" (UniqueName: \"kubernetes.io/projected/4f1a2f0a-2bdd-4765-b81c-38323e22fbe9-kube-api-access-286rw\") pod \"payload-pre-processing-e2e-ait-ee87bf29-5dbc87966c-x9gwt\" (UID: \"4f1a2f0a-2bdd-4765-b81c-38323e22fbe9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ee87bf29-5dbc87966c-x9gwt"
Oct 30 18:47:40.262211 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:40.262121    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f1a2f0a-2bdd-4765-b81c-38323e22fbe9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ee87bf29-5dbc87966c-x9gwt\" (UID: \"4f1a2f0a-2bdd-4765-b81c-38323e22fbe9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ee87bf29-5dbc87966c-x9gwt"
Oct 30 18:47:40.363390 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:40.363341    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f1a2f0a-2bdd-4765-b81c-38323e22fbe9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ee87bf29-5dbc87966c-x9gwt\" (UID: \"4f1a2f0a-2bdd-4765-b81c-38323e22fbe9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ee87bf29-5dbc87966c-x9gwt"
Oct 30 18:47:40.363581 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:40.363520    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-286rw\" (UniqueName: \"kubernetes.io/projected/4f1a2f0a-2bdd-4765-b81c-38323e22fbe9-kube-api-access-286rw\") pod \"payload-pre-processing-e2e-ait-ee87bf29-5dbc87966c-x9gwt\" (UID: \"4f1a2f0a-2bdd-4765-b81c-38323e22fbe9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ee87bf29-5dbc87966c-x9gwt"
Oct 30 18:47:40.364712 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:40.364671    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f1a2f0a-2bdd-4765-b81c-38323e22fbe9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ee87bf29-5dbc87966c-x9gwt\" (UID: \"4f1a2f0a-2bdd-4765-b81c-38323e22fbe9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ee87bf29-5dbc87966c-x9gwt"
Oct 30 18:47:40.373447 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:40.373420    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-286rw\" (UniqueName: \"kubernetes.io/projected/4f1a2f0a-2bdd-4765-b81c-38323e22fbe9-kube-api-access-286rw\") pod \"payload-pre-processing-e2e-ait-ee87bf29-5dbc87966c-x9gwt\" (UID: \"4f1a2f0a-2bdd-4765-b81c-38323e22fbe9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ee87bf29-5dbc87966c-x9gwt"
Oct 30 18:47:40.475974 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:40.475846    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ee87bf29-5dbc87966c-x9gwt"
Oct 30 18:47:40.788051 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:40.787974    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ee87bf29-5dbc87966c-x9gwt"]
Oct 30 18:47:40.949288 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:40.949196    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e4242163-d74b5477f-fwc2h" event={"ID":"58554b4f-e880-4ac7-ac7a-d1d4cf33851a","Type":"ContainerStarted","Data":"e3bcf3a9e9e66d983b4b6227c4bd27e79f8877d0987a116eb006a9573804ad33"}
Oct 30 18:47:40.952084 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:40.952005    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ee87bf29-5dbc87966c-x9gwt" event={"ID":"4f1a2f0a-2bdd-4765-b81c-38323e22fbe9","Type":"ContainerStarted","Data":"49ef42b723f3b09045fb908466ecc8e5a895100bc1ed97d1b82a20b602029602"}
Oct 30 18:47:40.955576 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:40.955524    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e4242163-6fddb59d56-jvvtf" event={"ID":"17744dc5-6240-4081-bc2b-768529e1ed13","Type":"ContainerStarted","Data":"a78e4f2835ceb32e5786593771b76cb2b523b9592bc37c404d4e7241f5b4c0a6"}
Oct 30 18:47:42.959020 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:42.958984    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-f951dfa9-openshift-default-5ddd96b597-dzjdp"]
Oct 30 18:47:42.964728 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:42.964700    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-f951dfa9-openshift-default-5ddd96b597-dzjdp"
Oct 30 18:47:42.977542 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:42.977517    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-f951dfa9-openshift-default-dockercfg-nvm4g\""
Oct 30 18:47:42.987525 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:42.987450    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e4242163-6fddb59d56-jvvtf" event={"ID":"17744dc5-6240-4081-bc2b-768529e1ed13","Type":"ContainerStarted","Data":"b127defd743b7f16bb5fa59d158822e6e3e56056c8c5994168c9d4589bcda1a7"}
Oct 30 18:47:42.988007 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:42.987986    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-e4242163-6fddb59d56-jvvtf"
Oct 30 18:47:42.990317 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:42.988818    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-f951dfa9-openshift-default-5ddd96b597-dzjdp"]
Oct 30 18:47:42.990773 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:42.990479    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6156d5d6-55fcbb8767-4w5g4" event={"ID":"1cec3994-12ce-4931-9d31-1b620974dd99","Type":"ContainerStarted","Data":"13b0f20ad0b600e075628047ec06ebd902e3dda65123b9adfe77e5394c00f429"}
Oct 30 18:47:42.990773 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:42.990561    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6156d5d6-55fcbb8767-4w5g4"
Oct 30 18:47:42.996162 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:42.995557    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e4242163-d74b5477f-fwc2h" event={"ID":"58554b4f-e880-4ac7-ac7a-d1d4cf33851a","Type":"ContainerStarted","Data":"d665521a08bfe260a306388acef0050fcf79f7d6195288686625d7bb597f0e9e"}
Oct 30 18:47:42.996162 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:42.996109    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-e4242163-d74b5477f-fwc2h"
Oct 30 18:47:43.057421 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:43.056633    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-e4242163-d74b5477f-fwc2h" podStartSLOduration=1.62624965 podStartE2EDuration="4.056613896s" totalImagesPullingTime="2.430364246s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:47:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:43.054179839 +0000 UTC m=+685.679849803" watchObservedRunningTime="2026-10-30 18:47:43.056613896 +0000 UTC m=+685.682283861"
Oct 30 18:47:43.083788 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:43.083729    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-e4242163-6fddb59d56-jvvtf" podStartSLOduration=1.519779099 podStartE2EDuration="4.083715952s" totalImagesPullingTime="2.563936853s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:47:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:43.081576192 +0000 UTC m=+685.707246157" watchObservedRunningTime="2026-10-30 18:47:43.083715952 +0000 UTC m=+685.709385917"
Oct 30 18:47:43.107703 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:43.107659    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a29fd997-17ba-4260-8279-05e6f1c6346a-istio-data\") pod \"e2e-derive-f951dfa9-openshift-default-5ddd96b597-dzjdp\" (UID: \"a29fd997-17ba-4260-8279-05e6f1c6346a\") " pod="openshift-ingress/e2e-derive-f951dfa9-openshift-default-5ddd96b597-dzjdp"
Oct 30 18:47:43.107805 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:43.107714    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a29fd997-17ba-4260-8279-05e6f1c6346a-istio-token\") pod \"e2e-derive-f951dfa9-openshift-default-5ddd96b597-dzjdp\" (UID: \"a29fd997-17ba-4260-8279-05e6f1c6346a\") " pod="openshift-ingress/e2e-derive-f951dfa9-openshift-default-5ddd96b597-dzjdp"
Oct 30 18:47:43.107805 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:43.107734    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wttcv\" (UniqueName: \"kubernetes.io/projected/a29fd997-17ba-4260-8279-05e6f1c6346a-kube-api-access-wttcv\") pod \"e2e-derive-f951dfa9-openshift-default-5ddd96b597-dzjdp\" (UID: \"a29fd997-17ba-4260-8279-05e6f1c6346a\") " pod="openshift-ingress/e2e-derive-f951dfa9-openshift-default-5ddd96b597-dzjdp"
Oct 30 18:47:43.107805 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:43.107779    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a29fd997-17ba-4260-8279-05e6f1c6346a-workload-certs\") pod \"e2e-derive-f951dfa9-openshift-default-5ddd96b597-dzjdp\" (UID: \"a29fd997-17ba-4260-8279-05e6f1c6346a\") " pod="openshift-ingress/e2e-derive-f951dfa9-openshift-default-5ddd96b597-dzjdp"
Oct 30 18:47:43.107805 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:43.107796    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a29fd997-17ba-4260-8279-05e6f1c6346a-istio-envoy\") pod \"e2e-derive-f951dfa9-openshift-default-5ddd96b597-dzjdp\" (UID: \"a29fd997-17ba-4260-8279-05e6f1c6346a\") " pod="openshift-ingress/e2e-derive-f951dfa9-openshift-default-5ddd96b597-dzjdp"
Oct 30 18:47:43.107969 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:43.107839    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a29fd997-17ba-4260-8279-05e6f1c6346a-istio-podinfo\") pod \"e2e-derive-f951dfa9-openshift-default-5ddd96b597-dzjdp\" (UID: \"a29fd997-17ba-4260-8279-05e6f1c6346a\") " pod="openshift-ingress/e2e-derive-f951dfa9-openshift-default-5ddd96b597-dzjdp"
Oct 30 18:47:43.107969 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:43.107922    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a29fd997-17ba-4260-8279-05e6f1c6346a-workload-socket\") pod \"e2e-derive-f951dfa9-openshift-default-5ddd96b597-dzjdp\" (UID: \"a29fd997-17ba-4260-8279-05e6f1c6346a\") " pod="openshift-ingress/e2e-derive-f951dfa9-openshift-default-5ddd96b597-dzjdp"
Oct 30 18:47:43.108034 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:43.108011    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a29fd997-17ba-4260-8279-05e6f1c6346a-credential-socket\") pod \"e2e-derive-f951dfa9-openshift-default-5ddd96b597-dzjdp\" (UID: \"a29fd997-17ba-4260-8279-05e6f1c6346a\") " pod="openshift-ingress/e2e-derive-f951dfa9-openshift-default-5ddd96b597-dzjdp"
Oct 30 18:47:43.108064 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:43.108048    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a29fd997-17ba-4260-8279-05e6f1c6346a-istiod-ca-cert\") pod \"e2e-derive-f951dfa9-openshift-default-5ddd96b597-dzjdp\" (UID: \"a29fd997-17ba-4260-8279-05e6f1c6346a\") " pod="openshift-ingress/e2e-derive-f951dfa9-openshift-default-5ddd96b597-dzjdp"
Oct 30 18:47:43.208902 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:43.208872    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a29fd997-17ba-4260-8279-05e6f1c6346a-workload-certs\") pod \"e2e-derive-f951dfa9-openshift-default-5ddd96b597-dzjdp\" (UID: \"a29fd997-17ba-4260-8279-05e6f1c6346a\") " pod="openshift-ingress/e2e-derive-f951dfa9-openshift-default-5ddd96b597-dzjdp"
Oct 30 18:47:43.209065 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:43.208914    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a29fd997-17ba-4260-8279-05e6f1c6346a-istio-envoy\") pod \"e2e-derive-f951dfa9-openshift-default-5ddd96b597-dzjdp\" (UID: \"a29fd997-17ba-4260-8279-05e6f1c6346a\") " pod="openshift-ingress/e2e-derive-f951dfa9-openshift-default-5ddd96b597-dzjdp"
Oct 30 18:47:43.209065 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:43.208951    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a29fd997-17ba-4260-8279-05e6f1c6346a-istio-podinfo\") pod \"e2e-derive-f951dfa9-openshift-default-5ddd96b597-dzjdp\" (UID: \"a29fd997-17ba-4260-8279-05e6f1c6346a\") " pod="openshift-ingress/e2e-derive-f951dfa9-openshift-default-5ddd96b597-dzjdp"
Oct 30 18:47:43.209065 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:43.208991    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a29fd997-17ba-4260-8279-05e6f1c6346a-workload-socket\") pod \"e2e-derive-f951dfa9-openshift-default-5ddd96b597-dzjdp\" (UID: \"a29fd997-17ba-4260-8279-05e6f1c6346a\") " pod="openshift-ingress/e2e-derive-f951dfa9-openshift-default-5ddd96b597-dzjdp"
Oct 30 18:47:43.209065 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:43.209044    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a29fd997-17ba-4260-8279-05e6f1c6346a-credential-socket\") pod \"e2e-derive-f951dfa9-openshift-default-5ddd96b597-dzjdp\" (UID: \"a29fd997-17ba-4260-8279-05e6f1c6346a\") " pod="openshift-ingress/e2e-derive-f951dfa9-openshift-default-5ddd96b597-dzjdp"
Oct 30 18:47:43.209258 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:43.209091    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a29fd997-17ba-4260-8279-05e6f1c6346a-istiod-ca-cert\") pod \"e2e-derive-f951dfa9-openshift-default-5ddd96b597-dzjdp\" (UID: \"a29fd997-17ba-4260-8279-05e6f1c6346a\") " pod="openshift-ingress/e2e-derive-f951dfa9-openshift-default-5ddd96b597-dzjdp"
Oct 30 18:47:43.209258 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:43.209130    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a29fd997-17ba-4260-8279-05e6f1c6346a-istio-data\") pod \"e2e-derive-f951dfa9-openshift-default-5ddd96b597-dzjdp\" (UID: \"a29fd997-17ba-4260-8279-05e6f1c6346a\") " pod="openshift-ingress/e2e-derive-f951dfa9-openshift-default-5ddd96b597-dzjdp"
Oct 30 18:47:43.209258 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:43.209156    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a29fd997-17ba-4260-8279-05e6f1c6346a-istio-token\") pod \"e2e-derive-f951dfa9-openshift-default-5ddd96b597-dzjdp\" (UID: \"a29fd997-17ba-4260-8279-05e6f1c6346a\") " pod="openshift-ingress/e2e-derive-f951dfa9-openshift-default-5ddd96b597-dzjdp"
Oct 30 18:47:43.209258 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:43.209180    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wttcv\" (UniqueName: \"kubernetes.io/projected/a29fd997-17ba-4260-8279-05e6f1c6346a-kube-api-access-wttcv\") pod \"e2e-derive-f951dfa9-openshift-default-5ddd96b597-dzjdp\" (UID: \"a29fd997-17ba-4260-8279-05e6f1c6346a\") " pod="openshift-ingress/e2e-derive-f951dfa9-openshift-default-5ddd96b597-dzjdp"
Oct 30 18:47:43.209450 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:43.209270    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a29fd997-17ba-4260-8279-05e6f1c6346a-workload-certs\") pod \"e2e-derive-f951dfa9-openshift-default-5ddd96b597-dzjdp\" (UID: \"a29fd997-17ba-4260-8279-05e6f1c6346a\") " pod="openshift-ingress/e2e-derive-f951dfa9-openshift-default-5ddd96b597-dzjdp"
Oct 30 18:47:43.209512 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:43.209492    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a29fd997-17ba-4260-8279-05e6f1c6346a-credential-socket\") pod \"e2e-derive-f951dfa9-openshift-default-5ddd96b597-dzjdp\" (UID: \"a29fd997-17ba-4260-8279-05e6f1c6346a\") " pod="openshift-ingress/e2e-derive-f951dfa9-openshift-default-5ddd96b597-dzjdp"
Oct 30 18:47:43.210108 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:43.210077    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a29fd997-17ba-4260-8279-05e6f1c6346a-istio-data\") pod \"e2e-derive-f951dfa9-openshift-default-5ddd96b597-dzjdp\" (UID: \"a29fd997-17ba-4260-8279-05e6f1c6346a\") " pod="openshift-ingress/e2e-derive-f951dfa9-openshift-default-5ddd96b597-dzjdp"
Oct 30 18:47:43.210225 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:43.210150    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a29fd997-17ba-4260-8279-05e6f1c6346a-istiod-ca-cert\") pod \"e2e-derive-f951dfa9-openshift-default-5ddd96b597-dzjdp\" (UID: \"a29fd997-17ba-4260-8279-05e6f1c6346a\") " pod="openshift-ingress/e2e-derive-f951dfa9-openshift-default-5ddd96b597-dzjdp"
Oct 30 18:47:43.210402 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:43.210330    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a29fd997-17ba-4260-8279-05e6f1c6346a-workload-socket\") pod \"e2e-derive-f951dfa9-openshift-default-5ddd96b597-dzjdp\" (UID: \"a29fd997-17ba-4260-8279-05e6f1c6346a\") " pod="openshift-ingress/e2e-derive-f951dfa9-openshift-default-5ddd96b597-dzjdp"
Oct 30 18:47:43.225089 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:43.225056    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a29fd997-17ba-4260-8279-05e6f1c6346a-istio-podinfo\") pod \"e2e-derive-f951dfa9-openshift-default-5ddd96b597-dzjdp\" (UID: \"a29fd997-17ba-4260-8279-05e6f1c6346a\") " pod="openshift-ingress/e2e-derive-f951dfa9-openshift-default-5ddd96b597-dzjdp"
Oct 30 18:47:43.226361 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:43.226314    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a29fd997-17ba-4260-8279-05e6f1c6346a-istio-envoy\") pod \"e2e-derive-f951dfa9-openshift-default-5ddd96b597-dzjdp\" (UID: \"a29fd997-17ba-4260-8279-05e6f1c6346a\") " pod="openshift-ingress/e2e-derive-f951dfa9-openshift-default-5ddd96b597-dzjdp"
Oct 30 18:47:43.228359 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:43.228333    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a29fd997-17ba-4260-8279-05e6f1c6346a-istio-token\") pod \"e2e-derive-f951dfa9-openshift-default-5ddd96b597-dzjdp\" (UID: \"a29fd997-17ba-4260-8279-05e6f1c6346a\") " pod="openshift-ingress/e2e-derive-f951dfa9-openshift-default-5ddd96b597-dzjdp"
Oct 30 18:47:43.228497 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:43.228385    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wttcv\" (UniqueName: \"kubernetes.io/projected/a29fd997-17ba-4260-8279-05e6f1c6346a-kube-api-access-wttcv\") pod \"e2e-derive-f951dfa9-openshift-default-5ddd96b597-dzjdp\" (UID: \"a29fd997-17ba-4260-8279-05e6f1c6346a\") " pod="openshift-ingress/e2e-derive-f951dfa9-openshift-default-5ddd96b597-dzjdp"
Oct 30 18:47:43.283520 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:43.283427    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-f951dfa9-openshift-default-5ddd96b597-dzjdp"
Oct 30 18:47:43.595948 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:43.595036    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6156d5d6-55fcbb8767-4w5g4" podStartSLOduration=2.749713912 podStartE2EDuration="6.595013883s" totalImagesPullingTime="3.845299971s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:47:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:43.107262332 +0000 UTC m=+685.732932297" watchObservedRunningTime="2026-10-30 18:47:43.595013883 +0000 UTC m=+686.220683848"
Oct 30 18:47:43.598371 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:43.598333    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-f951dfa9-openshift-default-5ddd96b597-dzjdp"]
Oct 30 18:47:43.599937 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:47:43.599911    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda29fd997_17ba_4260_8279_05e6f1c6346a.slice/crio-1ed4ee44df54a937d84fe1612f6ba1e9990b982377adc2bc9919281f16d159bd WatchSource:0}: Error finding container 1ed4ee44df54a937d84fe1612f6ba1e9990b982377adc2bc9919281f16d159bd: Status 404 returned error can't find the container with id 1ed4ee44df54a937d84fe1612f6ba1e9990b982377adc2bc9919281f16d159bd
Oct 30 18:47:43.605584 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:43.604097    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:47:43.605584 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:43.604333    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:47:43.605584 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:43.604452    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:47:44.002820 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:44.002780    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ee87bf29-5dbc87966c-x9gwt" event={"ID":"4f1a2f0a-2bdd-4765-b81c-38323e22fbe9","Type":"ContainerStarted","Data":"9d8ccae45f0e180b0649552c26d0705ad140f1e4fe51d4e1e41725f805f2c0f5"}
Oct 30 18:47:44.003334 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:44.002895    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ee87bf29-5dbc87966c-x9gwt"
Oct 30 18:47:44.004613 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:44.004583    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f951dfa9-openshift-default-5ddd96b597-dzjdp" event={"ID":"a29fd997-17ba-4260-8279-05e6f1c6346a","Type":"ContainerStarted","Data":"004fe667a937ac9a8bc60137eeedd2bcd7407bd1ca8df0fd781dbc0ebad252b2"}
Oct 30 18:47:44.004783 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:44.004629    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f951dfa9-openshift-default-5ddd96b597-dzjdp" event={"ID":"a29fd997-17ba-4260-8279-05e6f1c6346a","Type":"ContainerStarted","Data":"1ed4ee44df54a937d84fe1612f6ba1e9990b982377adc2bc9919281f16d159bd"}
Oct 30 18:47:44.006130 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:44.006107    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-226ecbcb-5d69cc78dd-rjzsb" event={"ID":"52ca0496-3987-481c-8efa-28c13e432616","Type":"ContainerStarted","Data":"e1684db67038f8b9d24c3d17848bf2a5c9f86be6d37613a987ff060991029960"}
Oct 30 18:47:44.006267 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:44.006244    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-226ecbcb-5d69cc78dd-rjzsb"
Oct 30 18:47:44.007804 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:44.007773    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-226ecbcb-69b85788bc-8xszf" event={"ID":"f6bb3a0f-8779-436c-b404-a26849ff7943","Type":"ContainerStarted","Data":"0376469a7047eec342bef411fb5114e4b1370362bc70d632a6a5e67ad0835c9d"}
Oct 30 18:47:44.007968 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:44.007953    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-226ecbcb-69b85788bc-8xszf"
Oct 30 18:47:44.026373 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:44.026330    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-ee87bf29-5dbc87966c-x9gwt" podStartSLOduration=1.726565599 podStartE2EDuration="4.026319281s" totalImagesPullingTime="2.299753682s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:47:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:44.024227386 +0000 UTC m=+686.649897350" watchObservedRunningTime="2026-10-30 18:47:44.026319281 +0000 UTC m=+686.651989285"
Oct 30 18:47:44.043999 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:44.043957    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-226ecbcb-69b85788bc-8xszf" podStartSLOduration=2.253883395 podStartE2EDuration="6.043945408s" totalImagesPullingTime="3.790062013s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:47:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:44.04233105 +0000 UTC m=+686.668001016" watchObservedRunningTime="2026-10-30 18:47:44.043945408 +0000 UTC m=+686.669615372"
Oct 30 18:47:44.067193 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:44.067148    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-f951dfa9-openshift-default-5ddd96b597-dzjdp" podStartSLOduration=2.067136295 podStartE2EDuration="2.067136295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:47:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:44.06443277 +0000 UTC m=+686.690102731" watchObservedRunningTime="2026-10-30 18:47:44.067136295 +0000 UTC m=+686.692806259"
Oct 30 18:47:44.087797 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:44.087758    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-226ecbcb-5d69cc78dd-rjzsb" podStartSLOduration=1.306420646 podStartE2EDuration="5.087742781s" totalImagesPullingTime="3.781322135s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:47:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:44.085954712 +0000 UTC m=+686.711624675" watchObservedRunningTime="2026-10-30 18:47:44.087742781 +0000 UTC m=+686.713412747"
Oct 30 18:47:44.284279 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:44.284191    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-f951dfa9-openshift-default-5ddd96b597-dzjdp"
Oct 30 18:47:44.290768 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:44.290741    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-f951dfa9-openshift-default-5ddd96b597-dzjdp"
Oct 30 18:47:45.022824 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:45.022760    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-f951dfa9-openshift-default-5ddd96b597-dzjdp"
Oct 30 18:47:45.025717 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:45.025590    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-f951dfa9-openshift-default-5ddd96b597-dzjdp"
Oct 30 18:47:47.397744 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:47:47.397708    4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d49l" containerName="extractor"
Oct 30 18:47:52.098216 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:52.098183    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6156d5d6-7rts5"]
Oct 30 18:47:52.103171 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:52.103151    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6156d5d6-7rts5"
Oct 30 18:47:52.104666 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:52.104645    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2f88b91-9a74-4024-9218-9e35482e8d8d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-6156d5d6-7rts5\" (UID: \"f2f88b91-9a74-4024-9218-9e35482e8d8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6156d5d6-7rts5"
Oct 30 18:47:52.104776 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:52.104760    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2f88b91-9a74-4024-9218-9e35482e8d8d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-6156d5d6-7rts5\" (UID: \"f2f88b91-9a74-4024-9218-9e35482e8d8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6156d5d6-7rts5"
Oct 30 18:47:52.104827 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:52.104787    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sr5sx\" (UniqueName: \"kubernetes.io/projected/f2f88b91-9a74-4024-9218-9e35482e8d8d-kube-api-access-sr5sx\") pod \"maas-api-revoke-keys-e2e-migrate-6156d5d6-7rts5\" (UID: \"f2f88b91-9a74-4024-9218-9e35482e8d8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6156d5d6-7rts5"
Oct 30 18:47:52.106446 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:52.106425    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:47:52.107388 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:52.107371    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94hcn\""
Oct 30 18:47:52.118662 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:52.118635    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6156d5d6-7rts5"]
Oct 30 18:47:52.205301 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:52.205263    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2f88b91-9a74-4024-9218-9e35482e8d8d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-6156d5d6-7rts5\" (UID: \"f2f88b91-9a74-4024-9218-9e35482e8d8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6156d5d6-7rts5"
Oct 30 18:47:52.205467 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:52.205306    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sr5sx\" (UniqueName: \"kubernetes.io/projected/f2f88b91-9a74-4024-9218-9e35482e8d8d-kube-api-access-sr5sx\") pod \"maas-api-revoke-keys-e2e-migrate-6156d5d6-7rts5\" (UID: \"f2f88b91-9a74-4024-9218-9e35482e8d8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6156d5d6-7rts5"
Oct 30 18:47:52.205467 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:52.205418    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2f88b91-9a74-4024-9218-9e35482e8d8d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-6156d5d6-7rts5\" (UID: \"f2f88b91-9a74-4024-9218-9e35482e8d8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6156d5d6-7rts5"
Oct 30 18:47:52.206062 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:52.206040    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2f88b91-9a74-4024-9218-9e35482e8d8d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-6156d5d6-7rts5\" (UID: \"f2f88b91-9a74-4024-9218-9e35482e8d8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6156d5d6-7rts5"
Oct 30 18:47:52.207742 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:52.207726    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2f88b91-9a74-4024-9218-9e35482e8d8d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-6156d5d6-7rts5\" (UID: \"f2f88b91-9a74-4024-9218-9e35482e8d8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6156d5d6-7rts5"
Oct 30 18:47:52.214068 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:52.214048    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sr5sx\" (UniqueName: \"kubernetes.io/projected/f2f88b91-9a74-4024-9218-9e35482e8d8d-kube-api-access-sr5sx\") pod \"maas-api-revoke-keys-e2e-migrate-6156d5d6-7rts5\" (UID: \"f2f88b91-9a74-4024-9218-9e35482e8d8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6156d5d6-7rts5"
Oct 30 18:47:52.416176 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:52.416144    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6156d5d6-7rts5"
Oct 30 18:47:52.692312 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:52.692282    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6156d5d6-7rts5"]
Oct 30 18:47:52.692915 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:47:52.692880    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2f88b91_9a74_4024_9218_9e35482e8d8d.slice/crio-244ebe1732037a14b218fb08d772a7e84663d1f39275ca54f0fc500516900f0c WatchSource:0}: Error finding container 244ebe1732037a14b218fb08d772a7e84663d1f39275ca54f0fc500516900f0c: Status 404 returned error can't find the container with id 244ebe1732037a14b218fb08d772a7e84663d1f39275ca54f0fc500516900f0c
Oct 30 18:47:53.068344 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:53.068265    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6156d5d6-7rts5" event={"ID":"f2f88b91-9a74-4024-9218-9e35482e8d8d","Type":"ContainerStarted","Data":"244ebe1732037a14b218fb08d772a7e84663d1f39275ca54f0fc500516900f0c"}
Oct 30 18:47:54.011190 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:54.011161    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6156d5d6-55fcbb8767-4w5g4"
Oct 30 18:47:54.011579 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:54.011233    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-e4242163-6fddb59d56-jvvtf"
Oct 30 18:47:55.025771 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:55.024857    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ee87bf29-5dbc87966c-x9gwt"
Oct 30 18:47:55.025771 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:55.025612    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-e4242163-d74b5477f-fwc2h"
Oct 30 18:47:55.026248 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:55.026091    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-226ecbcb-5d69cc78dd-rjzsb"
Oct 30 18:47:55.026468 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:55.026446    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-226ecbcb-69b85788bc-8xszf"
Oct 30 18:47:57.127969 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:57.127934    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6156d5d6-7rts5" event={"ID":"f2f88b91-9a74-4024-9218-9e35482e8d8d","Type":"ContainerStarted","Data":"cc675924e87bef2fafee0a17ec9becf29ee2222b946d72ac6784116c1224accd"}
Oct 30 18:47:57.145925 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:57.145857    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6156d5d6-7rts5" podStartSLOduration=1.6911312619999999 podStartE2EDuration="5.145841611s" totalImagesPullingTime="3.454710349s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:47:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:57.14446879 +0000 UTC m=+699.770138744" watchObservedRunningTime="2026-10-30 18:47:57.145841611 +0000 UTC m=+699.771511574"
Oct 30 18:47:57.959129 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:57.959091    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-226ecbcb-jmv2f"]
Oct 30 18:47:57.962809 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:57.962793    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-226ecbcb-jmv2f"
Oct 30 18:47:57.971673 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:57.971646    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-226ecbcb-jmv2f"]
Oct 30 18:47:58.003213 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:58.003156    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06b3b567-76cd-437c-a3b4-1324adad5279-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-226ecbcb-jmv2f\" (UID: \"06b3b567-76cd-437c-a3b4-1324adad5279\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-226ecbcb-jmv2f"
Oct 30 18:47:58.003373 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:58.003245    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06b3b567-76cd-437c-a3b4-1324adad5279-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-226ecbcb-jmv2f\" (UID: \"06b3b567-76cd-437c-a3b4-1324adad5279\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-226ecbcb-jmv2f"
Oct 30 18:47:58.003422 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:58.003406    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7jv4\" (UniqueName: \"kubernetes.io/projected/06b3b567-76cd-437c-a3b4-1324adad5279-kube-api-access-l7jv4\") pod \"maas-api-revoke-keys-e2e-ait-226ecbcb-jmv2f\" (UID: \"06b3b567-76cd-437c-a3b4-1324adad5279\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-226ecbcb-jmv2f"
Oct 30 18:47:58.104048 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:58.104012    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7jv4\" (UniqueName: \"kubernetes.io/projected/06b3b567-76cd-437c-a3b4-1324adad5279-kube-api-access-l7jv4\") pod \"maas-api-revoke-keys-e2e-ait-226ecbcb-jmv2f\" (UID: \"06b3b567-76cd-437c-a3b4-1324adad5279\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-226ecbcb-jmv2f"
Oct 30 18:47:58.104251 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:58.104066    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06b3b567-76cd-437c-a3b4-1324adad5279-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-226ecbcb-jmv2f\" (UID: \"06b3b567-76cd-437c-a3b4-1324adad5279\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-226ecbcb-jmv2f"
Oct 30 18:47:58.104251 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:58.104106    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06b3b567-76cd-437c-a3b4-1324adad5279-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-226ecbcb-jmv2f\" (UID: \"06b3b567-76cd-437c-a3b4-1324adad5279\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-226ecbcb-jmv2f"
Oct 30 18:47:58.104787 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:58.104762    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06b3b567-76cd-437c-a3b4-1324adad5279-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-226ecbcb-jmv2f\" (UID: \"06b3b567-76cd-437c-a3b4-1324adad5279\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-226ecbcb-jmv2f"
Oct 30 18:47:58.106668 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:58.106643    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06b3b567-76cd-437c-a3b4-1324adad5279-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-226ecbcb-jmv2f\" (UID: \"06b3b567-76cd-437c-a3b4-1324adad5279\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-226ecbcb-jmv2f"
Oct 30 18:47:58.112307 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:58.112285    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7jv4\" (UniqueName: \"kubernetes.io/projected/06b3b567-76cd-437c-a3b4-1324adad5279-kube-api-access-l7jv4\") pod \"maas-api-revoke-keys-e2e-ait-226ecbcb-jmv2f\" (UID: \"06b3b567-76cd-437c-a3b4-1324adad5279\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-226ecbcb-jmv2f"
Oct 30 18:47:58.134768 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:58.134740    4525 generic.go:356] "Generic (PLEG): container finished" podID="f2f88b91-9a74-4024-9218-9e35482e8d8d" containerID="cc675924e87bef2fafee0a17ec9becf29ee2222b946d72ac6784116c1224accd" exitCode=0
Oct 30 18:47:58.135062 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:58.134775    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6156d5d6-7rts5" event={"ID":"f2f88b91-9a74-4024-9218-9e35482e8d8d","Type":"ContainerDied","Data":"cc675924e87bef2fafee0a17ec9becf29ee2222b946d72ac6784116c1224accd"}
Oct 30 18:47:58.205958 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:58.205927    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e4242163-rfw7b"]
Oct 30 18:47:58.209951 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:58.209894    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e4242163-rfw7b"
Oct 30 18:47:58.219596 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:58.218921    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e4242163-rfw7b"]
Oct 30 18:47:58.306453 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:58.306414    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6976fa8c-d768-45ff-854a-e519843fda8b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-e4242163-rfw7b\" (UID: \"6976fa8c-d768-45ff-854a-e519843fda8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e4242163-rfw7b"
Oct 30 18:47:58.306661 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:58.306461    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfwjf\" (UniqueName: \"kubernetes.io/projected/6976fa8c-d768-45ff-854a-e519843fda8b-kube-api-access-pfwjf\") pod \"maas-api-revoke-keys-e2e-ait-e4242163-rfw7b\" (UID: \"6976fa8c-d768-45ff-854a-e519843fda8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e4242163-rfw7b"
Oct 30 18:47:58.306661 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:58.306489    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6976fa8c-d768-45ff-854a-e519843fda8b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-e4242163-rfw7b\" (UID: \"6976fa8c-d768-45ff-854a-e519843fda8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e4242163-rfw7b"
Oct 30 18:47:58.307130 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:58.307108    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-226ecbcb-jmv2f"
Oct 30 18:47:58.407533 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:58.407498    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6976fa8c-d768-45ff-854a-e519843fda8b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-e4242163-rfw7b\" (UID: \"6976fa8c-d768-45ff-854a-e519843fda8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e4242163-rfw7b"
Oct 30 18:47:58.407727 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:58.407544    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfwjf\" (UniqueName: \"kubernetes.io/projected/6976fa8c-d768-45ff-854a-e519843fda8b-kube-api-access-pfwjf\") pod \"maas-api-revoke-keys-e2e-ait-e4242163-rfw7b\" (UID: \"6976fa8c-d768-45ff-854a-e519843fda8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e4242163-rfw7b"
Oct 30 18:47:58.407727 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:58.407574    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6976fa8c-d768-45ff-854a-e519843fda8b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-e4242163-rfw7b\" (UID: \"6976fa8c-d768-45ff-854a-e519843fda8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e4242163-rfw7b"
Oct 30 18:47:58.408560 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:58.408539    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6976fa8c-d768-45ff-854a-e519843fda8b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-e4242163-rfw7b\" (UID: \"6976fa8c-d768-45ff-854a-e519843fda8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e4242163-rfw7b"
Oct 30 18:47:58.410204 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:58.410179    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6976fa8c-d768-45ff-854a-e519843fda8b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-e4242163-rfw7b\" (UID: \"6976fa8c-d768-45ff-854a-e519843fda8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e4242163-rfw7b"
Oct 30 18:47:58.416661 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:58.416636    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfwjf\" (UniqueName: \"kubernetes.io/projected/6976fa8c-d768-45ff-854a-e519843fda8b-kube-api-access-pfwjf\") pod \"maas-api-revoke-keys-e2e-ait-e4242163-rfw7b\" (UID: \"6976fa8c-d768-45ff-854a-e519843fda8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e4242163-rfw7b"
Oct 30 18:47:58.527069 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:58.526991    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e4242163-rfw7b"
Oct 30 18:47:58.606670 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:58.606629    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-226ecbcb-jmv2f"]
Oct 30 18:47:58.613181 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:47:58.613141    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06b3b567_76cd_437c_a3b4_1324adad5279.slice/crio-96fb295e41c0941e22583cfdd2e4f620b49f6fe509e0bd39086c8bbcbfcd6a3f WatchSource:0}: Error finding container 96fb295e41c0941e22583cfdd2e4f620b49f6fe509e0bd39086c8bbcbfcd6a3f: Status 404 returned error can't find the container with id 96fb295e41c0941e22583cfdd2e4f620b49f6fe509e0bd39086c8bbcbfcd6a3f
Oct 30 18:47:58.831857 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:58.831831    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e4242163-rfw7b"]
Oct 30 18:47:58.833015 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:47:58.832984    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6976fa8c_d768_45ff_854a_e519843fda8b.slice/crio-27efa951292a8eb086dc55ea5a7d51009414e5817b048212bce4a2ac9d4d76ed WatchSource:0}: Error finding container 27efa951292a8eb086dc55ea5a7d51009414e5817b048212bce4a2ac9d4d76ed: Status 404 returned error can't find the container with id 27efa951292a8eb086dc55ea5a7d51009414e5817b048212bce4a2ac9d4d76ed
Oct 30 18:47:59.142464 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:59.142435    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-226ecbcb-jmv2f" event={"ID":"06b3b567-76cd-437c-a3b4-1324adad5279","Type":"ContainerStarted","Data":"2c04f801a091f0141881686961cda69199a8ea9a01154e89c4a194e15920a0a9"}
Oct 30 18:47:59.142881 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:59.142496    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-226ecbcb-jmv2f" event={"ID":"06b3b567-76cd-437c-a3b4-1324adad5279","Type":"ContainerStarted","Data":"96fb295e41c0941e22583cfdd2e4f620b49f6fe509e0bd39086c8bbcbfcd6a3f"}
Oct 30 18:47:59.144107 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:59.144084    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e4242163-rfw7b" event={"ID":"6976fa8c-d768-45ff-854a-e519843fda8b","Type":"ContainerStarted","Data":"d921bb8aaa58a2522595dffdcf4092ccab2f6a300401eb6d0e698104683da4c3"}
Oct 30 18:47:59.144212 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:59.144129    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e4242163-rfw7b" event={"ID":"6976fa8c-d768-45ff-854a-e519843fda8b","Type":"ContainerStarted","Data":"27efa951292a8eb086dc55ea5a7d51009414e5817b048212bce4a2ac9d4d76ed"}
Oct 30 18:47:59.159445 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:59.159394    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-226ecbcb-jmv2f" podStartSLOduration=2.159375784 podStartE2EDuration="2.159375784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:47:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:59.157882389 +0000 UTC m=+701.783552351" watchObservedRunningTime="2026-10-30 18:47:59.159375784 +0000 UTC m=+701.785045750"
Oct 30 18:47:59.192706 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:59.176601    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e4242163-rfw7b" podStartSLOduration=1.176583092 podStartE2EDuration="1.176583092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:47:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:59.174594815 +0000 UTC m=+701.800264781" watchObservedRunningTime="2026-10-30 18:47:59.176583092 +0000 UTC m=+701.802253056"
Oct 30 18:47:59.315875 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:59.315850    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6156d5d6-7rts5"
Oct 30 18:47:59.355022 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:59.354936    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ee87bf29-9lp94"]
Oct 30 18:47:59.363226 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:59.363186    4525 cpu_manager.go:414] "containerMap: removing container" podUID="f2f88b91-9a74-4024-9218-9e35482e8d8d" containerName="revoke-keys"
Oct 30 18:47:59.363226 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:59.363224    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2f88b91-9a74-4024-9218-9e35482e8d8d" containerName="revoke-keys"
Oct 30 18:47:59.363428 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:59.363364    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2f88b91-9a74-4024-9218-9e35482e8d8d" containerName="revoke-keys"
Oct 30 18:47:59.367233 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:59.367208    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ee87bf29-9lp94"
Oct 30 18:47:59.370410 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:59.370386    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ee87bf29-9lp94"]
Oct 30 18:47:59.417582 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:59.417552    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2f88b91-9a74-4024-9218-9e35482e8d8d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2f88b91-9a74-4024-9218-9e35482e8d8d-maas-api-openshift-ca\") pod \"f2f88b91-9a74-4024-9218-9e35482e8d8d\" (UID: \"f2f88b91-9a74-4024-9218-9e35482e8d8d\") "
Oct 30 18:47:59.417754 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:59.417601    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2f88b91-9a74-4024-9218-9e35482e8d8d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2f88b91-9a74-4024-9218-9e35482e8d8d-maas-api-xks-ca\") pod \"f2f88b91-9a74-4024-9218-9e35482e8d8d\" (UID: \"f2f88b91-9a74-4024-9218-9e35482e8d8d\") "
Oct 30 18:47:59.417754 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:59.417644    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2f88b91-9a74-4024-9218-9e35482e8d8d-kube-api-access-sr5sx\" (UniqueName: \"kubernetes.io/projected/f2f88b91-9a74-4024-9218-9e35482e8d8d-kube-api-access-sr5sx\") pod \"f2f88b91-9a74-4024-9218-9e35482e8d8d\" (UID: \"f2f88b91-9a74-4024-9218-9e35482e8d8d\") "
Oct 30 18:47:59.417880 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:59.417863    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpg6c\" (UniqueName: \"kubernetes.io/projected/cb7bfd18-df49-48ab-8e37-3819f19248af-kube-api-access-tpg6c\") pod \"maas-api-revoke-keys-e2e-ait-ee87bf29-9lp94\" (UID: \"cb7bfd18-df49-48ab-8e37-3819f19248af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ee87bf29-9lp94"
Oct 30 18:47:59.417945 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:59.417896    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2f88b91-9a74-4024-9218-9e35482e8d8d-maas-api-openshift-ca" pod "f2f88b91-9a74-4024-9218-9e35482e8d8d" (UID: "f2f88b91-9a74-4024-9218-9e35482e8d8d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:47:59.418013 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:59.417989    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb7bfd18-df49-48ab-8e37-3819f19248af-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ee87bf29-9lp94\" (UID: \"cb7bfd18-df49-48ab-8e37-3819f19248af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ee87bf29-9lp94"
Oct 30 18:47:59.418047 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:59.418032    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb7bfd18-df49-48ab-8e37-3819f19248af-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ee87bf29-9lp94\" (UID: \"cb7bfd18-df49-48ab-8e37-3819f19248af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ee87bf29-9lp94"
Oct 30 18:47:59.418140 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:59.418110    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2f88b91-9a74-4024-9218-9e35482e8d8d-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:47:59.419969 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:59.419948    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2f88b91-9a74-4024-9218-9e35482e8d8d-maas-api-xks-ca" pod "f2f88b91-9a74-4024-9218-9e35482e8d8d" (UID: "f2f88b91-9a74-4024-9218-9e35482e8d8d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:47:59.420076 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:59.420054    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2f88b91-9a74-4024-9218-9e35482e8d8d-kube-api-access-sr5sx" pod "f2f88b91-9a74-4024-9218-9e35482e8d8d" (UID: "f2f88b91-9a74-4024-9218-9e35482e8d8d"). InnerVolumeSpecName "kube-api-access-sr5sx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:47:59.518859 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:59.518817    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpg6c\" (UniqueName: \"kubernetes.io/projected/cb7bfd18-df49-48ab-8e37-3819f19248af-kube-api-access-tpg6c\") pod \"maas-api-revoke-keys-e2e-ait-ee87bf29-9lp94\" (UID: \"cb7bfd18-df49-48ab-8e37-3819f19248af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ee87bf29-9lp94"
Oct 30 18:47:59.519058 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:59.518921    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb7bfd18-df49-48ab-8e37-3819f19248af-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ee87bf29-9lp94\" (UID: \"cb7bfd18-df49-48ab-8e37-3819f19248af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ee87bf29-9lp94"
Oct 30 18:47:59.519058 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:59.518956    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb7bfd18-df49-48ab-8e37-3819f19248af-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ee87bf29-9lp94\" (UID: \"cb7bfd18-df49-48ab-8e37-3819f19248af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ee87bf29-9lp94"
Oct 30 18:47:59.519058 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:59.519038    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2f88b91-9a74-4024-9218-9e35482e8d8d-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:47:59.519058 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:59.519054    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sr5sx\" (UniqueName: \"kubernetes.io/projected/f2f88b91-9a74-4024-9218-9e35482e8d8d-kube-api-access-sr5sx\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:47:59.519577 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:59.519556    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb7bfd18-df49-48ab-8e37-3819f19248af-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ee87bf29-9lp94\" (UID: \"cb7bfd18-df49-48ab-8e37-3819f19248af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ee87bf29-9lp94"
Oct 30 18:47:59.521447 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:59.521426    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb7bfd18-df49-48ab-8e37-3819f19248af-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ee87bf29-9lp94\" (UID: \"cb7bfd18-df49-48ab-8e37-3819f19248af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ee87bf29-9lp94"
Oct 30 18:47:59.526322 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:59.526305    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpg6c\" (UniqueName: \"kubernetes.io/projected/cb7bfd18-df49-48ab-8e37-3819f19248af-kube-api-access-tpg6c\") pod \"maas-api-revoke-keys-e2e-ait-ee87bf29-9lp94\" (UID: \"cb7bfd18-df49-48ab-8e37-3819f19248af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ee87bf29-9lp94"
Oct 30 18:47:59.710241 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:59.710213    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ee87bf29-9lp94"
Oct 30 18:47:59.985860 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:47:59.985825    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ee87bf29-9lp94"]
Oct 30 18:47:59.987112 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:47:59.987083    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb7bfd18_df49_48ab_8e37_3819f19248af.slice/crio-699736a20cb367369396f81ff0002dfbf1bb86c8140495497b8fb3c12956194c WatchSource:0}: Error finding container 699736a20cb367369396f81ff0002dfbf1bb86c8140495497b8fb3c12956194c: Status 404 returned error can't find the container with id 699736a20cb367369396f81ff0002dfbf1bb86c8140495497b8fb3c12956194c
Oct 30 18:48:00.152060 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:00.152033    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6156d5d6-7rts5"
Oct 30 18:48:00.152422 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:00.152028    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6156d5d6-7rts5" event={"ID":"f2f88b91-9a74-4024-9218-9e35482e8d8d","Type":"ContainerDied","Data":"244ebe1732037a14b218fb08d772a7e84663d1f39275ca54f0fc500516900f0c"}
Oct 30 18:48:00.152422 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:00.152151    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="244ebe1732037a14b218fb08d772a7e84663d1f39275ca54f0fc500516900f0c"
Oct 30 18:48:00.154439 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:00.154411    4525 generic.go:356] "Generic (PLEG): container finished" podID="06b3b567-76cd-437c-a3b4-1324adad5279" containerID="2c04f801a091f0141881686961cda69199a8ea9a01154e89c4a194e15920a0a9" exitCode=0
Oct 30 18:48:00.154579 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:00.154536    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-226ecbcb-jmv2f" event={"ID":"06b3b567-76cd-437c-a3b4-1324adad5279","Type":"ContainerDied","Data":"2c04f801a091f0141881686961cda69199a8ea9a01154e89c4a194e15920a0a9"}
Oct 30 18:48:00.156248 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:00.156228    4525 generic.go:356] "Generic (PLEG): container finished" podID="6976fa8c-d768-45ff-854a-e519843fda8b" containerID="d921bb8aaa58a2522595dffdcf4092ccab2f6a300401eb6d0e698104683da4c3" exitCode=0
Oct 30 18:48:00.156341 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:00.156307    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e4242163-rfw7b" event={"ID":"6976fa8c-d768-45ff-854a-e519843fda8b","Type":"ContainerDied","Data":"d921bb8aaa58a2522595dffdcf4092ccab2f6a300401eb6d0e698104683da4c3"}
Oct 30 18:48:00.157573 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:00.157552    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ee87bf29-9lp94" event={"ID":"cb7bfd18-df49-48ab-8e37-3819f19248af","Type":"ContainerStarted","Data":"699736a20cb367369396f81ff0002dfbf1bb86c8140495497b8fb3c12956194c"}
Oct 30 18:48:01.064902 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:01.064812    4525 patch_prober.go:29] interesting pod/e2e-derive-f951dfa9-openshift-default-5ddd96b597-dzjdp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 18:48:01.064902 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:01.064871    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-f951dfa9-openshift-default-5ddd96b597-dzjdp" podUID="a29fd997-17ba-4260-8279-05e6f1c6346a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:01.165939 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:01.165897    4525 generic.go:356] "Generic (PLEG): container finished" podID="cb7bfd18-df49-48ab-8e37-3819f19248af" containerID="d72b48501abc705a2b29750c7a1303f8088da99fc26b73504e2ed20d0613b25e" exitCode=0
Oct 30 18:48:01.166421 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:01.166039    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ee87bf29-9lp94" event={"ID":"cb7bfd18-df49-48ab-8e37-3819f19248af","Type":"ContainerDied","Data":"d72b48501abc705a2b29750c7a1303f8088da99fc26b73504e2ed20d0613b25e"}
Oct 30 18:48:01.324710 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:01.324565    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e4242163-rfw7b"
Oct 30 18:48:01.366155 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:01.366108    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-226ecbcb-jmv2f"
Oct 30 18:48:01.439398 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:01.439360    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06b3b567-76cd-437c-a3b4-1324adad5279-kube-api-access-l7jv4\" (UniqueName: \"kubernetes.io/projected/06b3b567-76cd-437c-a3b4-1324adad5279-kube-api-access-l7jv4\") pod \"06b3b567-76cd-437c-a3b4-1324adad5279\" (UID: \"06b3b567-76cd-437c-a3b4-1324adad5279\") "
Oct 30 18:48:01.439593 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:01.439415    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6976fa8c-d768-45ff-854a-e519843fda8b-kube-api-access-pfwjf\" (UniqueName: \"kubernetes.io/projected/6976fa8c-d768-45ff-854a-e519843fda8b-kube-api-access-pfwjf\") pod \"6976fa8c-d768-45ff-854a-e519843fda8b\" (UID: \"6976fa8c-d768-45ff-854a-e519843fda8b\") "
Oct 30 18:48:01.439593 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:01.439467    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6976fa8c-d768-45ff-854a-e519843fda8b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6976fa8c-d768-45ff-854a-e519843fda8b-maas-api-xks-ca\") pod \"6976fa8c-d768-45ff-854a-e519843fda8b\" (UID: \"6976fa8c-d768-45ff-854a-e519843fda8b\") "
Oct 30 18:48:01.439593 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:01.439487    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6976fa8c-d768-45ff-854a-e519843fda8b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6976fa8c-d768-45ff-854a-e519843fda8b-maas-api-openshift-ca\") pod \"6976fa8c-d768-45ff-854a-e519843fda8b\" (UID: \"6976fa8c-d768-45ff-854a-e519843fda8b\") "
Oct 30 18:48:01.439593 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:01.439556    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06b3b567-76cd-437c-a3b4-1324adad5279-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06b3b567-76cd-437c-a3b4-1324adad5279-maas-api-xks-ca\") pod \"06b3b567-76cd-437c-a3b4-1324adad5279\" (UID: \"06b3b567-76cd-437c-a3b4-1324adad5279\") "
Oct 30 18:48:01.439890 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:01.439605    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06b3b567-76cd-437c-a3b4-1324adad5279-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06b3b567-76cd-437c-a3b4-1324adad5279-maas-api-openshift-ca\") pod \"06b3b567-76cd-437c-a3b4-1324adad5279\" (UID: \"06b3b567-76cd-437c-a3b4-1324adad5279\") "
Oct 30 18:48:01.440050 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:01.439970    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6976fa8c-d768-45ff-854a-e519843fda8b-maas-api-openshift-ca" pod "6976fa8c-d768-45ff-854a-e519843fda8b" (UID: "6976fa8c-d768-45ff-854a-e519843fda8b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:48:01.440342 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:01.440131    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06b3b567-76cd-437c-a3b4-1324adad5279-maas-api-openshift-ca" pod "06b3b567-76cd-437c-a3b4-1324adad5279" (UID: "06b3b567-76cd-437c-a3b4-1324adad5279"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:48:01.441805 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:01.441772    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6976fa8c-d768-45ff-854a-e519843fda8b-kube-api-access-pfwjf" pod "6976fa8c-d768-45ff-854a-e519843fda8b" (UID: "6976fa8c-d768-45ff-854a-e519843fda8b"). InnerVolumeSpecName "kube-api-access-pfwjf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:48:01.441909 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:01.441807    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6976fa8c-d768-45ff-854a-e519843fda8b-maas-api-xks-ca" pod "6976fa8c-d768-45ff-854a-e519843fda8b" (UID: "6976fa8c-d768-45ff-854a-e519843fda8b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:48:01.442276 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:01.442247    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06b3b567-76cd-437c-a3b4-1324adad5279-maas-api-xks-ca" pod "06b3b567-76cd-437c-a3b4-1324adad5279" (UID: "06b3b567-76cd-437c-a3b4-1324adad5279"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:48:01.442366 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:01.442345    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06b3b567-76cd-437c-a3b4-1324adad5279-kube-api-access-l7jv4" pod "06b3b567-76cd-437c-a3b4-1324adad5279" (UID: "06b3b567-76cd-437c-a3b4-1324adad5279"). InnerVolumeSpecName "kube-api-access-l7jv4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:48:01.540481 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:01.540450    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06b3b567-76cd-437c-a3b4-1324adad5279-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:48:01.540481 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:01.540476    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7jv4\" (UniqueName: \"kubernetes.io/projected/06b3b567-76cd-437c-a3b4-1324adad5279-kube-api-access-l7jv4\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:48:01.540481 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:01.540485    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfwjf\" (UniqueName: \"kubernetes.io/projected/6976fa8c-d768-45ff-854a-e519843fda8b-kube-api-access-pfwjf\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:48:01.540694 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:01.540495    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6976fa8c-d768-45ff-854a-e519843fda8b-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:48:01.540694 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:01.540504    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6976fa8c-d768-45ff-854a-e519843fda8b-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:48:01.540694 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:01.540512    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06b3b567-76cd-437c-a3b4-1324adad5279-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:48:02.173025 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:02.172987    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-226ecbcb-jmv2f"
Oct 30 18:48:02.173440 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:02.172992    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-226ecbcb-jmv2f" event={"ID":"06b3b567-76cd-437c-a3b4-1324adad5279","Type":"ContainerDied","Data":"96fb295e41c0941e22583cfdd2e4f620b49f6fe509e0bd39086c8bbcbfcd6a3f"}
Oct 30 18:48:02.173440 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:02.173120    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96fb295e41c0941e22583cfdd2e4f620b49f6fe509e0bd39086c8bbcbfcd6a3f"
Oct 30 18:48:02.174399 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:02.174367    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e4242163-rfw7b" event={"ID":"6976fa8c-d768-45ff-854a-e519843fda8b","Type":"ContainerDied","Data":"27efa951292a8eb086dc55ea5a7d51009414e5817b048212bce4a2ac9d4d76ed"}
Oct 30 18:48:02.174499 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:02.174419    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="27efa951292a8eb086dc55ea5a7d51009414e5817b048212bce4a2ac9d4d76ed"
Oct 30 18:48:02.174562 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:02.174545    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e4242163-rfw7b"
Oct 30 18:48:02.316621 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:02.316590    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ee87bf29-9lp94"
Oct 30 18:48:02.348019 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:02.347991    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb7bfd18-df49-48ab-8e37-3819f19248af-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb7bfd18-df49-48ab-8e37-3819f19248af-maas-api-openshift-ca\") pod \"cb7bfd18-df49-48ab-8e37-3819f19248af\" (UID: \"cb7bfd18-df49-48ab-8e37-3819f19248af\") "
Oct 30 18:48:02.348189 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:02.348047    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb7bfd18-df49-48ab-8e37-3819f19248af-kube-api-access-tpg6c\" (UniqueName: \"kubernetes.io/projected/cb7bfd18-df49-48ab-8e37-3819f19248af-kube-api-access-tpg6c\") pod \"cb7bfd18-df49-48ab-8e37-3819f19248af\" (UID: \"cb7bfd18-df49-48ab-8e37-3819f19248af\") "
Oct 30 18:48:02.348189 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:02.348170    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb7bfd18-df49-48ab-8e37-3819f19248af-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb7bfd18-df49-48ab-8e37-3819f19248af-maas-api-xks-ca\") pod \"cb7bfd18-df49-48ab-8e37-3819f19248af\" (UID: \"cb7bfd18-df49-48ab-8e37-3819f19248af\") "
Oct 30 18:48:02.348404 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:02.348379    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb7bfd18-df49-48ab-8e37-3819f19248af-maas-api-openshift-ca" pod "cb7bfd18-df49-48ab-8e37-3819f19248af" (UID: "cb7bfd18-df49-48ab-8e37-3819f19248af"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:48:02.350546 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:02.350506    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb7bfd18-df49-48ab-8e37-3819f19248af-maas-api-xks-ca" pod "cb7bfd18-df49-48ab-8e37-3819f19248af" (UID: "cb7bfd18-df49-48ab-8e37-3819f19248af"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:48:02.350673 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:02.350634    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb7bfd18-df49-48ab-8e37-3819f19248af-kube-api-access-tpg6c" pod "cb7bfd18-df49-48ab-8e37-3819f19248af" (UID: "cb7bfd18-df49-48ab-8e37-3819f19248af"). InnerVolumeSpecName "kube-api-access-tpg6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:48:02.449664 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:02.449583    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb7bfd18-df49-48ab-8e37-3819f19248af-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:48:02.449664 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:02.449612    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb7bfd18-df49-48ab-8e37-3819f19248af-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:48:02.449664 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:02.449622    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpg6c\" (UniqueName: \"kubernetes.io/projected/cb7bfd18-df49-48ab-8e37-3819f19248af-kube-api-access-tpg6c\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:48:02.523895 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:02.523859    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-6156d5d6-55fcbb8767-4w5g4"]
Oct 30 18:48:03.182013 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:03.181995    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ee87bf29-9lp94"
Oct 30 18:48:03.182367 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:03.182028    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ee87bf29-9lp94" event={"ID":"cb7bfd18-df49-48ab-8e37-3819f19248af","Type":"ContainerDied","Data":"699736a20cb367369396f81ff0002dfbf1bb86c8140495497b8fb3c12956194c"}
Oct 30 18:48:03.182367 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:03.182085    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="699736a20cb367369396f81ff0002dfbf1bb86c8140495497b8fb3c12956194c"
Oct 30 18:48:04.189453 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:04.189417    4525 generic.go:356] "Generic (PLEG): container finished" podID="a29fd997-17ba-4260-8279-05e6f1c6346a" containerID="004fe667a937ac9a8bc60137eeedd2bcd7407bd1ca8df0fd781dbc0ebad252b2" exitCode=0
Oct 30 18:48:04.189931 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:04.189492    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f951dfa9-openshift-default-5ddd96b597-dzjdp" event={"ID":"a29fd997-17ba-4260-8279-05e6f1c6346a","Type":"ContainerDied","Data":"004fe667a937ac9a8bc60137eeedd2bcd7407bd1ca8df0fd781dbc0ebad252b2"}
Oct 30 18:48:04.189931 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:04.189867    4525 scope.go:122] "RemoveContainer" containerID="004fe667a937ac9a8bc60137eeedd2bcd7407bd1ca8df0fd781dbc0ebad252b2"
Oct 30 18:48:04.191165 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:04.191140    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:48:04.191234 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:04.191208    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:48:04.191273 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:04.191240    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:48:05.208097 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:05.208060    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f951dfa9-openshift-default-5ddd96b597-dzjdp" event={"ID":"a29fd997-17ba-4260-8279-05e6f1c6346a","Type":"ContainerStarted","Data":"0734ce8a9028d0129512de7d641ebd9b420d93a54ab46128b44dd2b21c2b867f"}
Oct 30 18:48:05.298705 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:05.297817    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-f951dfa9-openshift-default-5ddd96b597-dzjdp"
Oct 30 18:48:05.298705 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:05.297955    4525 patch_prober.go:29] interesting pod/e2e-derive-f951dfa9-openshift-default-5ddd96b597-dzjdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 30 18:48:05.298705 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:05.298004    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-f951dfa9-openshift-default-5ddd96b597-dzjdp" podUID="a29fd997-17ba-4260-8279-05e6f1c6346a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 30 18:48:06.045604 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:06.045574    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6156d5d6-7rts5"]
Oct 30 18:48:06.050666 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:06.050640    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6156d5d6-7rts5"]
Oct 30 18:48:06.284716 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:06.284092    4525 patch_prober.go:29] interesting pod/e2e-derive-f951dfa9-openshift-default-5ddd96b597-dzjdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 30 18:48:06.284716 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:06.284166    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-f951dfa9-openshift-default-5ddd96b597-dzjdp" podUID="a29fd997-17ba-4260-8279-05e6f1c6346a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 30 18:48:07.288708 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:07.285129    4525 patch_prober.go:29] interesting pod/e2e-derive-f951dfa9-openshift-default-5ddd96b597-dzjdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 30 18:48:07.288708 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:07.285200    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-f951dfa9-openshift-default-5ddd96b597-dzjdp" podUID="a29fd997-17ba-4260-8279-05e6f1c6346a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 30 18:48:07.422169 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:07.422136    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j"]
Oct 30 18:48:07.422656 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:07.422627    4525 cpu_manager.go:414] "containerMap: removing container" podUID="cb7bfd18-df49-48ab-8e37-3819f19248af" containerName="revoke-keys"
Oct 30 18:48:07.422656 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:07.422648    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb7bfd18-df49-48ab-8e37-3819f19248af" containerName="revoke-keys"
Oct 30 18:48:07.422830 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:07.422706    4525 cpu_manager.go:414] "containerMap: removing container" podUID="06b3b567-76cd-437c-a3b4-1324adad5279" containerName="revoke-keys"
Oct 30 18:48:07.422830 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:07.422715    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06b3b567-76cd-437c-a3b4-1324adad5279" containerName="revoke-keys"
Oct 30 18:48:07.422830 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:07.422735    4525 cpu_manager.go:414] "containerMap: removing container" podUID="6976fa8c-d768-45ff-854a-e519843fda8b" containerName="revoke-keys"
Oct 30 18:48:07.422830 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:07.422742    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6976fa8c-d768-45ff-854a-e519843fda8b" containerName="revoke-keys"
Oct 30 18:48:07.423093 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:07.422838    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb7bfd18-df49-48ab-8e37-3819f19248af" containerName="revoke-keys"
Oct 30 18:48:07.423093 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:07.422856    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="06b3b567-76cd-437c-a3b4-1324adad5279" containerName="revoke-keys"
Oct 30 18:48:07.423093 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:07.422869    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="6976fa8c-d768-45ff-854a-e519843fda8b" containerName="revoke-keys"
Oct 30 18:48:07.426930 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:07.426864    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j"
Oct 30 18:48:07.438872 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:07.438839    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-2889dd-openshift-default-dockercfg-qgr7b\""
Oct 30 18:48:07.443217 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:07.443184    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j"]
Oct 30 18:48:07.524779 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:07.524738    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6156d5d6-55fcbb8767-4w5g4" podUID="1cec3994-12ce-4931-9d31-1b620974dd99" containerName="payload-pre-processing" containerID="cri-o://13b0f20ad0b600e075628047ec06ebd902e3dda65123b9adfe77e5394c00f429" gracePeriod=25
Oct 30 18:48:07.528624 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:07.528041    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6061ede7-981c-4c07-acfc-04be1cae252d-credential-socket\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j\" (UID: \"6061ede7-981c-4c07-acfc-04be1cae252d\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j"
Oct 30 18:48:07.528624 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:07.528093    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6061ede7-981c-4c07-acfc-04be1cae252d-istiod-ca-cert\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j\" (UID: \"6061ede7-981c-4c07-acfc-04be1cae252d\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j"
Oct 30 18:48:07.528624 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:07.528125    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6061ede7-981c-4c07-acfc-04be1cae252d-workload-certs\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j\" (UID: \"6061ede7-981c-4c07-acfc-04be1cae252d\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j"
Oct 30 18:48:07.528624 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:07.528164    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2sfg\" (UniqueName: \"kubernetes.io/projected/6061ede7-981c-4c07-acfc-04be1cae252d-kube-api-access-s2sfg\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j\" (UID: \"6061ede7-981c-4c07-acfc-04be1cae252d\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j"
Oct 30 18:48:07.528624 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:07.528214    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6061ede7-981c-4c07-acfc-04be1cae252d-istio-podinfo\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j\" (UID: \"6061ede7-981c-4c07-acfc-04be1cae252d\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j"
Oct 30 18:48:07.528624 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:07.528254    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6061ede7-981c-4c07-acfc-04be1cae252d-workload-socket\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j\" (UID: \"6061ede7-981c-4c07-acfc-04be1cae252d\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j"
Oct 30 18:48:07.528624 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:07.528287    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6061ede7-981c-4c07-acfc-04be1cae252d-istio-data\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j\" (UID: \"6061ede7-981c-4c07-acfc-04be1cae252d\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j"
Oct 30 18:48:07.528624 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:07.528328    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6061ede7-981c-4c07-acfc-04be1cae252d-istio-token\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j\" (UID: \"6061ede7-981c-4c07-acfc-04be1cae252d\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j"
Oct 30 18:48:07.528624 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:07.528378    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6061ede7-981c-4c07-acfc-04be1cae252d-istio-envoy\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j\" (UID: \"6061ede7-981c-4c07-acfc-04be1cae252d\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j"
Oct 30 18:48:07.551564 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:07.551528    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2f88b91-9a74-4024-9218-9e35482e8d8d" path="/var/lib/kubelet/pods/f2f88b91-9a74-4024-9218-9e35482e8d8d/volumes"
Oct 30 18:48:07.629248 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:07.629205    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6061ede7-981c-4c07-acfc-04be1cae252d-credential-socket\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j\" (UID: \"6061ede7-981c-4c07-acfc-04be1cae252d\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j"
Oct 30 18:48:07.629461 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:07.629293    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6061ede7-981c-4c07-acfc-04be1cae252d-istiod-ca-cert\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j\" (UID: \"6061ede7-981c-4c07-acfc-04be1cae252d\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j"
Oct 30 18:48:07.629461 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:07.629333    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6061ede7-981c-4c07-acfc-04be1cae252d-workload-certs\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j\" (UID: \"6061ede7-981c-4c07-acfc-04be1cae252d\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j"
Oct 30 18:48:07.629461 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:07.629387    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2sfg\" (UniqueName: \"kubernetes.io/projected/6061ede7-981c-4c07-acfc-04be1cae252d-kube-api-access-s2sfg\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j\" (UID: \"6061ede7-981c-4c07-acfc-04be1cae252d\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j"
Oct 30 18:48:07.629626 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:07.629474    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6061ede7-981c-4c07-acfc-04be1cae252d-istio-podinfo\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j\" (UID: \"6061ede7-981c-4c07-acfc-04be1cae252d\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j"
Oct 30 18:48:07.629626 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:07.629530    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6061ede7-981c-4c07-acfc-04be1cae252d-workload-socket\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j\" (UID: \"6061ede7-981c-4c07-acfc-04be1cae252d\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j"
Oct 30 18:48:07.629626 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:07.629565    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6061ede7-981c-4c07-acfc-04be1cae252d-istio-data\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j\" (UID: \"6061ede7-981c-4c07-acfc-04be1cae252d\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j"
Oct 30 18:48:07.629626 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:07.629620    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6061ede7-981c-4c07-acfc-04be1cae252d-istio-token\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j\" (UID: \"6061ede7-981c-4c07-acfc-04be1cae252d\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j"
Oct 30 18:48:07.630013 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:07.629640    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6061ede7-981c-4c07-acfc-04be1cae252d-credential-socket\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j\" (UID: \"6061ede7-981c-4c07-acfc-04be1cae252d\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j"
Oct 30 18:48:07.630013 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:07.629667    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6061ede7-981c-4c07-acfc-04be1cae252d-istio-envoy\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j\" (UID: \"6061ede7-981c-4c07-acfc-04be1cae252d\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j"
Oct 30 18:48:07.630013 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:07.629670    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6061ede7-981c-4c07-acfc-04be1cae252d-workload-certs\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j\" (UID: \"6061ede7-981c-4c07-acfc-04be1cae252d\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j"
Oct 30 18:48:07.630013 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:07.629919    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6061ede7-981c-4c07-acfc-04be1cae252d-workload-socket\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j\" (UID: \"6061ede7-981c-4c07-acfc-04be1cae252d\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j"
Oct 30 18:48:07.630252 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:07.630039    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6061ede7-981c-4c07-acfc-04be1cae252d-istiod-ca-cert\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j\" (UID: \"6061ede7-981c-4c07-acfc-04be1cae252d\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j"
Oct 30 18:48:07.630252 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:07.630071    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6061ede7-981c-4c07-acfc-04be1cae252d-istio-data\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j\" (UID: \"6061ede7-981c-4c07-acfc-04be1cae252d\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j"
Oct 30 18:48:07.631987 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:07.631966    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6061ede7-981c-4c07-acfc-04be1cae252d-istio-envoy\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j\" (UID: \"6061ede7-981c-4c07-acfc-04be1cae252d\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j"
Oct 30 18:48:07.632566 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:07.632544    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6061ede7-981c-4c07-acfc-04be1cae252d-istio-podinfo\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j\" (UID: \"6061ede7-981c-4c07-acfc-04be1cae252d\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j"
Oct 30 18:48:07.637751 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:07.637730    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6061ede7-981c-4c07-acfc-04be1cae252d-istio-token\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j\" (UID: \"6061ede7-981c-4c07-acfc-04be1cae252d\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j"
Oct 30 18:48:07.638012 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:07.637992    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2sfg\" (UniqueName: \"kubernetes.io/projected/6061ede7-981c-4c07-acfc-04be1cae252d-kube-api-access-s2sfg\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j\" (UID: \"6061ede7-981c-4c07-acfc-04be1cae252d\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j"
Oct 30 18:48:07.740209 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:07.740161    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j"
Oct 30 18:48:07.825871 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:07.824216    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6156d5d6-55fcbb8767-4w5g4"
Oct 30 18:48:07.937548 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:07.934108    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cec3994-12ce-4931-9d31-1b620974dd99-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cec3994-12ce-4931-9d31-1b620974dd99-plugins-config-volume\") pod \"1cec3994-12ce-4931-9d31-1b620974dd99\" (UID: \"1cec3994-12ce-4931-9d31-1b620974dd99\") "
Oct 30 18:48:07.937548 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:07.934226    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cec3994-12ce-4931-9d31-1b620974dd99-kube-api-access-rvmck\" (UniqueName: \"kubernetes.io/projected/1cec3994-12ce-4931-9d31-1b620974dd99-kube-api-access-rvmck\") pod \"1cec3994-12ce-4931-9d31-1b620974dd99\" (UID: \"1cec3994-12ce-4931-9d31-1b620974dd99\") "
Oct 30 18:48:07.937548 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:07.934605    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cec3994-12ce-4931-9d31-1b620974dd99-plugins-config-volume" pod "1cec3994-12ce-4931-9d31-1b620974dd99" (UID: "1cec3994-12ce-4931-9d31-1b620974dd99"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:48:07.938137 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:07.938086    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cec3994-12ce-4931-9d31-1b620974dd99-kube-api-access-rvmck" pod "1cec3994-12ce-4931-9d31-1b620974dd99" (UID: "1cec3994-12ce-4931-9d31-1b620974dd99"). InnerVolumeSpecName "kube-api-access-rvmck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:48:08.035700 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:08.035654    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvmck\" (UniqueName: \"kubernetes.io/projected/1cec3994-12ce-4931-9d31-1b620974dd99-kube-api-access-rvmck\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:48:08.035700 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:08.035698    4525 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cec3994-12ce-4931-9d31-1b620974dd99-plugins-config-volume\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:48:08.060936 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:08.060907    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j"]
Oct 30 18:48:08.062365 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:48:08.062335    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6061ede7_981c_4c07_acfc_04be1cae252d.slice/crio-a65c38e22d7c9cfe58089211935c7902a4414e3bfeb160f389b4042d9c3cc9cd WatchSource:0}: Error finding container a65c38e22d7c9cfe58089211935c7902a4414e3bfeb160f389b4042d9c3cc9cd: Status 404 returned error can't find the container with id a65c38e22d7c9cfe58089211935c7902a4414e3bfeb160f389b4042d9c3cc9cd
Oct 30 18:48:08.064540 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:08.064515    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:48:08.064586 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:08.064573    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:48:08.064622 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:08.064601    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:48:08.101987 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:08.101947    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-226ecbcb-5d69cc78dd-rjzsb"]
Oct 30 18:48:08.106880 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:08.106855    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-226ecbcb-69b85788bc-8xszf"]
Oct 30 18:48:08.229284 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:08.229230    4525 generic.go:356] "Generic (PLEG): container finished" podID="1cec3994-12ce-4931-9d31-1b620974dd99" containerID="13b0f20ad0b600e075628047ec06ebd902e3dda65123b9adfe77e5394c00f429" exitCode=0
Oct 30 18:48:08.229493 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:08.229431    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6156d5d6-55fcbb8767-4w5g4" event={"ID":"1cec3994-12ce-4931-9d31-1b620974dd99","Type":"ContainerDied","Data":"13b0f20ad0b600e075628047ec06ebd902e3dda65123b9adfe77e5394c00f429"}
Oct 30 18:48:08.229493 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:08.229486    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6156d5d6-55fcbb8767-4w5g4" event={"ID":"1cec3994-12ce-4931-9d31-1b620974dd99","Type":"ContainerDied","Data":"97eb60d2dac0fd337eda1fe3f0bd8bd0a619f8ab9232a202d3ced7e55266cd42"}
Oct 30 18:48:08.229615 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:08.229514    4525 scope.go:122] "RemoveContainer" containerID="13b0f20ad0b600e075628047ec06ebd902e3dda65123b9adfe77e5394c00f429"
Oct 30 18:48:08.229720 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:08.229659    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6156d5d6-55fcbb8767-4w5g4"
Oct 30 18:48:08.232728 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:08.232197    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j" event={"ID":"6061ede7-981c-4c07-acfc-04be1cae252d","Type":"ContainerStarted","Data":"95be28c0a2e6e7efa2daa699bd3d87456912a01b2d5df468ae82634e02d7a3b7"}
Oct 30 18:48:08.232728 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:08.232248    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j" event={"ID":"6061ede7-981c-4c07-acfc-04be1cae252d","Type":"ContainerStarted","Data":"a65c38e22d7c9cfe58089211935c7902a4414e3bfeb160f389b4042d9c3cc9cd"}
Oct 30 18:48:08.252900 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:08.251219    4525 scope.go:122] "RemoveContainer" containerID="13b0f20ad0b600e075628047ec06ebd902e3dda65123b9adfe77e5394c00f429"
Oct 30 18:48:08.252900 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:48:08.251891    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13b0f20ad0b600e075628047ec06ebd902e3dda65123b9adfe77e5394c00f429\": container with ID starting with 13b0f20ad0b600e075628047ec06ebd902e3dda65123b9adfe77e5394c00f429 not found: ID does not exist" containerID="13b0f20ad0b600e075628047ec06ebd902e3dda65123b9adfe77e5394c00f429"
Oct 30 18:48:08.252900 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:08.251929    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13b0f20ad0b600e075628047ec06ebd902e3dda65123b9adfe77e5394c00f429"} err="failed to get container status \"13b0f20ad0b600e075628047ec06ebd902e3dda65123b9adfe77e5394c00f429\": rpc error: code = NotFound desc = could not find container \"13b0f20ad0b600e075628047ec06ebd902e3dda65123b9adfe77e5394c00f429\": container with ID starting with 13b0f20ad0b600e075628047ec06ebd902e3dda65123b9adfe77e5394c00f429 not found: ID does not exist"
Oct 30 18:48:08.255873 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:08.255810    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j" podStartSLOduration=1.255793963 podStartE2EDuration="1.255793963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:48:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:48:08.254781218 +0000 UTC m=+710.880451175" watchObservedRunningTime="2026-10-30 18:48:08.255793963 +0000 UTC m=+710.881463921"
Oct 30 18:48:08.271965 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:08.271934    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-6156d5d6-55fcbb8767-4w5g4"]
Oct 30 18:48:08.274425 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:08.274401    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-6156d5d6-55fcbb8767-4w5g4"]
Oct 30 18:48:08.303706 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:08.298087    4525 patch_prober.go:29] interesting pod/e2e-derive-f951dfa9-openshift-default-5ddd96b597-dzjdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 30 18:48:08.303706 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:08.298168    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-f951dfa9-openshift-default-5ddd96b597-dzjdp" podUID="a29fd997-17ba-4260-8279-05e6f1c6346a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 30 18:48:08.338896 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:08.338811    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e4242163-6fddb59d56-jvvtf"]
Oct 30 18:48:08.357818 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:08.357782    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e4242163-d74b5477f-fwc2h"]
Oct 30 18:48:08.741335 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:08.741189    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j"
Oct 30 18:48:09.284714 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:09.284101    4525 patch_prober.go:29] interesting pod/e2e-derive-f951dfa9-openshift-default-5ddd96b597-dzjdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 30 18:48:09.284714 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:09.284166    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-f951dfa9-openshift-default-5ddd96b597-dzjdp" podUID="a29fd997-17ba-4260-8279-05e6f1c6346a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 30 18:48:09.509717 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:09.508512    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ee87bf29-5dbc87966c-x9gwt"]
Oct 30 18:48:09.544271 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:09.544197    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cec3994-12ce-4931-9d31-1b620974dd99" path="/var/lib/kubelet/pods/1cec3994-12ce-4931-9d31-1b620974dd99/volumes"
Oct 30 18:48:09.756849 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:09.756810    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j"
Oct 30 18:48:10.247015 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:10.246977    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j"
Oct 30 18:48:10.247970 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:10.247944    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j"
Oct 30 18:48:10.378633 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:10.378604    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-f951dfa9-openshift-default-5ddd96b597-dzjdp"
Oct 30 18:48:10.379626 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:10.379600    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-f951dfa9-openshift-default-5ddd96b597-dzjdp"
Oct 30 18:48:10.384976 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:10.384954    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-f951dfa9-openshift-default-5ddd96b597-dzjdp"
Oct 30 18:48:10.413188 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:10.413156    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-226ecbcb-jmv2f"]
Oct 30 18:48:10.415497 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:10.415476    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-226ecbcb-jmv2f"]
Oct 30 18:48:10.791017 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:10.790910    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e4242163-rfw7b"]
Oct 30 18:48:10.793504 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:10.793482    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e4242163-rfw7b"]
Oct 30 18:48:11.019630 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:11.019593    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ee87bf29-9lp94"]
Oct 30 18:48:11.023802 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:11.023776    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ee87bf29-9lp94"]
Oct 30 18:48:11.553165 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:11.553129    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06b3b567-76cd-437c-a3b4-1324adad5279" path="/var/lib/kubelet/pods/06b3b567-76cd-437c-a3b4-1324adad5279/volumes"
Oct 30 18:48:11.553612 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:11.553596    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6976fa8c-d768-45ff-854a-e519843fda8b" path="/var/lib/kubelet/pods/6976fa8c-d768-45ff-854a-e519843fda8b/volumes"
Oct 30 18:48:11.553991 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:11.553978    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb7bfd18-df49-48ab-8e37-3819f19248af" path="/var/lib/kubelet/pods/cb7bfd18-df49-48ab-8e37-3819f19248af/volumes"
Oct 30 18:48:12.426707 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:12.426014    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-f951dfa9-openshift-default-5ddd96b597-dzjdp"]
Oct 30 18:48:13.103262 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:13.103210    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-226ecbcb-5d69cc78dd-rjzsb" podUID="52ca0496-3987-481c-8efa-28c13e432616" containerName="payload-processing" containerID="cri-o://e1684db67038f8b9d24c3d17848bf2a5c9f86be6d37613a987ff060991029960" gracePeriod=25
Oct 30 18:48:13.107817 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:13.107789    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-226ecbcb-69b85788bc-8xszf" podUID="f6bb3a0f-8779-436c-b404-a26849ff7943" containerName="payload-pre-processing" containerID="cri-o://0376469a7047eec342bef411fb5114e4b1370362bc70d632a6a5e67ad0835c9d" gracePeriod=25
Oct 30 18:48:13.274273 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:13.274070    4525 generic.go:356] "Generic (PLEG): container finished" podID="52ca0496-3987-481c-8efa-28c13e432616" containerID="e1684db67038f8b9d24c3d17848bf2a5c9f86be6d37613a987ff060991029960" exitCode=0
Oct 30 18:48:13.274273 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:13.274177    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-226ecbcb-5d69cc78dd-rjzsb" event={"ID":"52ca0496-3987-481c-8efa-28c13e432616","Type":"ContainerDied","Data":"e1684db67038f8b9d24c3d17848bf2a5c9f86be6d37613a987ff060991029960"}
Oct 30 18:48:13.277083 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:13.276157    4525 generic.go:356] "Generic (PLEG): container finished" podID="f6bb3a0f-8779-436c-b404-a26849ff7943" containerID="0376469a7047eec342bef411fb5114e4b1370362bc70d632a6a5e67ad0835c9d" exitCode=0
Oct 30 18:48:13.277083 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:13.276490    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-f951dfa9-openshift-default-5ddd96b597-dzjdp" podUID="a29fd997-17ba-4260-8279-05e6f1c6346a" containerName="istio-proxy" containerID="cri-o://0734ce8a9028d0129512de7d641ebd9b420d93a54ab46128b44dd2b21c2b867f" gracePeriod=30
Oct 30 18:48:13.277083 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:13.276852    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-226ecbcb-69b85788bc-8xszf" event={"ID":"f6bb3a0f-8779-436c-b404-a26849ff7943","Type":"ContainerDied","Data":"0376469a7047eec342bef411fb5114e4b1370362bc70d632a6a5e67ad0835c9d"}
Oct 30 18:48:13.339936 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:13.339895    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-e4242163-6fddb59d56-jvvtf" podUID="17744dc5-6240-4081-bc2b-768529e1ed13" containerName="payload-processing" containerID="cri-o://b127defd743b7f16bb5fa59d158822e6e3e56056c8c5994168c9d4589bcda1a7" gracePeriod=25
Oct 30 18:48:13.358545 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:13.358502    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-e4242163-d74b5477f-fwc2h" podUID="58554b4f-e880-4ac7-ac7a-d1d4cf33851a" containerName="payload-pre-processing" containerID="cri-o://d665521a08bfe260a306388acef0050fcf79f7d6195288686625d7bb597f0e9e" gracePeriod=25
Oct 30 18:48:13.481693 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:13.481653    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-226ecbcb-5d69cc78dd-rjzsb"
Oct 30 18:48:13.487098 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:13.487074    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-226ecbcb-69b85788bc-8xszf"
Oct 30 18:48:13.602734 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:13.602708    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52ca0496-3987-481c-8efa-28c13e432616-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52ca0496-3987-481c-8efa-28c13e432616-plugins-config-volume\") pod \"52ca0496-3987-481c-8efa-28c13e432616\" (UID: \"52ca0496-3987-481c-8efa-28c13e432616\") "
Oct 30 18:48:13.602887 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:13.602834    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6bb3a0f-8779-436c-b404-a26849ff7943-kube-api-access-zcxh2\" (UniqueName: \"kubernetes.io/projected/f6bb3a0f-8779-436c-b404-a26849ff7943-kube-api-access-zcxh2\") pod \"f6bb3a0f-8779-436c-b404-a26849ff7943\" (UID: \"f6bb3a0f-8779-436c-b404-a26849ff7943\") "
Oct 30 18:48:13.602887 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:13.602868    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52ca0496-3987-481c-8efa-28c13e432616-kube-api-access-p99gf\" (UniqueName: \"kubernetes.io/projected/52ca0496-3987-481c-8efa-28c13e432616-kube-api-access-p99gf\") pod \"52ca0496-3987-481c-8efa-28c13e432616\" (UID: \"52ca0496-3987-481c-8efa-28c13e432616\") "
Oct 30 18:48:13.603002 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:13.602930    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6bb3a0f-8779-436c-b404-a26849ff7943-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6bb3a0f-8779-436c-b404-a26849ff7943-plugins-config-volume\") pod \"f6bb3a0f-8779-436c-b404-a26849ff7943\" (UID: \"f6bb3a0f-8779-436c-b404-a26849ff7943\") "
Oct 30 18:48:13.603178 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:13.603144    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52ca0496-3987-481c-8efa-28c13e432616-plugins-config-volume" pod "52ca0496-3987-481c-8efa-28c13e432616" (UID: "52ca0496-3987-481c-8efa-28c13e432616"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:48:13.603335 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:13.603271    4525 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52ca0496-3987-481c-8efa-28c13e432616-plugins-config-volume\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:48:13.603605 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:13.603509    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6bb3a0f-8779-436c-b404-a26849ff7943-plugins-config-volume" pod "f6bb3a0f-8779-436c-b404-a26849ff7943" (UID: "f6bb3a0f-8779-436c-b404-a26849ff7943"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:48:13.605451 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:13.605385    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6bb3a0f-8779-436c-b404-a26849ff7943-kube-api-access-zcxh2" pod "f6bb3a0f-8779-436c-b404-a26849ff7943" (UID: "f6bb3a0f-8779-436c-b404-a26849ff7943"). InnerVolumeSpecName "kube-api-access-zcxh2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:48:13.605831 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:13.605807    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52ca0496-3987-481c-8efa-28c13e432616-kube-api-access-p99gf" pod "52ca0496-3987-481c-8efa-28c13e432616" (UID: "52ca0496-3987-481c-8efa-28c13e432616"). InnerVolumeSpecName "kube-api-access-p99gf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:48:13.655048 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:13.655016    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e4242163-d74b5477f-fwc2h"
Oct 30 18:48:13.657147 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:13.657127    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e4242163-6fddb59d56-jvvtf"
Oct 30 18:48:13.703609 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:13.703576    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58554b4f-e880-4ac7-ac7a-d1d4cf33851a-kube-api-access-hb4vb\" (UniqueName: \"kubernetes.io/projected/58554b4f-e880-4ac7-ac7a-d1d4cf33851a-kube-api-access-hb4vb\") pod \"58554b4f-e880-4ac7-ac7a-d1d4cf33851a\" (UID: \"58554b4f-e880-4ac7-ac7a-d1d4cf33851a\") "
Oct 30 18:48:13.703987 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:13.703961    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58554b4f-e880-4ac7-ac7a-d1d4cf33851a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58554b4f-e880-4ac7-ac7a-d1d4cf33851a-plugins-config-volume\") pod \"58554b4f-e880-4ac7-ac7a-d1d4cf33851a\" (UID: \"58554b4f-e880-4ac7-ac7a-d1d4cf33851a\") "
Oct 30 18:48:13.704121 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:13.704007    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17744dc5-6240-4081-bc2b-768529e1ed13-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17744dc5-6240-4081-bc2b-768529e1ed13-plugins-config-volume\") pod \"17744dc5-6240-4081-bc2b-768529e1ed13\" (UID: \"17744dc5-6240-4081-bc2b-768529e1ed13\") "
Oct 30 18:48:13.704121 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:13.704058    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17744dc5-6240-4081-bc2b-768529e1ed13-kube-api-access-c4wg6\" (UniqueName: \"kubernetes.io/projected/17744dc5-6240-4081-bc2b-768529e1ed13-kube-api-access-c4wg6\") pod \"17744dc5-6240-4081-bc2b-768529e1ed13\" (UID: \"17744dc5-6240-4081-bc2b-768529e1ed13\") "
Oct 30 18:48:13.704927 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:13.704334    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcxh2\" (UniqueName: \"kubernetes.io/projected/f6bb3a0f-8779-436c-b404-a26849ff7943-kube-api-access-zcxh2\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:48:13.704927 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:13.704354    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p99gf\" (UniqueName: \"kubernetes.io/projected/52ca0496-3987-481c-8efa-28c13e432616-kube-api-access-p99gf\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:48:13.704927 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:13.704371    4525 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6bb3a0f-8779-436c-b404-a26849ff7943-plugins-config-volume\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:48:13.704927 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:13.704366    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58554b4f-e880-4ac7-ac7a-d1d4cf33851a-plugins-config-volume" pod "58554b4f-e880-4ac7-ac7a-d1d4cf33851a" (UID: "58554b4f-e880-4ac7-ac7a-d1d4cf33851a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:48:13.704927 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:13.704714    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17744dc5-6240-4081-bc2b-768529e1ed13-plugins-config-volume" pod "17744dc5-6240-4081-bc2b-768529e1ed13" (UID: "17744dc5-6240-4081-bc2b-768529e1ed13"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:48:13.706150 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:13.706123    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58554b4f-e880-4ac7-ac7a-d1d4cf33851a-kube-api-access-hb4vb" pod "58554b4f-e880-4ac7-ac7a-d1d4cf33851a" (UID: "58554b4f-e880-4ac7-ac7a-d1d4cf33851a"). InnerVolumeSpecName "kube-api-access-hb4vb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:48:13.706528 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:13.706507    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17744dc5-6240-4081-bc2b-768529e1ed13-kube-api-access-c4wg6" pod "17744dc5-6240-4081-bc2b-768529e1ed13" (UID: "17744dc5-6240-4081-bc2b-768529e1ed13"). InnerVolumeSpecName "kube-api-access-c4wg6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:48:13.735672 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:13.735633    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp"]
Oct 30 18:48:13.736296 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:13.736264    4525 cpu_manager.go:414] "containerMap: removing container" podUID="58554b4f-e880-4ac7-ac7a-d1d4cf33851a" containerName="payload-pre-processing"
Oct 30 18:48:13.736296 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:13.736288    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58554b4f-e880-4ac7-ac7a-d1d4cf33851a" containerName="payload-pre-processing"
Oct 30 18:48:13.736458 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:13.736334    4525 cpu_manager.go:414] "containerMap: removing container" podUID="52ca0496-3987-481c-8efa-28c13e432616" containerName="payload-processing"
Oct 30 18:48:13.736458 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:13.736342    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52ca0496-3987-481c-8efa-28c13e432616" containerName="payload-processing"
Oct 30 18:48:13.736458 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:13.736351    4525 cpu_manager.go:414] "containerMap: removing container" podUID="f6bb3a0f-8779-436c-b404-a26849ff7943" containerName="payload-pre-processing"
Oct 30 18:48:13.736458 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:13.736359    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6bb3a0f-8779-436c-b404-a26849ff7943" containerName="payload-pre-processing"
Oct 30 18:48:13.736458 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:13.736375    4525 cpu_manager.go:414] "containerMap: removing container" podUID="17744dc5-6240-4081-bc2b-768529e1ed13" containerName="payload-processing"
Oct 30 18:48:13.736458 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:13.736382    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17744dc5-6240-4081-bc2b-768529e1ed13" containerName="payload-processing"
Oct 30 18:48:13.736458 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:13.736395    4525 cpu_manager.go:414] "containerMap: removing container" podUID="1cec3994-12ce-4931-9d31-1b620974dd99" containerName="payload-pre-processing"
Oct 30 18:48:13.736458 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:13.736402    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cec3994-12ce-4931-9d31-1b620974dd99" containerName="payload-pre-processing"
Oct 30 18:48:13.736711 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:13.736483    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="52ca0496-3987-481c-8efa-28c13e432616" containerName="payload-processing"
Oct 30 18:48:13.736711 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:13.736492    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6bb3a0f-8779-436c-b404-a26849ff7943" containerName="payload-pre-processing"
Oct 30 18:48:13.736711 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:13.736503    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="17744dc5-6240-4081-bc2b-768529e1ed13" containerName="payload-processing"
Oct 30 18:48:13.736711 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:13.736512    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cec3994-12ce-4931-9d31-1b620974dd99" containerName="payload-pre-processing"
Oct 30 18:48:13.736711 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:13.736521    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="58554b4f-e880-4ac7-ac7a-d1d4cf33851a" containerName="payload-pre-processing"
Oct 30 18:48:13.740299 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:13.740279    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp"
Oct 30 18:48:13.766833 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:13.766803    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp"]
Oct 30 18:48:13.767354 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:13.767302    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-0458df-openshift-default-dockercfg-2vrss\""
Oct 30 18:48:13.807584 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:13.805215    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99aac9ad-b8dc-46cf-ac3f-69a814014bd2-istio-token\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp\" (UID: \"99aac9ad-b8dc-46cf-ac3f-69a814014bd2\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp"
Oct 30 18:48:13.807584 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:13.805295    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99aac9ad-b8dc-46cf-ac3f-69a814014bd2-istio-envoy\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp\" (UID: \"99aac9ad-b8dc-46cf-ac3f-69a814014bd2\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp"
Oct 30 18:48:13.807584 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:13.805326    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99aac9ad-b8dc-46cf-ac3f-69a814014bd2-istiod-ca-cert\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp\" (UID: \"99aac9ad-b8dc-46cf-ac3f-69a814014bd2\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp"
Oct 30 18:48:13.807584 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:13.805424    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccjq4\" (UniqueName: \"kubernetes.io/projected/99aac9ad-b8dc-46cf-ac3f-69a814014bd2-kube-api-access-ccjq4\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp\" (UID: \"99aac9ad-b8dc-46cf-ac3f-69a814014bd2\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp"
Oct 30 18:48:13.807584 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:13.805483    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99aac9ad-b8dc-46cf-ac3f-69a814014bd2-workload-certs\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp\" (UID: \"99aac9ad-b8dc-46cf-ac3f-69a814014bd2\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp"
Oct 30 18:48:13.807584 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:13.805605    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99aac9ad-b8dc-46cf-ac3f-69a814014bd2-istio-podinfo\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp\" (UID: \"99aac9ad-b8dc-46cf-ac3f-69a814014bd2\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp"
Oct 30 18:48:13.807584 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:13.805782    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99aac9ad-b8dc-46cf-ac3f-69a814014bd2-workload-socket\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp\" (UID: \"99aac9ad-b8dc-46cf-ac3f-69a814014bd2\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp"
Oct 30 18:48:13.807584 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:13.805814    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99aac9ad-b8dc-46cf-ac3f-69a814014bd2-credential-socket\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp\" (UID: \"99aac9ad-b8dc-46cf-ac3f-69a814014bd2\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp"
Oct 30 18:48:13.807584 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:13.805936    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99aac9ad-b8dc-46cf-ac3f-69a814014bd2-istio-data\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp\" (UID: \"99aac9ad-b8dc-46cf-ac3f-69a814014bd2\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp"
Oct 30 18:48:13.807584 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:13.806071    4525 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58554b4f-e880-4ac7-ac7a-d1d4cf33851a-plugins-config-volume\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:48:13.807584 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:13.806110    4525 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17744dc5-6240-4081-bc2b-768529e1ed13-plugins-config-volume\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:48:13.807584 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:13.806126    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4wg6\" (UniqueName: \"kubernetes.io/projected/17744dc5-6240-4081-bc2b-768529e1ed13-kube-api-access-c4wg6\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:48:13.807584 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:13.806141    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hb4vb\" (UniqueName: \"kubernetes.io/projected/58554b4f-e880-4ac7-ac7a-d1d4cf33851a-kube-api-access-hb4vb\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:48:13.906865 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:13.906769    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99aac9ad-b8dc-46cf-ac3f-69a814014bd2-workload-socket\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp\" (UID: \"99aac9ad-b8dc-46cf-ac3f-69a814014bd2\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp"
Oct 30 18:48:13.906865 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:13.906816    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99aac9ad-b8dc-46cf-ac3f-69a814014bd2-credential-socket\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp\" (UID: \"99aac9ad-b8dc-46cf-ac3f-69a814014bd2\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp"
Oct 30 18:48:13.906865 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:13.906860    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99aac9ad-b8dc-46cf-ac3f-69a814014bd2-istio-data\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp\" (UID: \"99aac9ad-b8dc-46cf-ac3f-69a814014bd2\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp"
Oct 30 18:48:13.907143 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:13.906902    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99aac9ad-b8dc-46cf-ac3f-69a814014bd2-istio-token\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp\" (UID: \"99aac9ad-b8dc-46cf-ac3f-69a814014bd2\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp"
Oct 30 18:48:13.907143 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:13.906946    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99aac9ad-b8dc-46cf-ac3f-69a814014bd2-istio-envoy\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp\" (UID: \"99aac9ad-b8dc-46cf-ac3f-69a814014bd2\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp"
Oct 30 18:48:13.907143 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:13.906972    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99aac9ad-b8dc-46cf-ac3f-69a814014bd2-istiod-ca-cert\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp\" (UID: \"99aac9ad-b8dc-46cf-ac3f-69a814014bd2\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp"
Oct 30 18:48:13.907143 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:13.907001    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccjq4\" (UniqueName: \"kubernetes.io/projected/99aac9ad-b8dc-46cf-ac3f-69a814014bd2-kube-api-access-ccjq4\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp\" (UID: \"99aac9ad-b8dc-46cf-ac3f-69a814014bd2\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp"
Oct 30 18:48:13.907143 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:13.907028    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99aac9ad-b8dc-46cf-ac3f-69a814014bd2-workload-certs\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp\" (UID: \"99aac9ad-b8dc-46cf-ac3f-69a814014bd2\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp"
Oct 30 18:48:13.907143 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:13.907072    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99aac9ad-b8dc-46cf-ac3f-69a814014bd2-istio-podinfo\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp\" (UID: \"99aac9ad-b8dc-46cf-ac3f-69a814014bd2\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp"
Oct 30 18:48:13.907407 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:13.907191    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99aac9ad-b8dc-46cf-ac3f-69a814014bd2-workload-socket\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp\" (UID: \"99aac9ad-b8dc-46cf-ac3f-69a814014bd2\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp"
Oct 30 18:48:13.907585 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:13.907557    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99aac9ad-b8dc-46cf-ac3f-69a814014bd2-workload-certs\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp\" (UID: \"99aac9ad-b8dc-46cf-ac3f-69a814014bd2\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp"
Oct 30 18:48:13.909904 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:13.909876    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99aac9ad-b8dc-46cf-ac3f-69a814014bd2-credential-socket\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp\" (UID: \"99aac9ad-b8dc-46cf-ac3f-69a814014bd2\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp"
Oct 30 18:48:13.910011 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:13.909987    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99aac9ad-b8dc-46cf-ac3f-69a814014bd2-istio-data\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp\" (UID: \"99aac9ad-b8dc-46cf-ac3f-69a814014bd2\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp"
Oct 30 18:48:13.910480 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:13.910412    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99aac9ad-b8dc-46cf-ac3f-69a814014bd2-istiod-ca-cert\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp\" (UID: \"99aac9ad-b8dc-46cf-ac3f-69a814014bd2\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp"
Oct 30 18:48:13.912387 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:13.912366    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99aac9ad-b8dc-46cf-ac3f-69a814014bd2-istio-envoy\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp\" (UID: \"99aac9ad-b8dc-46cf-ac3f-69a814014bd2\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp"
Oct 30 18:48:13.912500 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:13.912478    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99aac9ad-b8dc-46cf-ac3f-69a814014bd2-istio-podinfo\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp\" (UID: \"99aac9ad-b8dc-46cf-ac3f-69a814014bd2\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp"
Oct 30 18:48:13.942533 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:13.942501    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-fz9ck"]
Oct 30 18:48:13.947704 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:13.947672    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-fz9ck"
Oct 30 18:48:13.963794 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:13.963765    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccjq4\" (UniqueName: \"kubernetes.io/projected/99aac9ad-b8dc-46cf-ac3f-69a814014bd2-kube-api-access-ccjq4\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp\" (UID: \"99aac9ad-b8dc-46cf-ac3f-69a814014bd2\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp"
Oct 30 18:48:13.965447 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:13.965428    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-85cb41-openshift-default-dockercfg-s655r\""
Oct 30 18:48:13.969381 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:13.969348    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99aac9ad-b8dc-46cf-ac3f-69a814014bd2-istio-token\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp\" (UID: \"99aac9ad-b8dc-46cf-ac3f-69a814014bd2\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp"
Oct 30 18:48:13.985890 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:13.985869    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-fz9ck"]
Oct 30 18:48:14.007840 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:14.007805    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg8zb\" (UniqueName: \"kubernetes.io/projected/f236b7d6-975d-447f-bcaa-8e51d6ee0fc9-kube-api-access-xg8zb\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-fz9ck\" (UID: \"f236b7d6-975d-447f-bcaa-8e51d6ee0fc9\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-fz9ck"
Oct 30 18:48:14.008086 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:14.008061    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f236b7d6-975d-447f-bcaa-8e51d6ee0fc9-credential-socket\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-fz9ck\" (UID: \"f236b7d6-975d-447f-bcaa-8e51d6ee0fc9\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-fz9ck"
Oct 30 18:48:14.008348 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:14.008315    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f236b7d6-975d-447f-bcaa-8e51d6ee0fc9-istiod-ca-cert\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-fz9ck\" (UID: \"f236b7d6-975d-447f-bcaa-8e51d6ee0fc9\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-fz9ck"
Oct 30 18:48:14.008483 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:14.008405    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f236b7d6-975d-447f-bcaa-8e51d6ee0fc9-istio-envoy\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-fz9ck\" (UID: \"f236b7d6-975d-447f-bcaa-8e51d6ee0fc9\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-fz9ck"
Oct 30 18:48:14.008483 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:14.008458    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f236b7d6-975d-447f-bcaa-8e51d6ee0fc9-istio-podinfo\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-fz9ck\" (UID: \"f236b7d6-975d-447f-bcaa-8e51d6ee0fc9\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-fz9ck"
Oct 30 18:48:14.008581 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:14.008488    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f236b7d6-975d-447f-bcaa-8e51d6ee0fc9-istio-token\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-fz9ck\" (UID: \"f236b7d6-975d-447f-bcaa-8e51d6ee0fc9\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-fz9ck"
Oct 30 18:48:14.008581 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:14.008563    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f236b7d6-975d-447f-bcaa-8e51d6ee0fc9-workload-socket\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-fz9ck\" (UID: \"f236b7d6-975d-447f-bcaa-8e51d6ee0fc9\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-fz9ck"
Oct 30 18:48:14.008655 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:14.008590    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f236b7d6-975d-447f-bcaa-8e51d6ee0fc9-istio-data\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-fz9ck\" (UID: \"f236b7d6-975d-447f-bcaa-8e51d6ee0fc9\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-fz9ck"
Oct 30 18:48:14.008655 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:14.008615    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f236b7d6-975d-447f-bcaa-8e51d6ee0fc9-workload-certs\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-fz9ck\" (UID: \"f236b7d6-975d-447f-bcaa-8e51d6ee0fc9\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-fz9ck"
Oct 30 18:48:14.058375 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:14.058346    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp"
Oct 30 18:48:14.110400 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:14.110258    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg8zb\" (UniqueName: \"kubernetes.io/projected/f236b7d6-975d-447f-bcaa-8e51d6ee0fc9-kube-api-access-xg8zb\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-fz9ck\" (UID: \"f236b7d6-975d-447f-bcaa-8e51d6ee0fc9\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-fz9ck"
Oct 30 18:48:14.110400 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:14.110342    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f236b7d6-975d-447f-bcaa-8e51d6ee0fc9-credential-socket\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-fz9ck\" (UID: \"f236b7d6-975d-447f-bcaa-8e51d6ee0fc9\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-fz9ck"
Oct 30 18:48:14.110804 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:14.110764    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f236b7d6-975d-447f-bcaa-8e51d6ee0fc9-istiod-ca-cert\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-fz9ck\" (UID: \"f236b7d6-975d-447f-bcaa-8e51d6ee0fc9\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-fz9ck"
Oct 30 18:48:14.110903 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:14.110827    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f236b7d6-975d-447f-bcaa-8e51d6ee0fc9-istio-envoy\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-fz9ck\" (UID: \"f236b7d6-975d-447f-bcaa-8e51d6ee0fc9\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-fz9ck"
Oct 30 18:48:14.110903 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:14.110881    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f236b7d6-975d-447f-bcaa-8e51d6ee0fc9-istio-podinfo\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-fz9ck\" (UID: \"f236b7d6-975d-447f-bcaa-8e51d6ee0fc9\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-fz9ck"
Oct 30 18:48:14.111150 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:14.110912    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f236b7d6-975d-447f-bcaa-8e51d6ee0fc9-istio-token\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-fz9ck\" (UID: \"f236b7d6-975d-447f-bcaa-8e51d6ee0fc9\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-fz9ck"
Oct 30 18:48:14.111150 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:14.110962    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f236b7d6-975d-447f-bcaa-8e51d6ee0fc9-workload-socket\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-fz9ck\" (UID: \"f236b7d6-975d-447f-bcaa-8e51d6ee0fc9\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-fz9ck"
Oct 30 18:48:14.111150 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:14.111001    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f236b7d6-975d-447f-bcaa-8e51d6ee0fc9-istio-data\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-fz9ck\" (UID: \"f236b7d6-975d-447f-bcaa-8e51d6ee0fc9\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-fz9ck"
Oct 30 18:48:14.111150 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:14.111064    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f236b7d6-975d-447f-bcaa-8e51d6ee0fc9-workload-certs\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-fz9ck\" (UID: \"f236b7d6-975d-447f-bcaa-8e51d6ee0fc9\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-fz9ck"
Oct 30 18:48:14.111507 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:14.111486    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f236b7d6-975d-447f-bcaa-8e51d6ee0fc9-workload-certs\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-fz9ck\" (UID: \"f236b7d6-975d-447f-bcaa-8e51d6ee0fc9\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-fz9ck"
Oct 30 18:48:14.111973 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:14.111774    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f236b7d6-975d-447f-bcaa-8e51d6ee0fc9-credential-socket\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-fz9ck\" (UID: \"f236b7d6-975d-447f-bcaa-8e51d6ee0fc9\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-fz9ck"
Oct 30 18:48:14.112829 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:14.112441    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f236b7d6-975d-447f-bcaa-8e51d6ee0fc9-workload-socket\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-fz9ck\" (UID: \"f236b7d6-975d-447f-bcaa-8e51d6ee0fc9\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-fz9ck"
Oct 30 18:48:14.112829 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:14.112720    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f236b7d6-975d-447f-bcaa-8e51d6ee0fc9-istio-data\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-fz9ck\" (UID: \"f236b7d6-975d-447f-bcaa-8e51d6ee0fc9\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-fz9ck"
Oct 30 18:48:14.112829 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:14.112801    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f236b7d6-975d-447f-bcaa-8e51d6ee0fc9-istiod-ca-cert\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-fz9ck\" (UID: \"f236b7d6-975d-447f-bcaa-8e51d6ee0fc9\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-fz9ck"
Oct 30 18:48:14.122661 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:14.122633    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f236b7d6-975d-447f-bcaa-8e51d6ee0fc9-istio-envoy\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-fz9ck\" (UID: \"f236b7d6-975d-447f-bcaa-8e51d6ee0fc9\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-fz9ck"
Oct 30 18:48:14.125205 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:14.125177    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg8zb\" (UniqueName: \"kubernetes.io/projected/f236b7d6-975d-447f-bcaa-8e51d6ee0fc9-kube-api-access-xg8zb\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-fz9ck\" (UID: \"f236b7d6-975d-447f-bcaa-8e51d6ee0fc9\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-fz9ck"
Oct 30 18:48:14.126503 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:14.126448    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f236b7d6-975d-447f-bcaa-8e51d6ee0fc9-istio-token\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-fz9ck\" (UID: \"f236b7d6-975d-447f-bcaa-8e51d6ee0fc9\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-fz9ck"
Oct 30 18:48:14.133429 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:14.133404    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f236b7d6-975d-447f-bcaa-8e51d6ee0fc9-istio-podinfo\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-fz9ck\" (UID: \"f236b7d6-975d-447f-bcaa-8e51d6ee0fc9\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-fz9ck"
Oct 30 18:48:14.260879 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:14.260796    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-fz9ck"
Oct 30 18:48:14.285449 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:14.285423    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-226ecbcb-5d69cc78dd-rjzsb"
Oct 30 18:48:14.285580 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:14.285421    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-226ecbcb-5d69cc78dd-rjzsb" event={"ID":"52ca0496-3987-481c-8efa-28c13e432616","Type":"ContainerDied","Data":"ff9c1252eac0b002ec98ee54816df1a21492ae1d5958bfc2d88ad011211ecf4e"}
Oct 30 18:48:14.285637 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:14.285619    4525 scope.go:122] "RemoveContainer" containerID="e1684db67038f8b9d24c3d17848bf2a5c9f86be6d37613a987ff060991029960"
Oct 30 18:48:14.287646 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:14.287604    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-226ecbcb-69b85788bc-8xszf" event={"ID":"f6bb3a0f-8779-436c-b404-a26849ff7943","Type":"ContainerDied","Data":"8deb25233a75babfd1f6f84c0365d2edc76bcc2a3b83c3e75fa6a0270f18777a"}
Oct 30 18:48:14.287646 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:14.287621    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-226ecbcb-69b85788bc-8xszf"
Oct 30 18:48:14.296396 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:14.296370    4525 generic.go:356] "Generic (PLEG): container finished" podID="58554b4f-e880-4ac7-ac7a-d1d4cf33851a" containerID="d665521a08bfe260a306388acef0050fcf79f7d6195288686625d7bb597f0e9e" exitCode=0
Oct 30 18:48:14.296488 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:14.296430    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e4242163-d74b5477f-fwc2h" event={"ID":"58554b4f-e880-4ac7-ac7a-d1d4cf33851a","Type":"ContainerDied","Data":"d665521a08bfe260a306388acef0050fcf79f7d6195288686625d7bb597f0e9e"}
Oct 30 18:48:14.296488 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:14.296455    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e4242163-d74b5477f-fwc2h"
Oct 30 18:48:14.296488 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:14.296478    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e4242163-d74b5477f-fwc2h" event={"ID":"58554b4f-e880-4ac7-ac7a-d1d4cf33851a","Type":"ContainerDied","Data":"e3bcf3a9e9e66d983b4b6227c4bd27e79f8877d0987a116eb006a9573804ad33"}
Oct 30 18:48:14.299315 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:14.299267    4525 generic.go:356] "Generic (PLEG): container finished" podID="17744dc5-6240-4081-bc2b-768529e1ed13" containerID="b127defd743b7f16bb5fa59d158822e6e3e56056c8c5994168c9d4589bcda1a7" exitCode=0
Oct 30 18:48:14.299404 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:14.299387    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e4242163-6fddb59d56-jvvtf" event={"ID":"17744dc5-6240-4081-bc2b-768529e1ed13","Type":"ContainerDied","Data":"b127defd743b7f16bb5fa59d158822e6e3e56056c8c5994168c9d4589bcda1a7"}
Oct 30 18:48:14.299468 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:14.299427    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e4242163-6fddb59d56-jvvtf" event={"ID":"17744dc5-6240-4081-bc2b-768529e1ed13","Type":"ContainerDied","Data":"a78e4f2835ceb32e5786593771b76cb2b523b9592bc37c404d4e7241f5b4c0a6"}
Oct 30 18:48:14.299528 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:14.299517    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e4242163-6fddb59d56-jvvtf"
Oct 30 18:48:14.299575 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:14.299533    4525 scope.go:122] "RemoveContainer" containerID="0376469a7047eec342bef411fb5114e4b1370362bc70d632a6a5e67ad0835c9d"
Oct 30 18:48:14.317674 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:14.316743    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-226ecbcb-5d69cc78dd-rjzsb"]
Oct 30 18:48:14.317674 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:14.316985    4525 scope.go:122] "RemoveContainer" containerID="d665521a08bfe260a306388acef0050fcf79f7d6195288686625d7bb597f0e9e"
Oct 30 18:48:14.322785 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:14.322757    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-226ecbcb-5d69cc78dd-rjzsb"]
Oct 30 18:48:14.336923 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:14.336903    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e4242163-d74b5477f-fwc2h"]
Oct 30 18:48:14.337785 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:14.337164    4525 scope.go:122] "RemoveContainer" containerID="d665521a08bfe260a306388acef0050fcf79f7d6195288686625d7bb597f0e9e"
Oct 30 18:48:14.338408 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:48:14.338281    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d665521a08bfe260a306388acef0050fcf79f7d6195288686625d7bb597f0e9e\": container with ID starting with d665521a08bfe260a306388acef0050fcf79f7d6195288686625d7bb597f0e9e not found: ID does not exist" containerID="d665521a08bfe260a306388acef0050fcf79f7d6195288686625d7bb597f0e9e"
Oct 30 18:48:14.338408 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:14.338320    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d665521a08bfe260a306388acef0050fcf79f7d6195288686625d7bb597f0e9e"} err="failed to get container status \"d665521a08bfe260a306388acef0050fcf79f7d6195288686625d7bb597f0e9e\": rpc error: code = NotFound desc = could not find container \"d665521a08bfe260a306388acef0050fcf79f7d6195288686625d7bb597f0e9e\": container with ID starting with d665521a08bfe260a306388acef0050fcf79f7d6195288686625d7bb597f0e9e not found: ID does not exist"
Oct 30 18:48:14.338408 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:14.338344    4525 scope.go:122] "RemoveContainer" containerID="b127defd743b7f16bb5fa59d158822e6e3e56056c8c5994168c9d4589bcda1a7"
Oct 30 18:48:14.341643 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:14.340941    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e4242163-d74b5477f-fwc2h"]
Oct 30 18:48:14.343999 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:14.343982    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp"]
Oct 30 18:48:14.347192 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:48:14.347169    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99aac9ad_b8dc_46cf_ac3f_69a814014bd2.slice/crio-12a5024eb19a90cd471f14953d730849f3acd42d98ebd3540324639b2c6fb9e2 WatchSource:0}: Error finding container 12a5024eb19a90cd471f14953d730849f3acd42d98ebd3540324639b2c6fb9e2: Status 404 returned error can't find the container with id 12a5024eb19a90cd471f14953d730849f3acd42d98ebd3540324639b2c6fb9e2
Oct 30 18:48:14.350709 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:14.350660    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:48:14.350834 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:14.350748    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:48:14.350834 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:14.350800    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:48:14.350945 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:14.350859    4525 scope.go:122] "RemoveContainer" containerID="b127defd743b7f16bb5fa59d158822e6e3e56056c8c5994168c9d4589bcda1a7"
Oct 30 18:48:14.351218 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:48:14.351174    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b127defd743b7f16bb5fa59d158822e6e3e56056c8c5994168c9d4589bcda1a7\": container with ID starting with b127defd743b7f16bb5fa59d158822e6e3e56056c8c5994168c9d4589bcda1a7 not found: ID does not exist" containerID="b127defd743b7f16bb5fa59d158822e6e3e56056c8c5994168c9d4589bcda1a7"
Oct 30 18:48:14.351326 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:14.351220    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b127defd743b7f16bb5fa59d158822e6e3e56056c8c5994168c9d4589bcda1a7"} err="failed to get container status \"b127defd743b7f16bb5fa59d158822e6e3e56056c8c5994168c9d4589bcda1a7\": rpc error: code = NotFound desc = could not find container \"b127defd743b7f16bb5fa59d158822e6e3e56056c8c5994168c9d4589bcda1a7\": container with ID starting with b127defd743b7f16bb5fa59d158822e6e3e56056c8c5994168c9d4589bcda1a7 not found: ID does not exist"
Oct 30 18:48:14.353219 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:14.353203    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e4242163-6fddb59d56-jvvtf"]
Oct 30 18:48:14.360022 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:14.359986    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e4242163-6fddb59d56-jvvtf"]
Oct 30 18:48:14.369516 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:14.369495    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-226ecbcb-69b85788bc-8xszf"]
Oct 30 18:48:14.374492 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:14.374472    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-226ecbcb-69b85788bc-8xszf"]
Oct 30 18:48:14.509455 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:14.509395    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-ee87bf29-5dbc87966c-x9gwt" podUID="4f1a2f0a-2bdd-4765-b81c-38323e22fbe9" containerName="payload-pre-processing" containerID="cri-o://9d8ccae45f0e180b0649552c26d0705ad140f1e4fe51d4e1e41725f805f2c0f5" gracePeriod=25
Oct 30 18:48:14.552591 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:14.552562    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-fz9ck"]
Oct 30 18:48:14.590596 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:48:14.590567    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf236b7d6_975d_447f_bcaa_8e51d6ee0fc9.slice/crio-aefef72c23ac7e9a4862c88887060e09a76a2cf3e2e4f54a777f343952fb2fde WatchSource:0}: Error finding container aefef72c23ac7e9a4862c88887060e09a76a2cf3e2e4f54a777f343952fb2fde: Status 404 returned error can't find the container with id aefef72c23ac7e9a4862c88887060e09a76a2cf3e2e4f54a777f343952fb2fde
Oct 30 18:48:14.616134 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:14.616104    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:48:14.616233 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:14.616169    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:48:14.616233 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:14.616200    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:48:14.765236 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:14.764881    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ee87bf29-5dbc87966c-x9gwt"
Oct 30 18:48:14.828112 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:14.828082    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f1a2f0a-2bdd-4765-b81c-38323e22fbe9-kube-api-access-286rw\" (UniqueName: \"kubernetes.io/projected/4f1a2f0a-2bdd-4765-b81c-38323e22fbe9-kube-api-access-286rw\") pod \"4f1a2f0a-2bdd-4765-b81c-38323e22fbe9\" (UID: \"4f1a2f0a-2bdd-4765-b81c-38323e22fbe9\") "
Oct 30 18:48:14.828273 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:14.828229    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f1a2f0a-2bdd-4765-b81c-38323e22fbe9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f1a2f0a-2bdd-4765-b81c-38323e22fbe9-plugins-config-volume\") pod \"4f1a2f0a-2bdd-4765-b81c-38323e22fbe9\" (UID: \"4f1a2f0a-2bdd-4765-b81c-38323e22fbe9\") "
Oct 30 18:48:14.828631 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:14.828604    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f1a2f0a-2bdd-4765-b81c-38323e22fbe9-plugins-config-volume" pod "4f1a2f0a-2bdd-4765-b81c-38323e22fbe9" (UID: "4f1a2f0a-2bdd-4765-b81c-38323e22fbe9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:48:14.830430 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:14.830410    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f1a2f0a-2bdd-4765-b81c-38323e22fbe9-kube-api-access-286rw" pod "4f1a2f0a-2bdd-4765-b81c-38323e22fbe9" (UID: "4f1a2f0a-2bdd-4765-b81c-38323e22fbe9"). InnerVolumeSpecName "kube-api-access-286rw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:48:14.929177 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:14.929144    4525 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f1a2f0a-2bdd-4765-b81c-38323e22fbe9-plugins-config-volume\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:48:14.929177 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:14.929174    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-286rw\" (UniqueName: \"kubernetes.io/projected/4f1a2f0a-2bdd-4765-b81c-38323e22fbe9-kube-api-access-286rw\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:48:15.309093 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:15.309057    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp" event={"ID":"99aac9ad-b8dc-46cf-ac3f-69a814014bd2","Type":"ContainerStarted","Data":"a10ca651e5feab589cc19423939c8ccde1c0e0ea87a11276221f4529ff26c8d0"}
Oct 30 18:48:15.309275 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:15.309119    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp" event={"ID":"99aac9ad-b8dc-46cf-ac3f-69a814014bd2","Type":"ContainerStarted","Data":"12a5024eb19a90cd471f14953d730849f3acd42d98ebd3540324639b2c6fb9e2"}
Oct 30 18:48:15.310438 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:15.310411    4525 generic.go:356] "Generic (PLEG): container finished" podID="4f1a2f0a-2bdd-4765-b81c-38323e22fbe9" containerID="9d8ccae45f0e180b0649552c26d0705ad140f1e4fe51d4e1e41725f805f2c0f5" exitCode=0
Oct 30 18:48:15.310576 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:15.310451    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ee87bf29-5dbc87966c-x9gwt"
Oct 30 18:48:15.310576 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:15.310486    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ee87bf29-5dbc87966c-x9gwt" event={"ID":"4f1a2f0a-2bdd-4765-b81c-38323e22fbe9","Type":"ContainerDied","Data":"9d8ccae45f0e180b0649552c26d0705ad140f1e4fe51d4e1e41725f805f2c0f5"}
Oct 30 18:48:15.310576 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:15.310524    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ee87bf29-5dbc87966c-x9gwt" event={"ID":"4f1a2f0a-2bdd-4765-b81c-38323e22fbe9","Type":"ContainerDied","Data":"49ef42b723f3b09045fb908466ecc8e5a895100bc1ed97d1b82a20b602029602"}
Oct 30 18:48:15.310576 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:15.310565    4525 scope.go:122] "RemoveContainer" containerID="9d8ccae45f0e180b0649552c26d0705ad140f1e4fe51d4e1e41725f805f2c0f5"
Oct 30 18:48:15.314770 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:15.314741    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-fz9ck" event={"ID":"f236b7d6-975d-447f-bcaa-8e51d6ee0fc9","Type":"ContainerStarted","Data":"d181f6236df07f838dc96fb118ef6348173a3112ca2f7a7930028c1a496140db"}
Oct 30 18:48:15.314884 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:15.314789    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-fz9ck" event={"ID":"f236b7d6-975d-447f-bcaa-8e51d6ee0fc9","Type":"ContainerStarted","Data":"aefef72c23ac7e9a4862c88887060e09a76a2cf3e2e4f54a777f343952fb2fde"}
Oct 30 18:48:15.320664 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:15.320643    4525 scope.go:122] "RemoveContainer" containerID="9d8ccae45f0e180b0649552c26d0705ad140f1e4fe51d4e1e41725f805f2c0f5"
Oct 30 18:48:15.321007 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:48:15.320989    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d8ccae45f0e180b0649552c26d0705ad140f1e4fe51d4e1e41725f805f2c0f5\": container with ID starting with 9d8ccae45f0e180b0649552c26d0705ad140f1e4fe51d4e1e41725f805f2c0f5 not found: ID does not exist" containerID="9d8ccae45f0e180b0649552c26d0705ad140f1e4fe51d4e1e41725f805f2c0f5"
Oct 30 18:48:15.321096 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:15.321014    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d8ccae45f0e180b0649552c26d0705ad140f1e4fe51d4e1e41725f805f2c0f5"} err="failed to get container status \"9d8ccae45f0e180b0649552c26d0705ad140f1e4fe51d4e1e41725f805f2c0f5\": rpc error: code = NotFound desc = could not find container \"9d8ccae45f0e180b0649552c26d0705ad140f1e4fe51d4e1e41725f805f2c0f5\": container with ID starting with 9d8ccae45f0e180b0649552c26d0705ad140f1e4fe51d4e1e41725f805f2c0f5 not found: ID does not exist"
Oct 30 18:48:15.333593 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:15.333522    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp" podStartSLOduration=2.333512171 podStartE2EDuration="2.333512171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:48:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:48:15.331986887 +0000 UTC m=+717.957656851" watchObservedRunningTime="2026-10-30 18:48:15.333512171 +0000 UTC m=+717.959182204"
Oct 30 18:48:15.357150 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:15.357114    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-fz9ck" podStartSLOduration=2.357103786 podStartE2EDuration="2.357103786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:48:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:48:15.355706981 +0000 UTC m=+717.981376948" watchObservedRunningTime="2026-10-30 18:48:15.357103786 +0000 UTC m=+717.982773750"
Oct 30 18:48:15.374789 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:15.374759    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ee87bf29-5dbc87966c-x9gwt"]
Oct 30 18:48:15.379854 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:15.379807    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ee87bf29-5dbc87966c-x9gwt"]
Oct 30 18:48:15.543612 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:15.543581    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17744dc5-6240-4081-bc2b-768529e1ed13" path="/var/lib/kubelet/pods/17744dc5-6240-4081-bc2b-768529e1ed13/volumes"
Oct 30 18:48:15.544036 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:15.543957    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f1a2f0a-2bdd-4765-b81c-38323e22fbe9" path="/var/lib/kubelet/pods/4f1a2f0a-2bdd-4765-b81c-38323e22fbe9/volumes"
Oct 30 18:48:15.544273 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:15.544260    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52ca0496-3987-481c-8efa-28c13e432616" path="/var/lib/kubelet/pods/52ca0496-3987-481c-8efa-28c13e432616/volumes"
Oct 30 18:48:15.544569 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:15.544558    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58554b4f-e880-4ac7-ac7a-d1d4cf33851a" path="/var/lib/kubelet/pods/58554b4f-e880-4ac7-ac7a-d1d4cf33851a/volumes"
Oct 30 18:48:15.545238 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:15.545223    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6bb3a0f-8779-436c-b404-a26849ff7943" path="/var/lib/kubelet/pods/f6bb3a0f-8779-436c-b404-a26849ff7943/volumes"
Oct 30 18:48:16.058618 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:16.058573    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp"
Oct 30 18:48:16.064415 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:16.064385    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp"
Oct 30 18:48:16.261247 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:16.261208    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-fz9ck"
Oct 30 18:48:16.266343 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:16.266317    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-fz9ck"
Oct 30 18:48:16.323260 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:16.323181    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-fz9ck"
Oct 30 18:48:16.323260 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:16.323242    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp"
Oct 30 18:48:16.324363 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:16.324338    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-fz9ck"
Oct 30 18:48:16.324674 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:16.324659    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp"
Oct 30 18:48:17.421694 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:48:17.421639    4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d49l" containerName="extractor"
Oct 30 18:48:18.613358 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:18.613334    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-f951dfa9-openshift-default-5ddd96b597-dzjdp"
Oct 30 18:48:18.703751 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:18.703703    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a29fd997-17ba-4260-8279-05e6f1c6346a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a29fd997-17ba-4260-8279-05e6f1c6346a-workload-socket\") pod \"a29fd997-17ba-4260-8279-05e6f1c6346a\" (UID: \"a29fd997-17ba-4260-8279-05e6f1c6346a\") "
Oct 30 18:48:18.703971 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:18.703766    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a29fd997-17ba-4260-8279-05e6f1c6346a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a29fd997-17ba-4260-8279-05e6f1c6346a-istio-podinfo\") pod \"a29fd997-17ba-4260-8279-05e6f1c6346a\" (UID: \"a29fd997-17ba-4260-8279-05e6f1c6346a\") "
Oct 30 18:48:18.703971 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:18.703804    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a29fd997-17ba-4260-8279-05e6f1c6346a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a29fd997-17ba-4260-8279-05e6f1c6346a-workload-certs\") pod \"a29fd997-17ba-4260-8279-05e6f1c6346a\" (UID: \"a29fd997-17ba-4260-8279-05e6f1c6346a\") "
Oct 30 18:48:18.703971 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:18.703873    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a29fd997-17ba-4260-8279-05e6f1c6346a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a29fd997-17ba-4260-8279-05e6f1c6346a-istiod-ca-cert\") pod \"a29fd997-17ba-4260-8279-05e6f1c6346a\" (UID: \"a29fd997-17ba-4260-8279-05e6f1c6346a\") "
Oct 30 18:48:18.704242 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:18.703986    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a29fd997-17ba-4260-8279-05e6f1c6346a-kube-api-access-wttcv\" (UniqueName: \"kubernetes.io/projected/a29fd997-17ba-4260-8279-05e6f1c6346a-kube-api-access-wttcv\") pod \"a29fd997-17ba-4260-8279-05e6f1c6346a\" (UID: \"a29fd997-17ba-4260-8279-05e6f1c6346a\") "
Oct 30 18:48:18.704242 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:18.704033    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a29fd997-17ba-4260-8279-05e6f1c6346a-istio-token\" (UniqueName: \"kubernetes.io/projected/a29fd997-17ba-4260-8279-05e6f1c6346a-istio-token\") pod \"a29fd997-17ba-4260-8279-05e6f1c6346a\" (UID: \"a29fd997-17ba-4260-8279-05e6f1c6346a\") "
Oct 30 18:48:18.704242 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:18.704071    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a29fd997-17ba-4260-8279-05e6f1c6346a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a29fd997-17ba-4260-8279-05e6f1c6346a-credential-socket\") pod \"a29fd997-17ba-4260-8279-05e6f1c6346a\" (UID: \"a29fd997-17ba-4260-8279-05e6f1c6346a\") "
Oct 30 18:48:18.704242 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:18.704103    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a29fd997-17ba-4260-8279-05e6f1c6346a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a29fd997-17ba-4260-8279-05e6f1c6346a-istio-envoy\") pod \"a29fd997-17ba-4260-8279-05e6f1c6346a\" (UID: \"a29fd997-17ba-4260-8279-05e6f1c6346a\") "
Oct 30 18:48:18.704242 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:18.704143    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a29fd997-17ba-4260-8279-05e6f1c6346a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a29fd997-17ba-4260-8279-05e6f1c6346a-istio-data\") pod \"a29fd997-17ba-4260-8279-05e6f1c6346a\" (UID: \"a29fd997-17ba-4260-8279-05e6f1c6346a\") "
Oct 30 18:48:18.704599 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:18.704568    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a29fd997-17ba-4260-8279-05e6f1c6346a-istiod-ca-cert" pod "a29fd997-17ba-4260-8279-05e6f1c6346a" (UID: "a29fd997-17ba-4260-8279-05e6f1c6346a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:48:18.704889 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:18.704867    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a29fd997-17ba-4260-8279-05e6f1c6346a-workload-socket" pod "a29fd997-17ba-4260-8279-05e6f1c6346a" (UID: "a29fd997-17ba-4260-8279-05e6f1c6346a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:48:18.705667 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:18.705636    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a29fd997-17ba-4260-8279-05e6f1c6346a-credential-socket" pod "a29fd997-17ba-4260-8279-05e6f1c6346a" (UID: "a29fd997-17ba-4260-8279-05e6f1c6346a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:48:18.705850 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:18.705783    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a29fd997-17ba-4260-8279-05e6f1c6346a-workload-certs" pod "a29fd997-17ba-4260-8279-05e6f1c6346a" (UID: "a29fd997-17ba-4260-8279-05e6f1c6346a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:48:18.706235 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:18.706207    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a29fd997-17ba-4260-8279-05e6f1c6346a-istio-data" pod "a29fd997-17ba-4260-8279-05e6f1c6346a" (UID: "a29fd997-17ba-4260-8279-05e6f1c6346a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:48:18.718050 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:18.717820    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a29fd997-17ba-4260-8279-05e6f1c6346a-kube-api-access-wttcv" pod "a29fd997-17ba-4260-8279-05e6f1c6346a" (UID: "a29fd997-17ba-4260-8279-05e6f1c6346a"). InnerVolumeSpecName "kube-api-access-wttcv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:48:18.718050 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:18.717821    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a29fd997-17ba-4260-8279-05e6f1c6346a-istio-podinfo" pod "a29fd997-17ba-4260-8279-05e6f1c6346a" (UID: "a29fd997-17ba-4260-8279-05e6f1c6346a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 18:48:18.718050 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:18.717877    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a29fd997-17ba-4260-8279-05e6f1c6346a-istio-envoy" pod "a29fd997-17ba-4260-8279-05e6f1c6346a" (UID: "a29fd997-17ba-4260-8279-05e6f1c6346a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:48:18.718213 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:18.718143    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a29fd997-17ba-4260-8279-05e6f1c6346a-istio-token" pod "a29fd997-17ba-4260-8279-05e6f1c6346a" (UID: "a29fd997-17ba-4260-8279-05e6f1c6346a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:48:18.805842 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:18.805749    4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a29fd997-17ba-4260-8279-05e6f1c6346a-istio-token\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:48:18.805842 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:18.805788    4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a29fd997-17ba-4260-8279-05e6f1c6346a-credential-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:48:18.805842 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:18.805806    4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a29fd997-17ba-4260-8279-05e6f1c6346a-istio-envoy\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:48:18.805842 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:18.805821    4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a29fd997-17ba-4260-8279-05e6f1c6346a-istio-data\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:48:18.805842 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:18.805837    4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a29fd997-17ba-4260-8279-05e6f1c6346a-workload-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:48:18.805842 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:18.805851    4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a29fd997-17ba-4260-8279-05e6f1c6346a-istio-podinfo\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:48:18.806343 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:18.805865    4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a29fd997-17ba-4260-8279-05e6f1c6346a-workload-certs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:48:18.806343 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:18.805881    4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a29fd997-17ba-4260-8279-05e6f1c6346a-istiod-ca-cert\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:48:18.806343 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:18.805895    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wttcv\" (UniqueName: \"kubernetes.io/projected/a29fd997-17ba-4260-8279-05e6f1c6346a-kube-api-access-wttcv\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:48:19.344347 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:19.344313    4525 generic.go:356] "Generic (PLEG): container finished" podID="a29fd997-17ba-4260-8279-05e6f1c6346a" containerID="0734ce8a9028d0129512de7d641ebd9b420d93a54ab46128b44dd2b21c2b867f" exitCode=0
Oct 30 18:48:19.344517 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:19.344367    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-f951dfa9-openshift-default-5ddd96b597-dzjdp"
Oct 30 18:48:19.344517 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:19.344402    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f951dfa9-openshift-default-5ddd96b597-dzjdp" event={"ID":"a29fd997-17ba-4260-8279-05e6f1c6346a","Type":"ContainerDied","Data":"0734ce8a9028d0129512de7d641ebd9b420d93a54ab46128b44dd2b21c2b867f"}
Oct 30 18:48:19.344517 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:19.344479    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f951dfa9-openshift-default-5ddd96b597-dzjdp" event={"ID":"a29fd997-17ba-4260-8279-05e6f1c6346a","Type":"ContainerDied","Data":"1ed4ee44df54a937d84fe1612f6ba1e9990b982377adc2bc9919281f16d159bd"}
Oct 30 18:48:19.344517 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:19.344501    4525 scope.go:122] "RemoveContainer" containerID="0734ce8a9028d0129512de7d641ebd9b420d93a54ab46128b44dd2b21c2b867f"
Oct 30 18:48:19.357353 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:19.357331    4525 scope.go:122] "RemoveContainer" containerID="004fe667a937ac9a8bc60137eeedd2bcd7407bd1ca8df0fd781dbc0ebad252b2"
Oct 30 18:48:19.367183 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:19.367164    4525 scope.go:122] "RemoveContainer" containerID="0734ce8a9028d0129512de7d641ebd9b420d93a54ab46128b44dd2b21c2b867f"
Oct 30 18:48:19.367543 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:48:19.367522    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0734ce8a9028d0129512de7d641ebd9b420d93a54ab46128b44dd2b21c2b867f\": container with ID starting with 0734ce8a9028d0129512de7d641ebd9b420d93a54ab46128b44dd2b21c2b867f not found: ID does not exist" containerID="0734ce8a9028d0129512de7d641ebd9b420d93a54ab46128b44dd2b21c2b867f"
Oct 30 18:48:19.367632 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:19.367555    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0734ce8a9028d0129512de7d641ebd9b420d93a54ab46128b44dd2b21c2b867f"} err="failed to get container status \"0734ce8a9028d0129512de7d641ebd9b420d93a54ab46128b44dd2b21c2b867f\": rpc error: code = NotFound desc = could not find container \"0734ce8a9028d0129512de7d641ebd9b420d93a54ab46128b44dd2b21c2b867f\": container with ID starting with 0734ce8a9028d0129512de7d641ebd9b420d93a54ab46128b44dd2b21c2b867f not found: ID does not exist"
Oct 30 18:48:19.367632 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:19.367572    4525 scope.go:122] "RemoveContainer" containerID="004fe667a937ac9a8bc60137eeedd2bcd7407bd1ca8df0fd781dbc0ebad252b2"
Oct 30 18:48:19.367899 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:48:19.367875    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"004fe667a937ac9a8bc60137eeedd2bcd7407bd1ca8df0fd781dbc0ebad252b2\": container with ID starting with 004fe667a937ac9a8bc60137eeedd2bcd7407bd1ca8df0fd781dbc0ebad252b2 not found: ID does not exist" containerID="004fe667a937ac9a8bc60137eeedd2bcd7407bd1ca8df0fd781dbc0ebad252b2"
Oct 30 18:48:19.367984 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:19.367907    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"004fe667a937ac9a8bc60137eeedd2bcd7407bd1ca8df0fd781dbc0ebad252b2"} err="failed to get container status \"004fe667a937ac9a8bc60137eeedd2bcd7407bd1ca8df0fd781dbc0ebad252b2\": rpc error: code = NotFound desc = could not find container \"004fe667a937ac9a8bc60137eeedd2bcd7407bd1ca8df0fd781dbc0ebad252b2\": container with ID starting with 004fe667a937ac9a8bc60137eeedd2bcd7407bd1ca8df0fd781dbc0ebad252b2 not found: ID does not exist"
Oct 30 18:48:19.390788 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:19.390754    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-f951dfa9-openshift-default-5ddd96b597-dzjdp"]
Oct 30 18:48:19.392346 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:19.392320    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-f951dfa9-openshift-default-5ddd96b597-dzjdp"]
Oct 30 18:48:19.550100 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:19.550055    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a29fd997-17ba-4260-8279-05e6f1c6346a" path="/var/lib/kubelet/pods/a29fd997-17ba-4260-8279-05e6f1c6346a/volumes"
Oct 30 18:48:25.248874 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:25.248843    4525 patch_prober.go:29] interesting pod/e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 30 18:48:25.249474 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:25.248912    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j" podUID="6061ede7-981c-4c07-acfc-04be1cae252d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 30 18:48:25.407924 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:25.407849    4525 generic.go:356] "Generic (PLEG): container finished" podID="6061ede7-981c-4c07-acfc-04be1cae252d" containerID="95be28c0a2e6e7efa2daa699bd3d87456912a01b2d5df468ae82634e02d7a3b7" exitCode=0
Oct 30 18:48:25.407924 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:25.407923    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j" event={"ID":"6061ede7-981c-4c07-acfc-04be1cae252d","Type":"ContainerDied","Data":"95be28c0a2e6e7efa2daa699bd3d87456912a01b2d5df468ae82634e02d7a3b7"}
Oct 30 18:48:25.408261 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:25.408247    4525 scope.go:122] "RemoveContainer" containerID="95be28c0a2e6e7efa2daa699bd3d87456912a01b2d5df468ae82634e02d7a3b7"
Oct 30 18:48:25.409466 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:25.409438    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:48:25.409560 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:25.409507    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:48:25.409560 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:25.409539    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:48:26.430869 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:26.430822    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j" event={"ID":"6061ede7-981c-4c07-acfc-04be1cae252d","Type":"ContainerStarted","Data":"c805a23922cfcabad21322c61935bbb611eb1011d79b925c872c6c66f018364b"}
Oct 30 18:48:26.741248 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:26.741163    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j"
Oct 30 18:48:26.741458 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:26.741431    4525 patch_prober.go:29] interesting pod/e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 30 18:48:26.741519 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:26.741496    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j" podUID="6061ede7-981c-4c07-acfc-04be1cae252d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 30 18:48:27.466872 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:27.466840    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t"]
Oct 30 18:48:27.467406 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:27.467386    4525 cpu_manager.go:414] "containerMap: removing container" podUID="4f1a2f0a-2bdd-4765-b81c-38323e22fbe9" containerName="payload-pre-processing"
Oct 30 18:48:27.467484 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:27.467407    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f1a2f0a-2bdd-4765-b81c-38323e22fbe9" containerName="payload-pre-processing"
Oct 30 18:48:27.467484 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:27.467423    4525 cpu_manager.go:414] "containerMap: removing container" podUID="a29fd997-17ba-4260-8279-05e6f1c6346a" containerName="istio-proxy"
Oct 30 18:48:27.467484 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:27.467428    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a29fd997-17ba-4260-8279-05e6f1c6346a" containerName="istio-proxy"
Oct 30 18:48:27.467638 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:27.467503    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="a29fd997-17ba-4260-8279-05e6f1c6346a" containerName="istio-proxy"
Oct 30 18:48:27.467638 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:27.467516    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f1a2f0a-2bdd-4765-b81c-38323e22fbe9" containerName="payload-pre-processing"
Oct 30 18:48:27.470383 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:27.470365    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t"
Oct 30 18:48:27.499707 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:27.494654    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-2889dd\"/\"kube-root-ca.crt\""
Oct 30 18:48:27.499707 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:27.494775    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-2889dd\"/\"default-dockercfg-ptfkg\""
Oct 30 18:48:27.499707 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:27.494854    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-2889dd\"/\"simulator-w3-2889dd-kserve-self-signed-certs\""
Oct 30 18:48:27.499707 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:27.494943    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-2889dd\"/\"openshift-service-ca.crt\""
Oct 30 18:48:27.533919 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:27.512699    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t"]
Oct 30 18:48:27.545137 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:27.545097    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8b28\" (UniqueName: \"kubernetes.io/projected/424fca14-6d8a-4663-86ec-0c91165f34ee-kube-api-access-d8b28\") pod \"simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t\" (UID: \"424fca14-6d8a-4663-86ec-0c91165f34ee\") " pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t"
Oct 30 18:48:27.545315 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:27.545192    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/424fca14-6d8a-4663-86ec-0c91165f34ee-tmp-dir\") pod \"simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t\" (UID: \"424fca14-6d8a-4663-86ec-0c91165f34ee\") " pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t"
Oct 30 18:48:27.545315 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:27.545244    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/424fca14-6d8a-4663-86ec-0c91165f34ee-dshm\") pod \"simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t\" (UID: \"424fca14-6d8a-4663-86ec-0c91165f34ee\") " pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t"
Oct 30 18:48:27.545315 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:27.545299    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/424fca14-6d8a-4663-86ec-0c91165f34ee-home\") pod \"simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t\" (UID: \"424fca14-6d8a-4663-86ec-0c91165f34ee\") " pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t"
Oct 30 18:48:27.545485 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:27.545395    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/424fca14-6d8a-4663-86ec-0c91165f34ee-model-cache\") pod \"simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t\" (UID: \"424fca14-6d8a-4663-86ec-0c91165f34ee\") " pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t"
Oct 30 18:48:27.545485 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:27.545435    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/424fca14-6d8a-4663-86ec-0c91165f34ee-tls-certs\") pod \"simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t\" (UID: \"424fca14-6d8a-4663-86ec-0c91165f34ee\") " pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t"
Oct 30 18:48:27.646045 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:27.645934    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/424fca14-6d8a-4663-86ec-0c91165f34ee-dshm\") pod \"simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t\" (UID: \"424fca14-6d8a-4663-86ec-0c91165f34ee\") " pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t"
Oct 30 18:48:27.646045 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:27.645982    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/424fca14-6d8a-4663-86ec-0c91165f34ee-home\") pod \"simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t\" (UID: \"424fca14-6d8a-4663-86ec-0c91165f34ee\") " pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t"
Oct 30 18:48:27.646045 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:27.646023    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/424fca14-6d8a-4663-86ec-0c91165f34ee-model-cache\") pod \"simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t\" (UID: \"424fca14-6d8a-4663-86ec-0c91165f34ee\") " pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t"
Oct 30 18:48:27.646045 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:27.646052    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/424fca14-6d8a-4663-86ec-0c91165f34ee-tls-certs\") pod \"simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t\" (UID: \"424fca14-6d8a-4663-86ec-0c91165f34ee\") " pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t"
Oct 30 18:48:27.646388 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:27.646103    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8b28\" (UniqueName: \"kubernetes.io/projected/424fca14-6d8a-4663-86ec-0c91165f34ee-kube-api-access-d8b28\") pod \"simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t\" (UID: \"424fca14-6d8a-4663-86ec-0c91165f34ee\") " pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t"
Oct 30 18:48:27.646388 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:27.646150    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/424fca14-6d8a-4663-86ec-0c91165f34ee-tmp-dir\") pod \"simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t\" (UID: \"424fca14-6d8a-4663-86ec-0c91165f34ee\") " pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t"
Oct 30 18:48:27.646457 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:27.646403    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/424fca14-6d8a-4663-86ec-0c91165f34ee-tmp-dir\") pod \"simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t\" (UID: \"424fca14-6d8a-4663-86ec-0c91165f34ee\") " pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t"
Oct 30 18:48:27.646576 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:27.646555    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/424fca14-6d8a-4663-86ec-0c91165f34ee-model-cache\") pod \"simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t\" (UID: \"424fca14-6d8a-4663-86ec-0c91165f34ee\") " pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t"
Oct 30 18:48:27.646904 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:27.646882    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/424fca14-6d8a-4663-86ec-0c91165f34ee-home\") pod \"simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t\" (UID: \"424fca14-6d8a-4663-86ec-0c91165f34ee\") " pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t"
Oct 30 18:48:27.648372 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:27.648318    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/424fca14-6d8a-4663-86ec-0c91165f34ee-dshm\") pod \"simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t\" (UID: \"424fca14-6d8a-4663-86ec-0c91165f34ee\") " pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t"
Oct 30 18:48:27.648767 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:27.648748    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/424fca14-6d8a-4663-86ec-0c91165f34ee-tls-certs\") pod \"simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t\" (UID: \"424fca14-6d8a-4663-86ec-0c91165f34ee\") " pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t"
Oct 30 18:48:27.670833 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:27.665495    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8b28\" (UniqueName: \"kubernetes.io/projected/424fca14-6d8a-4663-86ec-0c91165f34ee-kube-api-access-d8b28\") pod \"simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t\" (UID: \"424fca14-6d8a-4663-86ec-0c91165f34ee\") " pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t"
Oct 30 18:48:27.742151 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:27.742059    4525 patch_prober.go:29] interesting pod/e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 30 18:48:27.742151 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:27.742123    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j" podUID="6061ede7-981c-4c07-acfc-04be1cae252d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 30 18:48:27.837545 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:27.837518    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t"
Oct 30 18:48:28.167229 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:28.167204    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t"]
Oct 30 18:48:28.168136 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:48:28.168112    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod424fca14_6d8a_4663_86ec_0c91165f34ee.slice/crio-145ce1a5461a2f5fdc34a2adf2210cbca7271df2909bb071632204bfa49f295c WatchSource:0}: Error finding container 145ce1a5461a2f5fdc34a2adf2210cbca7271df2909bb071632204bfa49f295c: Status 404 returned error can't find the container with id 145ce1a5461a2f5fdc34a2adf2210cbca7271df2909bb071632204bfa49f295c
Oct 30 18:48:28.443570 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:28.443485    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t" event={"ID":"424fca14-6d8a-4663-86ec-0c91165f34ee","Type":"ContainerStarted","Data":"3bf2d5d3b7b75971e6e70c2687e5ce91006ba56f8cf2121445ce947b7dfb8d08"}
Oct 30 18:48:28.443570 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:28.443534    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t" event={"ID":"424fca14-6d8a-4663-86ec-0c91165f34ee","Type":"ContainerStarted","Data":"145ce1a5461a2f5fdc34a2adf2210cbca7271df2909bb071632204bfa49f295c"}
Oct 30 18:48:28.464972 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:28.464928    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t" podStartSLOduration=1.464914016 podStartE2EDuration="1.464914016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:48:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:48:28.463076263 +0000 UTC m=+731.088746243" watchObservedRunningTime="2026-10-30 18:48:28.464914016 +0000 UTC m=+731.090583980"
Oct 30 18:48:28.740864 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:28.740784    4525 patch_prober.go:29] interesting pod/e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 30 18:48:28.741332 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:28.741307    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j" podUID="6061ede7-981c-4c07-acfc-04be1cae252d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 30 18:48:28.838449 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:28.838419    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t"
Oct 30 18:48:29.741358 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:29.741324    4525 patch_prober.go:29] interesting pod/e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 30 18:48:29.741879 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:29.741388    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j" podUID="6061ede7-981c-4c07-acfc-04be1cae252d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 30 18:48:30.741294 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:30.741245    4525 patch_prober.go:29] interesting pod/e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 30 18:48:30.741498 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:30.741317    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j" podUID="6061ede7-981c-4c07-acfc-04be1cae252d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 30 18:48:32.327718 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:32.326706    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j"
Oct 30 18:48:32.327718 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:32.327611    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j"
Oct 30 18:48:32.328531 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:32.328471    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j"
Oct 30 18:48:32.364911 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:32.364867    4525 patch_prober.go:29] interesting pod/e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 18:48:32.365120 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:32.364993    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp" podUID="99aac9ad-b8dc-46cf-ac3f-69a814014bd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:48:35.505175 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:35.505141    4525 generic.go:356] "Generic (PLEG): container finished" podID="99aac9ad-b8dc-46cf-ac3f-69a814014bd2" containerID="a10ca651e5feab589cc19423939c8ccde1c0e0ea87a11276221f4529ff26c8d0" exitCode=0
Oct 30 18:48:35.505747 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:35.505301    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp" event={"ID":"99aac9ad-b8dc-46cf-ac3f-69a814014bd2","Type":"ContainerDied","Data":"a10ca651e5feab589cc19423939c8ccde1c0e0ea87a11276221f4529ff26c8d0"}
Oct 30 18:48:35.505826 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:35.505812    4525 scope.go:122] "RemoveContainer" containerID="a10ca651e5feab589cc19423939c8ccde1c0e0ea87a11276221f4529ff26c8d0"
Oct 30 18:48:35.507224 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:35.507195    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:48:35.507291 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:35.507267    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:48:35.507331 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:35.507298    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:48:36.533380 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:36.533329    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp" event={"ID":"99aac9ad-b8dc-46cf-ac3f-69a814014bd2","Type":"ContainerStarted","Data":"491e4e8e384983e5e1bc92aa47ce905b5bd8c20fe88b7dcb3668f37382e7100c"}
Oct 30 18:48:37.058622 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:37.058587    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp"
Oct 30 18:48:37.058907 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:37.058882    4525 patch_prober.go:29] interesting pod/e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 18:48:37.058963 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:37.058941    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp" podUID="99aac9ad-b8dc-46cf-ac3f-69a814014bd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 18:48:37.069833 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:37.069803    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw"]
Oct 30 18:48:37.070272 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:37.070257    4525 cpu_manager.go:414] "containerMap: removing container" podUID="a29fd997-17ba-4260-8279-05e6f1c6346a" containerName="istio-proxy"
Oct 30 18:48:37.070322 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:37.070277    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a29fd997-17ba-4260-8279-05e6f1c6346a" containerName="istio-proxy"
Oct 30 18:48:37.070371 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:37.070360    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="a29fd997-17ba-4260-8279-05e6f1c6346a" containerName="istio-proxy"
Oct 30 18:48:37.073309 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:37.073289    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw"
Oct 30 18:48:37.118543 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:37.118338    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw"]
Oct 30 18:48:37.118742 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:37.118570    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-39bde3\"/\"default-dockercfg-pkwqf\""
Oct 30 18:48:37.118742 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:37.118656    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-39bde3\"/\"simulator-w2-39bde3-kserve-self-signed-certs\""
Oct 30 18:48:37.118878 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:37.118751    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-39bde3\"/\"kube-root-ca.crt\""
Oct 30 18:48:37.118878 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:37.118795    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-39bde3\"/\"openshift-service-ca.crt\""
Oct 30 18:48:37.216751 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:37.216613    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac6852fd-01be-4d69-8c9b-0c591e50b889-tls-certs\") pod \"simulator-w2-39bde3-kserve-58dd74ff99-ld5lw\" (UID: \"ac6852fd-01be-4d69-8c9b-0c591e50b889\") " pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw"
Oct 30 18:48:37.216751 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:37.216661    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csn4h\" (UniqueName: \"kubernetes.io/projected/ac6852fd-01be-4d69-8c9b-0c591e50b889-kube-api-access-csn4h\") pod \"simulator-w2-39bde3-kserve-58dd74ff99-ld5lw\" (UID: \"ac6852fd-01be-4d69-8c9b-0c591e50b889\") " pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw"
Oct 30 18:48:37.217001 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:37.216841    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac6852fd-01be-4d69-8c9b-0c591e50b889-home\") pod \"simulator-w2-39bde3-kserve-58dd74ff99-ld5lw\" (UID: \"ac6852fd-01be-4d69-8c9b-0c591e50b889\") " pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw"
Oct 30 18:48:37.217001 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:37.216896    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac6852fd-01be-4d69-8c9b-0c591e50b889-model-cache\") pod \"simulator-w2-39bde3-kserve-58dd74ff99-ld5lw\" (UID: \"ac6852fd-01be-4d69-8c9b-0c591e50b889\") " pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw"
Oct 30 18:48:37.217001 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:37.216982    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac6852fd-01be-4d69-8c9b-0c591e50b889-tmp-dir\") pod \"simulator-w2-39bde3-kserve-58dd74ff99-ld5lw\" (UID: \"ac6852fd-01be-4d69-8c9b-0c591e50b889\") " pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw"
Oct 30 18:48:37.217166 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:37.217038    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac6852fd-01be-4d69-8c9b-0c591e50b889-dshm\") pod \"simulator-w2-39bde3-kserve-58dd74ff99-ld5lw\" (UID: \"ac6852fd-01be-4d69-8c9b-0c591e50b889\") " pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw"
Oct 30 18:48:37.318510 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:37.318467    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac6852fd-01be-4d69-8c9b-0c591e50b889-tmp-dir\") pod \"simulator-w2-39bde3-kserve-58dd74ff99-ld5lw\" (UID: \"ac6852fd-01be-4d69-8c9b-0c591e50b889\") " pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw"
Oct 30 18:48:37.318720 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:37.318565    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac6852fd-01be-4d69-8c9b-0c591e50b889-dshm\") pod \"simulator-w2-39bde3-kserve-58dd74ff99-ld5lw\" (UID: \"ac6852fd-01be-4d69-8c9b-0c591e50b889\") " pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw"
Oct 30 18:48:37.318720 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:37.318629    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac6852fd-01be-4d69-8c9b-0c591e50b889-tls-certs\") pod \"simulator-w2-39bde3-kserve-58dd74ff99-ld5lw\" (UID: \"ac6852fd-01be-4d69-8c9b-0c591e50b889\") " pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw"
Oct 30 18:48:37.318720 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:37.318657    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csn4h\" (UniqueName: \"kubernetes.io/projected/ac6852fd-01be-4d69-8c9b-0c591e50b889-kube-api-access-csn4h\") pod \"simulator-w2-39bde3-kserve-58dd74ff99-ld5lw\" (UID: \"ac6852fd-01be-4d69-8c9b-0c591e50b889\") " pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw"
Oct 30 18:48:37.318902 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:37.318727    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac6852fd-01be-4d69-8c9b-0c591e50b889-home\") pod \"simulator-w2-39bde3-kserve-58dd74ff99-ld5lw\" (UID: \"ac6852fd-01be-4d69-8c9b-0c591e50b889\") " pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw"
Oct 30 18:48:37.318902 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:37.318757    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac6852fd-01be-4d69-8c9b-0c591e50b889-model-cache\") pod \"simulator-w2-39bde3-kserve-58dd74ff99-ld5lw\" (UID: \"ac6852fd-01be-4d69-8c9b-0c591e50b889\") " pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw"
Oct 30 18:48:37.319238 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:37.319213    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac6852fd-01be-4d69-8c9b-0c591e50b889-model-cache\") pod \"simulator-w2-39bde3-kserve-58dd74ff99-ld5lw\" (UID: \"ac6852fd-01be-4d69-8c9b-0c591e50b889\") " pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw"
Oct 30 18:48:37.319460 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:37.319443    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac6852fd-01be-4d69-8c9b-0c591e50b889-tmp-dir\") pod \"simulator-w2-39bde3-kserve-58dd74ff99-ld5lw\" (UID: \"ac6852fd-01be-4d69-8c9b-0c591e50b889\") " pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw"
Oct 30 18:48:37.320386 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:37.320339    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac6852fd-01be-4d69-8c9b-0c591e50b889-home\") pod \"simulator-w2-39bde3-kserve-58dd74ff99-ld5lw\" (UID: \"ac6852fd-01be-4d69-8c9b-0c591e50b889\") " pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw"
Oct 30 18:48:37.322433 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:37.322407    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac6852fd-01be-4d69-8c9b-0c591e50b889-dshm\") pod \"simulator-w2-39bde3-kserve-58dd74ff99-ld5lw\" (UID: \"ac6852fd-01be-4d69-8c9b-0c591e50b889\") " pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw"
Oct 30 18:48:37.335697 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:37.335260    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac6852fd-01be-4d69-8c9b-0c591e50b889-tls-certs\") pod \"simulator-w2-39bde3-kserve-58dd74ff99-ld5lw\" (UID: \"ac6852fd-01be-4d69-8c9b-0c591e50b889\") " pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw"
Oct 30 18:48:37.337668 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:37.337638    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csn4h\" (UniqueName: \"kubernetes.io/projected/ac6852fd-01be-4d69-8c9b-0c591e50b889-kube-api-access-csn4h\") pod \"simulator-w2-39bde3-kserve-58dd74ff99-ld5lw\" (UID: \"ac6852fd-01be-4d69-8c9b-0c591e50b889\") " pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw"
Oct 30 18:48:37.428607 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:37.428568    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw"
Oct 30 18:48:37.736432 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:37.736406    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw"]
Oct 30 18:48:37.738889 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:48:37.738857    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podac6852fd_01be_4d69_8c9b_0c591e50b889.slice/crio-12350b62126b9d0eed32b1d142c342ca8e20502912b9c9449e84c0ac34290110 WatchSource:0}: Error finding container 12350b62126b9d0eed32b1d142c342ca8e20502912b9c9449e84c0ac34290110: Status 404 returned error can't find the container with id 12350b62126b9d0eed32b1d142c342ca8e20502912b9c9449e84c0ac34290110
Oct 30 18:48:37.837824 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:37.837777    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t"
Oct 30 18:48:37.856014 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:37.855986    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t"
Oct 30 18:48:38.059036 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:38.058917    4525 patch_prober.go:29] interesting pod/e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 18:48:38.059379 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:38.059317    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp" podUID="99aac9ad-b8dc-46cf-ac3f-69a814014bd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 18:48:38.549811 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:38.549770    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw" event={"ID":"ac6852fd-01be-4d69-8c9b-0c591e50b889","Type":"ContainerStarted","Data":"9cb591ec249cb9cc35219afe413c4bec64533b92f0bd53f5dae1c45acb0101f3"}
Oct 30 18:48:38.549811 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:38.549814    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw" event={"ID":"ac6852fd-01be-4d69-8c9b-0c591e50b889","Type":"ContainerStarted","Data":"12350b62126b9d0eed32b1d142c342ca8e20502912b9c9449e84c0ac34290110"}
Oct 30 18:48:38.562911 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:38.562875    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t"
Oct 30 18:48:38.606320 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:38.606257    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw" podStartSLOduration=1.6062361950000001 podStartE2EDuration="1.606236195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:48:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:48:38.597069855 +0000 UTC m=+741.222739820" watchObservedRunningTime="2026-10-30 18:48:38.606236195 +0000 UTC m=+741.231906161"
Oct 30 18:48:39.059125 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:39.059086    4525 patch_prober.go:29] interesting pod/e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 18:48:39.059604 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:39.059145    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp" podUID="99aac9ad-b8dc-46cf-ac3f-69a814014bd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 18:48:39.429618 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:39.429579    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw"
Oct 30 18:48:40.059857 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:40.059819    4525 patch_prober.go:29] interesting pod/e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 18:48:40.060252 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:40.059874    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp" podUID="99aac9ad-b8dc-46cf-ac3f-69a814014bd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 18:48:41.058912 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:41.058878    4525 patch_prober.go:29] interesting pod/e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 18:48:41.061088 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:41.058933    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp" podUID="99aac9ad-b8dc-46cf-ac3f-69a814014bd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 18:48:41.568725 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:41.568674    4525 generic.go:356] "Generic (PLEG): container finished" podID="f236b7d6-975d-447f-bcaa-8e51d6ee0fc9" containerID="d181f6236df07f838dc96fb118ef6348173a3112ca2f7a7930028c1a496140db" exitCode=0
Oct 30 18:48:41.568912 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:41.568721    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-fz9ck" event={"ID":"f236b7d6-975d-447f-bcaa-8e51d6ee0fc9","Type":"ContainerDied","Data":"d181f6236df07f838dc96fb118ef6348173a3112ca2f7a7930028c1a496140db"}
Oct 30 18:48:41.569151 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:41.569137    4525 scope.go:122] "RemoveContainer" containerID="d181f6236df07f838dc96fb118ef6348173a3112ca2f7a7930028c1a496140db"
Oct 30 18:48:41.570453 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:41.570422    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:48:41.570559 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:41.570508    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:48:41.570625 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:41.570557    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:48:42.063465 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:42.063436    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp"
Oct 30 18:48:42.063876 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:42.063711    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp"
Oct 30 18:48:42.064559 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:42.064540    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp"
Oct 30 18:48:42.597707 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:42.597031    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-fz9ck" event={"ID":"f236b7d6-975d-447f-bcaa-8e51d6ee0fc9","Type":"ContainerStarted","Data":"4f207c5afaead434d93f0cb10b449b06795b0d9268e5256d05849de231369eb5"}
Oct 30 18:48:43.261703 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:43.261641    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-fz9ck"
Oct 30 18:48:43.262872 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:43.262845    4525 patch_prober.go:29] interesting pod/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-fz9ck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 18:48:43.262934 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:43.262915    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-fz9ck" podUID="f236b7d6-975d-447f-bcaa-8e51d6ee0fc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 18:48:44.261210 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:44.261178    4525 patch_prober.go:29] interesting pod/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-fz9ck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 18:48:44.261378 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:44.261239    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-fz9ck" podUID="f236b7d6-975d-447f-bcaa-8e51d6ee0fc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 18:48:45.261791 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:45.261752    4525 patch_prober.go:29] interesting pod/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-fz9ck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 18:48:45.262191 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:45.261823    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-fz9ck" podUID="f236b7d6-975d-447f-bcaa-8e51d6ee0fc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 18:48:46.262068 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:46.262029    4525 patch_prober.go:29] interesting pod/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-fz9ck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 18:48:46.262558 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:46.262115    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-fz9ck" podUID="f236b7d6-975d-447f-bcaa-8e51d6ee0fc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 18:48:46.323993 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:46.323956    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-fz9ck"
Oct 30 18:48:47.261789 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:47.261752    4525 patch_prober.go:29] interesting pod/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-fz9ck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 18:48:47.261979 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:47.261807    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-fz9ck" podUID="f236b7d6-975d-447f-bcaa-8e51d6ee0fc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 18:48:47.421913 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:48:47.421877    4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d49l" containerName="extractor"
Oct 30 18:48:47.429102 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:47.429076    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw"
Oct 30 18:48:47.441828 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:47.441803    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw"
Oct 30 18:48:47.642332 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:47.642297    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw"
Oct 30 18:48:48.265750 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:48.265717    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-fz9ck"
Oct 30 18:48:48.266624 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:48.266605    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-fz9ck"
Oct 30 18:48:49.915782 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:49.915737    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv"]
Oct 30 18:48:49.936638 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:49.936613    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv"
Oct 30 18:48:49.937474 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:49.936486    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv"]
Oct 30 18:48:49.946707 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:49.946665    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-2889dd\"/\"premium-w3-2889dd-kserve-self-signed-certs\""
Oct 30 18:48:49.983472 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:49.983430    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/595230e3-55f3-43eb-9a7b-6d89250af9b1-tmp-dir\") pod \"premium-w3-2889dd-kserve-7f95495776-mfmxv\" (UID: \"595230e3-55f3-43eb-9a7b-6d89250af9b1\") " pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv"
Oct 30 18:48:49.983654 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:49.983486    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/595230e3-55f3-43eb-9a7b-6d89250af9b1-tls-certs\") pod \"premium-w3-2889dd-kserve-7f95495776-mfmxv\" (UID: \"595230e3-55f3-43eb-9a7b-6d89250af9b1\") " pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv"
Oct 30 18:48:49.983654 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:49.983618    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/595230e3-55f3-43eb-9a7b-6d89250af9b1-model-cache\") pod \"premium-w3-2889dd-kserve-7f95495776-mfmxv\" (UID: \"595230e3-55f3-43eb-9a7b-6d89250af9b1\") " pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv"
Oct 30 18:48:49.983799 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:49.983776    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5nbk\" (UniqueName: \"kubernetes.io/projected/595230e3-55f3-43eb-9a7b-6d89250af9b1-kube-api-access-c5nbk\") pod \"premium-w3-2889dd-kserve-7f95495776-mfmxv\" (UID: \"595230e3-55f3-43eb-9a7b-6d89250af9b1\") " pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv"
Oct 30 18:48:49.983873 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:49.983854    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/595230e3-55f3-43eb-9a7b-6d89250af9b1-dshm\") pod \"premium-w3-2889dd-kserve-7f95495776-mfmxv\" (UID: \"595230e3-55f3-43eb-9a7b-6d89250af9b1\") " pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv"
Oct 30 18:48:49.983929 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:49.983914    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/595230e3-55f3-43eb-9a7b-6d89250af9b1-home\") pod \"premium-w3-2889dd-kserve-7f95495776-mfmxv\" (UID: \"595230e3-55f3-43eb-9a7b-6d89250af9b1\") " pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv"
Oct 30 18:48:50.084644 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:50.084587    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/595230e3-55f3-43eb-9a7b-6d89250af9b1-tmp-dir\") pod \"premium-w3-2889dd-kserve-7f95495776-mfmxv\" (UID: \"595230e3-55f3-43eb-9a7b-6d89250af9b1\") " pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv"
Oct 30 18:48:50.084644 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:50.084648    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/595230e3-55f3-43eb-9a7b-6d89250af9b1-tls-certs\") pod \"premium-w3-2889dd-kserve-7f95495776-mfmxv\" (UID: \"595230e3-55f3-43eb-9a7b-6d89250af9b1\") " pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv"
Oct 30 18:48:50.084922 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:50.084741    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/595230e3-55f3-43eb-9a7b-6d89250af9b1-model-cache\") pod \"premium-w3-2889dd-kserve-7f95495776-mfmxv\" (UID: \"595230e3-55f3-43eb-9a7b-6d89250af9b1\") " pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv"
Oct 30 18:48:50.084922 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:50.084833    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5nbk\" (UniqueName: \"kubernetes.io/projected/595230e3-55f3-43eb-9a7b-6d89250af9b1-kube-api-access-c5nbk\") pod \"premium-w3-2889dd-kserve-7f95495776-mfmxv\" (UID: \"595230e3-55f3-43eb-9a7b-6d89250af9b1\") " pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv"
Oct 30 18:48:50.084922 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:50.084862    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/595230e3-55f3-43eb-9a7b-6d89250af9b1-dshm\") pod \"premium-w3-2889dd-kserve-7f95495776-mfmxv\" (UID: \"595230e3-55f3-43eb-9a7b-6d89250af9b1\") " pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv"
Oct 30 18:48:50.084922 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:50.084910    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/595230e3-55f3-43eb-9a7b-6d89250af9b1-home\") pod \"premium-w3-2889dd-kserve-7f95495776-mfmxv\" (UID: \"595230e3-55f3-43eb-9a7b-6d89250af9b1\") " pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv"
Oct 30 18:48:50.085144 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:50.085062    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/595230e3-55f3-43eb-9a7b-6d89250af9b1-tmp-dir\") pod \"premium-w3-2889dd-kserve-7f95495776-mfmxv\" (UID: \"595230e3-55f3-43eb-9a7b-6d89250af9b1\") " pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv"
Oct 30 18:48:50.085298 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:50.085273    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/595230e3-55f3-43eb-9a7b-6d89250af9b1-home\") pod \"premium-w3-2889dd-kserve-7f95495776-mfmxv\" (UID: \"595230e3-55f3-43eb-9a7b-6d89250af9b1\") " pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv"
Oct 30 18:48:50.085463 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:50.085320    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/595230e3-55f3-43eb-9a7b-6d89250af9b1-model-cache\") pod \"premium-w3-2889dd-kserve-7f95495776-mfmxv\" (UID: \"595230e3-55f3-43eb-9a7b-6d89250af9b1\") " pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv"
Oct 30 18:48:50.093639 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:50.093609    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/595230e3-55f3-43eb-9a7b-6d89250af9b1-dshm\") pod \"premium-w3-2889dd-kserve-7f95495776-mfmxv\" (UID: \"595230e3-55f3-43eb-9a7b-6d89250af9b1\") " pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv"
Oct 30 18:48:50.097362 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:50.097300    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5nbk\" (UniqueName: \"kubernetes.io/projected/595230e3-55f3-43eb-9a7b-6d89250af9b1-kube-api-access-c5nbk\") pod \"premium-w3-2889dd-kserve-7f95495776-mfmxv\" (UID: \"595230e3-55f3-43eb-9a7b-6d89250af9b1\") " pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv"
Oct 30 18:48:50.098136 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:50.098115    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/595230e3-55f3-43eb-9a7b-6d89250af9b1-tls-certs\") pod \"premium-w3-2889dd-kserve-7f95495776-mfmxv\" (UID: \"595230e3-55f3-43eb-9a7b-6d89250af9b1\") " pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv"
Oct 30 18:48:50.255609 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:50.255508    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv"
Oct 30 18:48:50.579769 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:50.579735    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv"]
Oct 30 18:48:50.580219 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:48:50.580195    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod595230e3_55f3_43eb_9a7b_6d89250af9b1.slice/crio-764ea897293e1a4e9835fdd389457a033a0af866b02c30786620a9466241365e WatchSource:0}: Error finding container 764ea897293e1a4e9835fdd389457a033a0af866b02c30786620a9466241365e: Status 404 returned error can't find the container with id 764ea897293e1a4e9835fdd389457a033a0af866b02c30786620a9466241365e
Oct 30 18:48:50.658770 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:50.658728    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv" event={"ID":"595230e3-55f3-43eb-9a7b-6d89250af9b1","Type":"ContainerStarted","Data":"1f987361694e0e8596a7c31276cf9bbd2b27d8df7ae6928a9cce99c8c1dd3846"}
Oct 30 18:48:50.658921 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:50.658800    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv" event={"ID":"595230e3-55f3-43eb-9a7b-6d89250af9b1","Type":"ContainerStarted","Data":"764ea897293e1a4e9835fdd389457a033a0af866b02c30786620a9466241365e"}
Oct 30 18:48:50.678333 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:50.678276    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv" podStartSLOduration=1.678257517 podStartE2EDuration="1.678257517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:48:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:48:50.675536943 +0000 UTC m=+753.301206906" watchObservedRunningTime="2026-10-30 18:48:50.678257517 +0000 UTC m=+753.303927482"
Oct 30 18:48:51.256373 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:51.256329    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv"
Oct 30 18:48:56.448767 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:56.448730    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr"]
Oct 30 18:48:56.454995 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:56.454969    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr"
Oct 30 18:48:56.460803 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:56.460772    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-39bde3\"/\"premium-w2-39bde3-kserve-self-signed-certs\""
Oct 30 18:48:56.463973 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:56.463931    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr"]
Oct 30 18:48:56.467363 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:56.467329    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82bbbf85-ed93-48d2-9f17-49ac1f31c0a8-model-cache\") pod \"premium-w2-39bde3-kserve-8448cb9bdb-46npr\" (UID: \"82bbbf85-ed93-48d2-9f17-49ac1f31c0a8\") " pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr"
Oct 30 18:48:56.467478 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:56.467411    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82bbbf85-ed93-48d2-9f17-49ac1f31c0a8-tls-certs\") pod \"premium-w2-39bde3-kserve-8448cb9bdb-46npr\" (UID: \"82bbbf85-ed93-48d2-9f17-49ac1f31c0a8\") " pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr"
Oct 30 18:48:56.467543 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:56.467528    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82bbbf85-ed93-48d2-9f17-49ac1f31c0a8-tmp-dir\") pod \"premium-w2-39bde3-kserve-8448cb9bdb-46npr\" (UID: \"82bbbf85-ed93-48d2-9f17-49ac1f31c0a8\") " pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr"
Oct 30 18:48:56.467587 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:56.467556    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rf92\" (UniqueName: \"kubernetes.io/projected/82bbbf85-ed93-48d2-9f17-49ac1f31c0a8-kube-api-access-6rf92\") pod \"premium-w2-39bde3-kserve-8448cb9bdb-46npr\" (UID: \"82bbbf85-ed93-48d2-9f17-49ac1f31c0a8\") " pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr"
Oct 30 18:48:56.467645 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:56.467585    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82bbbf85-ed93-48d2-9f17-49ac1f31c0a8-home\") pod \"premium-w2-39bde3-kserve-8448cb9bdb-46npr\" (UID: \"82bbbf85-ed93-48d2-9f17-49ac1f31c0a8\") " pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr"
Oct 30 18:48:56.467823 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:56.467802    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82bbbf85-ed93-48d2-9f17-49ac1f31c0a8-dshm\") pod \"premium-w2-39bde3-kserve-8448cb9bdb-46npr\" (UID: \"82bbbf85-ed93-48d2-9f17-49ac1f31c0a8\") " pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr"
Oct 30 18:48:56.569330 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:56.569292    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82bbbf85-ed93-48d2-9f17-49ac1f31c0a8-tmp-dir\") pod \"premium-w2-39bde3-kserve-8448cb9bdb-46npr\" (UID: \"82bbbf85-ed93-48d2-9f17-49ac1f31c0a8\") " pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr"
Oct 30 18:48:56.569330 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:56.569331    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rf92\" (UniqueName: \"kubernetes.io/projected/82bbbf85-ed93-48d2-9f17-49ac1f31c0a8-kube-api-access-6rf92\") pod \"premium-w2-39bde3-kserve-8448cb9bdb-46npr\" (UID: \"82bbbf85-ed93-48d2-9f17-49ac1f31c0a8\") " pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr"
Oct 30 18:48:56.569568 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:56.569359    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82bbbf85-ed93-48d2-9f17-49ac1f31c0a8-home\") pod \"premium-w2-39bde3-kserve-8448cb9bdb-46npr\" (UID: \"82bbbf85-ed93-48d2-9f17-49ac1f31c0a8\") " pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr"
Oct 30 18:48:56.569568 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:56.569447    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82bbbf85-ed93-48d2-9f17-49ac1f31c0a8-dshm\") pod \"premium-w2-39bde3-kserve-8448cb9bdb-46npr\" (UID: \"82bbbf85-ed93-48d2-9f17-49ac1f31c0a8\") " pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr"
Oct 30 18:48:56.569568 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:56.569505    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82bbbf85-ed93-48d2-9f17-49ac1f31c0a8-model-cache\") pod \"premium-w2-39bde3-kserve-8448cb9bdb-46npr\" (UID: \"82bbbf85-ed93-48d2-9f17-49ac1f31c0a8\") " pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr"
Oct 30 18:48:56.569568 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:56.569553    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82bbbf85-ed93-48d2-9f17-49ac1f31c0a8-tls-certs\") pod \"premium-w2-39bde3-kserve-8448cb9bdb-46npr\" (UID: \"82bbbf85-ed93-48d2-9f17-49ac1f31c0a8\") " pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr"
Oct 30 18:48:56.569807 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:56.569782    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82bbbf85-ed93-48d2-9f17-49ac1f31c0a8-home\") pod \"premium-w2-39bde3-kserve-8448cb9bdb-46npr\" (UID: \"82bbbf85-ed93-48d2-9f17-49ac1f31c0a8\") " pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr"
Oct 30 18:48:56.569870 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:56.569816    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82bbbf85-ed93-48d2-9f17-49ac1f31c0a8-tmp-dir\") pod \"premium-w2-39bde3-kserve-8448cb9bdb-46npr\" (UID: \"82bbbf85-ed93-48d2-9f17-49ac1f31c0a8\") " pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr"
Oct 30 18:48:56.569974 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:56.569955    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82bbbf85-ed93-48d2-9f17-49ac1f31c0a8-model-cache\") pod \"premium-w2-39bde3-kserve-8448cb9bdb-46npr\" (UID: \"82bbbf85-ed93-48d2-9f17-49ac1f31c0a8\") " pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr"
Oct 30 18:48:56.571891 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:56.571865    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82bbbf85-ed93-48d2-9f17-49ac1f31c0a8-dshm\") pod \"premium-w2-39bde3-kserve-8448cb9bdb-46npr\" (UID: \"82bbbf85-ed93-48d2-9f17-49ac1f31c0a8\") " pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr"
Oct 30 18:48:56.572142 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:56.572125    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82bbbf85-ed93-48d2-9f17-49ac1f31c0a8-tls-certs\") pod \"premium-w2-39bde3-kserve-8448cb9bdb-46npr\" (UID: \"82bbbf85-ed93-48d2-9f17-49ac1f31c0a8\") " pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr"
Oct 30 18:48:56.585446 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:56.585420    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rf92\" (UniqueName: \"kubernetes.io/projected/82bbbf85-ed93-48d2-9f17-49ac1f31c0a8-kube-api-access-6rf92\") pod \"premium-w2-39bde3-kserve-8448cb9bdb-46npr\" (UID: \"82bbbf85-ed93-48d2-9f17-49ac1f31c0a8\") " pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr"
Oct 30 18:48:56.770445 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:56.770354    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr"
Oct 30 18:48:57.048634 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:57.048610    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr"]
Oct 30 18:48:57.050329 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:48:57.050300    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod82bbbf85_ed93_48d2_9f17_49ac1f31c0a8.slice/crio-c34e3b4b2e73cedde3d7ce39e0a90a06d0e5d68478ef285a3ec7812e36fd3f8a WatchSource:0}: Error finding container c34e3b4b2e73cedde3d7ce39e0a90a06d0e5d68478ef285a3ec7812e36fd3f8a: Status 404 returned error can't find the container with id c34e3b4b2e73cedde3d7ce39e0a90a06d0e5d68478ef285a3ec7812e36fd3f8a
Oct 30 18:48:57.703655 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:57.703622    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr" event={"ID":"82bbbf85-ed93-48d2-9f17-49ac1f31c0a8","Type":"ContainerStarted","Data":"eb4eecd9e9dc3d03699dafc4fea9e08845b6c8c2814a546b34d8b9de9d79e894"}
Oct 30 18:48:57.704105 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:57.703668    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr" event={"ID":"82bbbf85-ed93-48d2-9f17-49ac1f31c0a8","Type":"ContainerStarted","Data":"c34e3b4b2e73cedde3d7ce39e0a90a06d0e5d68478ef285a3ec7812e36fd3f8a"}
Oct 30 18:48:57.728269 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:57.728221    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr" podStartSLOduration=1.728208295 podStartE2EDuration="1.728208295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:48:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:48:57.728074058 +0000 UTC m=+760.353744021" watchObservedRunningTime="2026-10-30 18:48:57.728208295 +0000 UTC m=+760.353878259"
Oct 30 18:48:57.771293 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:48:57.771263    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr"
Oct 30 18:49:00.255895 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:00.255801    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv"
Oct 30 18:49:00.271706 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:00.271666    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv"
Oct 30 18:49:00.738228 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:00.738196    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv"
Oct 30 18:49:06.771295 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:06.771250    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr"
Oct 30 18:49:06.785076 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:06.785051    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr"
Oct 30 18:49:07.771805 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:07.771772    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr"
Oct 30 18:49:12.974081 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:12.974037    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts"]
Oct 30 18:49:12.977883 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:12.977861    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts"
Oct 30 18:49:12.992407 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:12.992376    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts"]
Oct 30 18:49:12.992537 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:12.992475    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-85cb41\"/\"openshift-service-ca.crt\""
Oct 30 18:49:12.992583 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:12.992555    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-85cb41\"/\"kube-root-ca.crt\""
Oct 30 18:49:12.992720 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:12.992705    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-85cb41\"/\"premium-w1-85cb41-kserve-self-signed-certs\""
Oct 30 18:49:12.992799 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:12.992759    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-85cb41\"/\"default-dockercfg-pfnh7\""
Oct 30 18:49:13.050844 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:13.050808    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be6d4969-285b-4da0-883e-4847f2cbf3af-model-cache\") pod \"premium-w1-85cb41-kserve-86b576cb74-7zzts\" (UID: \"be6d4969-285b-4da0-883e-4847f2cbf3af\") " pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts"
Oct 30 18:49:13.051031 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:13.050942    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42ksl\" (UniqueName: \"kubernetes.io/projected/be6d4969-285b-4da0-883e-4847f2cbf3af-kube-api-access-42ksl\") pod \"premium-w1-85cb41-kserve-86b576cb74-7zzts\" (UID: \"be6d4969-285b-4da0-883e-4847f2cbf3af\") " pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts"
Oct 30 18:49:13.051104 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:13.051047    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be6d4969-285b-4da0-883e-4847f2cbf3af-dshm\") pod \"premium-w1-85cb41-kserve-86b576cb74-7zzts\" (UID: \"be6d4969-285b-4da0-883e-4847f2cbf3af\") " pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts"
Oct 30 18:49:13.051166 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:13.051099    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be6d4969-285b-4da0-883e-4847f2cbf3af-tls-certs\") pod \"premium-w1-85cb41-kserve-86b576cb74-7zzts\" (UID: \"be6d4969-285b-4da0-883e-4847f2cbf3af\") " pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts"
Oct 30 18:49:13.051166 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:13.051135    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be6d4969-285b-4da0-883e-4847f2cbf3af-home\") pod \"premium-w1-85cb41-kserve-86b576cb74-7zzts\" (UID: \"be6d4969-285b-4da0-883e-4847f2cbf3af\") " pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts"
Oct 30 18:49:13.051234 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:13.051200    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be6d4969-285b-4da0-883e-4847f2cbf3af-tmp-dir\") pod \"premium-w1-85cb41-kserve-86b576cb74-7zzts\" (UID: \"be6d4969-285b-4da0-883e-4847f2cbf3af\") " pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts"
Oct 30 18:49:13.152213 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:13.152175    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42ksl\" (UniqueName: \"kubernetes.io/projected/be6d4969-285b-4da0-883e-4847f2cbf3af-kube-api-access-42ksl\") pod \"premium-w1-85cb41-kserve-86b576cb74-7zzts\" (UID: \"be6d4969-285b-4da0-883e-4847f2cbf3af\") " pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts"
Oct 30 18:49:13.152393 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:13.152238    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be6d4969-285b-4da0-883e-4847f2cbf3af-dshm\") pod \"premium-w1-85cb41-kserve-86b576cb74-7zzts\" (UID: \"be6d4969-285b-4da0-883e-4847f2cbf3af\") " pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts"
Oct 30 18:49:13.152393 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:13.152286    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be6d4969-285b-4da0-883e-4847f2cbf3af-tls-certs\") pod \"premium-w1-85cb41-kserve-86b576cb74-7zzts\" (UID: \"be6d4969-285b-4da0-883e-4847f2cbf3af\") " pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts"
Oct 30 18:49:13.152393 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:13.152317    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be6d4969-285b-4da0-883e-4847f2cbf3af-home\") pod \"premium-w1-85cb41-kserve-86b576cb74-7zzts\" (UID: \"be6d4969-285b-4da0-883e-4847f2cbf3af\") " pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts"
Oct 30 18:49:13.152591 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:13.152458    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be6d4969-285b-4da0-883e-4847f2cbf3af-tmp-dir\") pod \"premium-w1-85cb41-kserve-86b576cb74-7zzts\" (UID: \"be6d4969-285b-4da0-883e-4847f2cbf3af\") " pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts"
Oct 30 18:49:13.152591 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:13.152555    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be6d4969-285b-4da0-883e-4847f2cbf3af-model-cache\") pod \"premium-w1-85cb41-kserve-86b576cb74-7zzts\" (UID: \"be6d4969-285b-4da0-883e-4847f2cbf3af\") " pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts"
Oct 30 18:49:13.152767 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:13.152746    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be6d4969-285b-4da0-883e-4847f2cbf3af-home\") pod \"premium-w1-85cb41-kserve-86b576cb74-7zzts\" (UID: \"be6d4969-285b-4da0-883e-4847f2cbf3af\") " pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts"
Oct 30 18:49:13.152833 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:13.152803    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be6d4969-285b-4da0-883e-4847f2cbf3af-tmp-dir\") pod \"premium-w1-85cb41-kserve-86b576cb74-7zzts\" (UID: \"be6d4969-285b-4da0-883e-4847f2cbf3af\") " pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts"
Oct 30 18:49:13.152932 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:13.152912    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be6d4969-285b-4da0-883e-4847f2cbf3af-model-cache\") pod \"premium-w1-85cb41-kserve-86b576cb74-7zzts\" (UID: \"be6d4969-285b-4da0-883e-4847f2cbf3af\") " pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts"
Oct 30 18:49:13.154759 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:13.154734    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be6d4969-285b-4da0-883e-4847f2cbf3af-dshm\") pod \"premium-w1-85cb41-kserve-86b576cb74-7zzts\" (UID: \"be6d4969-285b-4da0-883e-4847f2cbf3af\") " pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts"
Oct 30 18:49:13.154993 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:13.154976    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be6d4969-285b-4da0-883e-4847f2cbf3af-tls-certs\") pod \"premium-w1-85cb41-kserve-86b576cb74-7zzts\" (UID: \"be6d4969-285b-4da0-883e-4847f2cbf3af\") " pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts"
Oct 30 18:49:13.167771 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:13.167748    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42ksl\" (UniqueName: \"kubernetes.io/projected/be6d4969-285b-4da0-883e-4847f2cbf3af-kube-api-access-42ksl\") pod \"premium-w1-85cb41-kserve-86b576cb74-7zzts\" (UID: \"be6d4969-285b-4da0-883e-4847f2cbf3af\") " pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts"
Oct 30 18:49:13.176755 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:13.176733    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-qv6r8"]
Oct 30 18:49:13.180648 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:13.180631    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-qv6r8"
Oct 30 18:49:13.189888 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:13.189869    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-9de872-openshift-default-dockercfg-9bf8l\""
Oct 30 18:49:13.206889 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:13.206864    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-qv6r8"]
Oct 30 18:49:13.254055 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:13.253992    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b-istio-podinfo\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-qv6r8\" (UID: \"36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-qv6r8"
Oct 30 18:49:13.254055 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:13.254024    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b-credential-socket\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-qv6r8\" (UID: \"36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-qv6r8"
Oct 30 18:49:13.254055 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:13.254055    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b-istio-envoy\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-qv6r8\" (UID: \"36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-qv6r8"
Oct 30 18:49:13.254273 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:13.254162    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b-istio-data\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-qv6r8\" (UID: \"36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-qv6r8"
Oct 30 18:49:13.254273 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:13.254183    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtpbq\" (UniqueName: \"kubernetes.io/projected/36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b-kube-api-access-jtpbq\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-qv6r8\" (UID: \"36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-qv6r8"
Oct 30 18:49:13.254273 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:13.254202    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b-istio-token\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-qv6r8\" (UID: \"36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-qv6r8"
Oct 30 18:49:13.254389 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:13.254278    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b-workload-certs\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-qv6r8\" (UID: \"36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-qv6r8"
Oct 30 18:49:13.254389 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:13.254334    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b-workload-socket\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-qv6r8\" (UID: \"36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-qv6r8"
Oct 30 18:49:13.254389 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:13.254369    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b-istiod-ca-cert\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-qv6r8\" (UID: \"36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-qv6r8"
Oct 30 18:49:13.291271 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:13.291243    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts"
Oct 30 18:49:13.356484 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:13.355332    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b-istio-podinfo\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-qv6r8\" (UID: \"36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-qv6r8"
Oct 30 18:49:13.356484 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:13.355370    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b-credential-socket\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-qv6r8\" (UID: \"36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-qv6r8"
Oct 30 18:49:13.356484 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:13.355406    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b-istio-envoy\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-qv6r8\" (UID: \"36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-qv6r8"
Oct 30 18:49:13.356484 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:13.355426    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b-istio-data\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-qv6r8\" (UID: \"36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-qv6r8"
Oct 30 18:49:13.356484 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:13.355442    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtpbq\" (UniqueName: \"kubernetes.io/projected/36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b-kube-api-access-jtpbq\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-qv6r8\" (UID: \"36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-qv6r8"
Oct 30 18:49:13.356484 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:13.355463    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b-istio-token\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-qv6r8\" (UID: \"36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-qv6r8"
Oct 30 18:49:13.356484 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:13.355507    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b-workload-certs\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-qv6r8\" (UID: \"36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-qv6r8"
Oct 30 18:49:13.356484 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:13.355536    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b-workload-socket\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-qv6r8\" (UID: \"36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-qv6r8"
Oct 30 18:49:13.356484 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:13.355569    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b-istiod-ca-cert\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-qv6r8\" (UID: \"36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-qv6r8"
Oct 30 18:49:13.356484 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:13.356074    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b-istio-data\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-qv6r8\" (UID: \"36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-qv6r8"
Oct 30 18:49:13.356484 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:13.356389    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b-workload-certs\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-qv6r8\" (UID: \"36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-qv6r8"
Oct 30 18:49:13.356484 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:13.356422    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b-istiod-ca-cert\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-qv6r8\" (UID: \"36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-qv6r8"
Oct 30 18:49:13.357204 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:13.356621    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b-workload-socket\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-qv6r8\" (UID: \"36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-qv6r8"
Oct 30 18:49:13.357204 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:13.356918    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b-credential-socket\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-qv6r8\" (UID: \"36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-qv6r8"
Oct 30 18:49:13.360216 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:13.360192    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b-istio-envoy\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-qv6r8\" (UID: \"36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-qv6r8"
Oct 30 18:49:13.361755 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:13.361733    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b-istio-podinfo\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-qv6r8\" (UID: \"36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-qv6r8"
Oct 30 18:49:13.364233 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:13.364207    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtpbq\" (UniqueName: \"kubernetes.io/projected/36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b-kube-api-access-jtpbq\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-qv6r8\" (UID: \"36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-qv6r8"
Oct 30 18:49:13.364233 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:13.364219    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b-istio-token\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-qv6r8\" (UID: \"36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-qv6r8"
Oct 30 18:49:13.494136 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:13.494098    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-qv6r8"
Oct 30 18:49:13.572006 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:13.571980    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts"]
Oct 30 18:49:13.577205 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:49:13.577182    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbe6d4969_285b_4da0_883e_4847f2cbf3af.slice/crio-06f0935f6ac358ac4bb171a70bf8b1e6097ff634cc5e95143fea57091d6523db WatchSource:0}: Error finding container 06f0935f6ac358ac4bb171a70bf8b1e6097ff634cc5e95143fea57091d6523db: Status 404 returned error can't find the container with id 06f0935f6ac358ac4bb171a70bf8b1e6097ff634cc5e95143fea57091d6523db
Oct 30 18:49:13.792298 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:13.792267    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-qv6r8"]
Oct 30 18:49:13.792431 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:49:13.792320    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36c2a8f4_b96a_4b4c_a996_e97e2fe6c90b.slice/crio-2f147dfbb3333d054aa4a906b2f5dae5695193bcd99649cb391c88dd3430af3a WatchSource:0}: Error finding container 2f147dfbb3333d054aa4a906b2f5dae5695193bcd99649cb391c88dd3430af3a: Status 404 returned error can't find the container with id 2f147dfbb3333d054aa4a906b2f5dae5695193bcd99649cb391c88dd3430af3a
Oct 30 18:49:13.794753 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:13.794714    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:49:13.794857 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:13.794784    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:49:13.794857 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:13.794827    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:49:13.801445 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:13.801415    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts" event={"ID":"be6d4969-285b-4da0-883e-4847f2cbf3af","Type":"ContainerStarted","Data":"8e453ac3401f70c8b0ba71c23f7db0ce5e80f6dc9e49daa3ff4ab12da9b514f0"}
Oct 30 18:49:13.801548 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:13.801454    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts" event={"ID":"be6d4969-285b-4da0-883e-4847f2cbf3af","Type":"ContainerStarted","Data":"06f0935f6ac358ac4bb171a70bf8b1e6097ff634cc5e95143fea57091d6523db"}
Oct 30 18:49:13.803083 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:13.803057    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-qv6r8" event={"ID":"36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b","Type":"ContainerStarted","Data":"2f147dfbb3333d054aa4a906b2f5dae5695193bcd99649cb391c88dd3430af3a"}
Oct 30 18:49:13.830794 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:13.830751    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts" podStartSLOduration=1.830736454 podStartE2EDuration="1.830736454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:49:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:49:13.826721884 +0000 UTC m=+776.452391850" watchObservedRunningTime="2026-10-30 18:49:13.830736454 +0000 UTC m=+776.456406417"
Oct 30 18:49:14.291380 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:14.291342    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts"
Oct 30 18:49:14.815835 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:14.815799    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-qv6r8" event={"ID":"36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b","Type":"ContainerStarted","Data":"32ecebc6ea5d4f7b9a8c28cd0fe1175887c16eabc9e5ce09032bd44d03278131"}
Oct 30 18:49:14.847177 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:14.847118    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-qv6r8" podStartSLOduration=1.847105849 podStartE2EDuration="1.847105849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:49:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:49:14.844152079 +0000 UTC m=+777.469822044" watchObservedRunningTime="2026-10-30 18:49:14.847105849 +0000 UTC m=+777.472775812"
Oct 30 18:49:15.497735 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:15.496593    4525 patch_prober.go:29] interesting pod/e2e-worker-w0-9de872-openshift-default-77cf8df555-qv6r8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 18:49:15.497735 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:15.496667    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-qv6r8" podUID="36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 18:49:15.497735 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:15.494851    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-qv6r8"
Oct 30 18:49:16.496156 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:16.496119    4525 patch_prober.go:29] interesting pod/e2e-worker-w0-9de872-openshift-default-77cf8df555-qv6r8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 18:49:16.496480 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:16.496452    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-qv6r8" podUID="36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 18:49:17.427016 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:49:17.426976    4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d49l" containerName="extractor"
Oct 30 18:49:17.497113 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:17.497076    4525 patch_prober.go:29] interesting pod/e2e-worker-w0-9de872-openshift-default-77cf8df555-qv6r8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 18:49:17.497385 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:17.497357    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-qv6r8" podUID="36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 18:49:18.494706 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:18.494445    4525 patch_prober.go:29] interesting pod/e2e-worker-w0-9de872-openshift-default-77cf8df555-qv6r8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 18:49:18.494706 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:18.494505    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-qv6r8" podUID="36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 18:49:19.496119 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:19.496079    4525 patch_prober.go:29] interesting pod/e2e-worker-w0-9de872-openshift-default-77cf8df555-qv6r8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 18:49:19.496756 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:19.496720    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-qv6r8" podUID="36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 18:49:20.665527 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:20.665486    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-qv6r8"
Oct 30 18:49:20.666902 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:20.666878    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-qv6r8"
Oct 30 18:49:20.669595 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:20.669571    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-qv6r8"
Oct 30 18:49:23.291540 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:23.291505    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts"
Oct 30 18:49:23.303854 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:23.303831    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts"
Oct 30 18:49:23.897830 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:23.897803    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts"
Oct 30 18:49:33.765643 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:33.765610    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7cfca6cc-bhbpv"]
Oct 30 18:49:33.769447 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:33.769426    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cfca6cc-bhbpv"
Oct 30 18:49:33.777708 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:33.777658    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7cfca6cc-bhbpv"]
Oct 30 18:49:33.778237 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:33.778214    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:49:33.778341 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:33.778214    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94hcn\""
Oct 30 18:49:33.841442 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:33.841413    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35a6c3d1-e874-4203-aa17-d48e3a99e9ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7cfca6cc-bhbpv\" (UID: \"35a6c3d1-e874-4203-aa17-d48e3a99e9ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cfca6cc-bhbpv"
Oct 30 18:49:33.841603 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:33.841542    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gcg4\" (UniqueName: \"kubernetes.io/projected/35a6c3d1-e874-4203-aa17-d48e3a99e9ee-kube-api-access-6gcg4\") pod \"maas-api-revoke-sub-7cfca6cc-bhbpv\" (UID: \"35a6c3d1-e874-4203-aa17-d48e3a99e9ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cfca6cc-bhbpv"
Oct 30 18:49:33.841603 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:33.841572    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35a6c3d1-e874-4203-aa17-d48e3a99e9ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7cfca6cc-bhbpv\" (UID: \"35a6c3d1-e874-4203-aa17-d48e3a99e9ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cfca6cc-bhbpv"
Oct 30 18:49:33.942811 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:33.942764    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35a6c3d1-e874-4203-aa17-d48e3a99e9ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7cfca6cc-bhbpv\" (UID: \"35a6c3d1-e874-4203-aa17-d48e3a99e9ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cfca6cc-bhbpv"
Oct 30 18:49:33.942979 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:33.942924    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gcg4\" (UniqueName: \"kubernetes.io/projected/35a6c3d1-e874-4203-aa17-d48e3a99e9ee-kube-api-access-6gcg4\") pod \"maas-api-revoke-sub-7cfca6cc-bhbpv\" (UID: \"35a6c3d1-e874-4203-aa17-d48e3a99e9ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cfca6cc-bhbpv"
Oct 30 18:49:33.942979 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:33.942969    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35a6c3d1-e874-4203-aa17-d48e3a99e9ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7cfca6cc-bhbpv\" (UID: \"35a6c3d1-e874-4203-aa17-d48e3a99e9ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cfca6cc-bhbpv"
Oct 30 18:49:33.943588 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:33.943562    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35a6c3d1-e874-4203-aa17-d48e3a99e9ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7cfca6cc-bhbpv\" (UID: \"35a6c3d1-e874-4203-aa17-d48e3a99e9ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cfca6cc-bhbpv"
Oct 30 18:49:33.945381 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:33.945355    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35a6c3d1-e874-4203-aa17-d48e3a99e9ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7cfca6cc-bhbpv\" (UID: \"35a6c3d1-e874-4203-aa17-d48e3a99e9ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cfca6cc-bhbpv"
Oct 30 18:49:33.951261 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:33.951236    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gcg4\" (UniqueName: \"kubernetes.io/projected/35a6c3d1-e874-4203-aa17-d48e3a99e9ee-kube-api-access-6gcg4\") pod \"maas-api-revoke-sub-7cfca6cc-bhbpv\" (UID: \"35a6c3d1-e874-4203-aa17-d48e3a99e9ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cfca6cc-bhbpv"
Oct 30 18:49:34.095943 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:34.095846    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cfca6cc-bhbpv"
Oct 30 18:49:34.551644 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:34.551601    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7cfca6cc-bhbpv"]
Oct 30 18:49:34.573581 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:49:34.572158    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35a6c3d1_e874_4203_aa17_d48e3a99e9ee.slice/crio-987b64ff26b91a3e7c3fbc9be2545145dee65d026d542568deb493aa13b64c18 WatchSource:0}: Error finding container 987b64ff26b91a3e7c3fbc9be2545145dee65d026d542568deb493aa13b64c18: Status 404 returned error can't find the container with id 987b64ff26b91a3e7c3fbc9be2545145dee65d026d542568deb493aa13b64c18
Oct 30 18:49:35.013757 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:35.012722    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cfca6cc-bhbpv" event={"ID":"35a6c3d1-e874-4203-aa17-d48e3a99e9ee","Type":"ContainerStarted","Data":"987b64ff26b91a3e7c3fbc9be2545145dee65d026d542568deb493aa13b64c18"}
Oct 30 18:49:36.021025 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:36.020989    4525 generic.go:356] "Generic (PLEG): container finished" podID="35a6c3d1-e874-4203-aa17-d48e3a99e9ee" containerID="bf8247c0c864f9d6bca294d99482dd416e6aac8f2de4e58f4390d7810b4db031" exitCode=0
Oct 30 18:49:36.021025 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:36.021030    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cfca6cc-bhbpv" event={"ID":"35a6c3d1-e874-4203-aa17-d48e3a99e9ee","Type":"ContainerDied","Data":"bf8247c0c864f9d6bca294d99482dd416e6aac8f2de4e58f4390d7810b4db031"}
Oct 30 18:49:37.163320 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:37.163296    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cfca6cc-bhbpv"
Oct 30 18:49:37.341116 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:37.341003    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35a6c3d1-e874-4203-aa17-d48e3a99e9ee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35a6c3d1-e874-4203-aa17-d48e3a99e9ee-maas-api-xks-ca\") pod \"35a6c3d1-e874-4203-aa17-d48e3a99e9ee\" (UID: \"35a6c3d1-e874-4203-aa17-d48e3a99e9ee\") "
Oct 30 18:49:37.341116 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:37.341066    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35a6c3d1-e874-4203-aa17-d48e3a99e9ee-kube-api-access-6gcg4\" (UniqueName: \"kubernetes.io/projected/35a6c3d1-e874-4203-aa17-d48e3a99e9ee-kube-api-access-6gcg4\") pod \"35a6c3d1-e874-4203-aa17-d48e3a99e9ee\" (UID: \"35a6c3d1-e874-4203-aa17-d48e3a99e9ee\") "
Oct 30 18:49:37.341357 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:37.341222    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35a6c3d1-e874-4203-aa17-d48e3a99e9ee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35a6c3d1-e874-4203-aa17-d48e3a99e9ee-maas-api-openshift-ca\") pod \"35a6c3d1-e874-4203-aa17-d48e3a99e9ee\" (UID: \"35a6c3d1-e874-4203-aa17-d48e3a99e9ee\") "
Oct 30 18:49:37.341577 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:37.341549    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35a6c3d1-e874-4203-aa17-d48e3a99e9ee-maas-api-openshift-ca" pod "35a6c3d1-e874-4203-aa17-d48e3a99e9ee" (UID: "35a6c3d1-e874-4203-aa17-d48e3a99e9ee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:49:37.343299 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:37.343271    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35a6c3d1-e874-4203-aa17-d48e3a99e9ee-maas-api-xks-ca" pod "35a6c3d1-e874-4203-aa17-d48e3a99e9ee" (UID: "35a6c3d1-e874-4203-aa17-d48e3a99e9ee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:49:37.343423 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:37.343319    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35a6c3d1-e874-4203-aa17-d48e3a99e9ee-kube-api-access-6gcg4" pod "35a6c3d1-e874-4203-aa17-d48e3a99e9ee" (UID: "35a6c3d1-e874-4203-aa17-d48e3a99e9ee"). InnerVolumeSpecName "kube-api-access-6gcg4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:49:37.442737 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:37.442671    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35a6c3d1-e874-4203-aa17-d48e3a99e9ee-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:49:37.442737 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:37.442734    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gcg4\" (UniqueName: \"kubernetes.io/projected/35a6c3d1-e874-4203-aa17-d48e3a99e9ee-kube-api-access-6gcg4\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:49:37.442737 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:37.442745    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35a6c3d1-e874-4203-aa17-d48e3a99e9ee-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:49:38.033451 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:38.033417    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cfca6cc-bhbpv" event={"ID":"35a6c3d1-e874-4203-aa17-d48e3a99e9ee","Type":"ContainerDied","Data":"987b64ff26b91a3e7c3fbc9be2545145dee65d026d542568deb493aa13b64c18"}
Oct 30 18:49:38.033630 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:38.033473    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="987b64ff26b91a3e7c3fbc9be2545145dee65d026d542568deb493aa13b64c18"
Oct 30 18:49:38.033630 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:38.033431    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cfca6cc-bhbpv"
Oct 30 18:49:44.792575 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:44.792533    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cce717cd-wxnlh"]
Oct 30 18:49:44.793189 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:44.793169    4525 cpu_manager.go:414] "containerMap: removing container" podUID="35a6c3d1-e874-4203-aa17-d48e3a99e9ee" containerName="revoke-keys"
Oct 30 18:49:44.793280 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:44.793192    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35a6c3d1-e874-4203-aa17-d48e3a99e9ee" containerName="revoke-keys"
Oct 30 18:49:44.793340 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:44.793327    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="35a6c3d1-e874-4203-aa17-d48e3a99e9ee" containerName="revoke-keys"
Oct 30 18:49:44.796757 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:44.796737    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cce717cd-wxnlh"
Oct 30 18:49:44.800925 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:44.800902    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:49:44.801032 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:44.800998    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94hcn\""
Oct 30 18:49:44.804240 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:44.803946    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cce717cd-wxnlh"]
Oct 30 18:49:44.842534 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:44.842506    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5bf0f22-bbf6-4d98-ae18-51f107bd13d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cce717cd-wxnlh\" (UID: \"a5bf0f22-bbf6-4d98-ae18-51f107bd13d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cce717cd-wxnlh"
Oct 30 18:49:44.842655 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:44.842556    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svzgs\" (UniqueName: \"kubernetes.io/projected/a5bf0f22-bbf6-4d98-ae18-51f107bd13d3-kube-api-access-svzgs\") pod \"maas-api-revoke-sub-cce717cd-wxnlh\" (UID: \"a5bf0f22-bbf6-4d98-ae18-51f107bd13d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cce717cd-wxnlh"
Oct 30 18:49:44.842655 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:44.842582    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5bf0f22-bbf6-4d98-ae18-51f107bd13d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cce717cd-wxnlh\" (UID: \"a5bf0f22-bbf6-4d98-ae18-51f107bd13d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cce717cd-wxnlh"
Oct 30 18:49:44.943795 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:44.943766    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svzgs\" (UniqueName: \"kubernetes.io/projected/a5bf0f22-bbf6-4d98-ae18-51f107bd13d3-kube-api-access-svzgs\") pod \"maas-api-revoke-sub-cce717cd-wxnlh\" (UID: \"a5bf0f22-bbf6-4d98-ae18-51f107bd13d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cce717cd-wxnlh"
Oct 30 18:49:44.943997 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:44.943819    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5bf0f22-bbf6-4d98-ae18-51f107bd13d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cce717cd-wxnlh\" (UID: \"a5bf0f22-bbf6-4d98-ae18-51f107bd13d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cce717cd-wxnlh"
Oct 30 18:49:44.943997 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:44.943986    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5bf0f22-bbf6-4d98-ae18-51f107bd13d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cce717cd-wxnlh\" (UID: \"a5bf0f22-bbf6-4d98-ae18-51f107bd13d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cce717cd-wxnlh"
Oct 30 18:49:44.944607 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:44.944586    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5bf0f22-bbf6-4d98-ae18-51f107bd13d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cce717cd-wxnlh\" (UID: \"a5bf0f22-bbf6-4d98-ae18-51f107bd13d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cce717cd-wxnlh"
Oct 30 18:49:44.946252 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:44.946230    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5bf0f22-bbf6-4d98-ae18-51f107bd13d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cce717cd-wxnlh\" (UID: \"a5bf0f22-bbf6-4d98-ae18-51f107bd13d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cce717cd-wxnlh"
Oct 30 18:49:44.951636 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:44.951614    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svzgs\" (UniqueName: \"kubernetes.io/projected/a5bf0f22-bbf6-4d98-ae18-51f107bd13d3-kube-api-access-svzgs\") pod \"maas-api-revoke-sub-cce717cd-wxnlh\" (UID: \"a5bf0f22-bbf6-4d98-ae18-51f107bd13d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cce717cd-wxnlh"
Oct 30 18:49:45.111283 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:45.111249    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cce717cd-wxnlh"
Oct 30 18:49:45.391573 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:45.391546    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cce717cd-wxnlh"]
Oct 30 18:49:45.392949 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:49:45.392924    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5bf0f22_bbf6_4d98_ae18_51f107bd13d3.slice/crio-961aad8497442ee66503984e503f65456a1a73dbaadb31e810063de26bc24bf8 WatchSource:0}: Error finding container 961aad8497442ee66503984e503f65456a1a73dbaadb31e810063de26bc24bf8: Status 404 returned error can't find the container with id 961aad8497442ee66503984e503f65456a1a73dbaadb31e810063de26bc24bf8
Oct 30 18:49:46.086989 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:46.086908    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cce717cd-wxnlh" event={"ID":"a5bf0f22-bbf6-4d98-ae18-51f107bd13d3","Type":"ContainerStarted","Data":"3b66cdb5e76d679a9c954b03d867ba1c4710007325158dd3682dfda2372e1242"}
Oct 30 18:49:46.086989 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:46.086963    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cce717cd-wxnlh" event={"ID":"a5bf0f22-bbf6-4d98-ae18-51f107bd13d3","Type":"ContainerStarted","Data":"961aad8497442ee66503984e503f65456a1a73dbaadb31e810063de26bc24bf8"}
Oct 30 18:49:46.109368 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:46.109317    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cce717cd-wxnlh" podStartSLOduration=2.10930429 podStartE2EDuration="2.10930429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:49:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:49:46.106893894 +0000 UTC m=+808.732563859" watchObservedRunningTime="2026-10-30 18:49:46.10930429 +0000 UTC m=+808.734974275"
Oct 30 18:49:47.094495 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:47.094463    4525 generic.go:356] "Generic (PLEG): container finished" podID="a5bf0f22-bbf6-4d98-ae18-51f107bd13d3" containerID="3b66cdb5e76d679a9c954b03d867ba1c4710007325158dd3682dfda2372e1242" exitCode=0
Oct 30 18:49:47.094934 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:47.094555    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cce717cd-wxnlh" event={"ID":"a5bf0f22-bbf6-4d98-ae18-51f107bd13d3","Type":"ContainerDied","Data":"3b66cdb5e76d679a9c954b03d867ba1c4710007325158dd3682dfda2372e1242"}
Oct 30 18:49:47.401541 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:49:47.401499    4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d49l" containerName="extractor"
Oct 30 18:49:48.295198 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:48.295167    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cce717cd-wxnlh"
Oct 30 18:49:48.370815 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:48.370732    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-436f4927-hz94c"]
Oct 30 18:49:48.371697 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:48.371652    4525 cpu_manager.go:414] "containerMap: removing container" podUID="a5bf0f22-bbf6-4d98-ae18-51f107bd13d3" containerName="revoke-keys"
Oct 30 18:49:48.371859 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:48.371836    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5bf0f22-bbf6-4d98-ae18-51f107bd13d3" containerName="revoke-keys"
Oct 30 18:49:48.372020 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:48.372007    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5bf0f22-bbf6-4d98-ae18-51f107bd13d3" containerName="revoke-keys"
Oct 30 18:49:48.377188 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:48.377166    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-436f4927-hz94c"]
Oct 30 18:49:48.377444 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:48.377428    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-436f4927-hz94c"
Oct 30 18:49:48.382704 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:48.382010    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5bf0f22-bbf6-4d98-ae18-51f107bd13d3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5bf0f22-bbf6-4d98-ae18-51f107bd13d3-maas-api-openshift-ca\") pod \"a5bf0f22-bbf6-4d98-ae18-51f107bd13d3\" (UID: \"a5bf0f22-bbf6-4d98-ae18-51f107bd13d3\") "
Oct 30 18:49:48.382704 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:48.382141    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5bf0f22-bbf6-4d98-ae18-51f107bd13d3-kube-api-access-svzgs\" (UniqueName: \"kubernetes.io/projected/a5bf0f22-bbf6-4d98-ae18-51f107bd13d3-kube-api-access-svzgs\") pod \"a5bf0f22-bbf6-4d98-ae18-51f107bd13d3\" (UID: \"a5bf0f22-bbf6-4d98-ae18-51f107bd13d3\") "
Oct 30 18:49:48.382704 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:48.382208    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5bf0f22-bbf6-4d98-ae18-51f107bd13d3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5bf0f22-bbf6-4d98-ae18-51f107bd13d3-maas-api-xks-ca\") pod \"a5bf0f22-bbf6-4d98-ae18-51f107bd13d3\" (UID: \"a5bf0f22-bbf6-4d98-ae18-51f107bd13d3\") "
Oct 30 18:49:48.383230 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:48.383204    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5bf0f22-bbf6-4d98-ae18-51f107bd13d3-maas-api-openshift-ca" pod "a5bf0f22-bbf6-4d98-ae18-51f107bd13d3" (UID: "a5bf0f22-bbf6-4d98-ae18-51f107bd13d3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:49:48.385263 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:48.385219    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5bf0f22-bbf6-4d98-ae18-51f107bd13d3-maas-api-xks-ca" pod "a5bf0f22-bbf6-4d98-ae18-51f107bd13d3" (UID: "a5bf0f22-bbf6-4d98-ae18-51f107bd13d3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:49:48.387299 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:48.387276    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5bf0f22-bbf6-4d98-ae18-51f107bd13d3-kube-api-access-svzgs" pod "a5bf0f22-bbf6-4d98-ae18-51f107bd13d3" (UID: "a5bf0f22-bbf6-4d98-ae18-51f107bd13d3"). InnerVolumeSpecName "kube-api-access-svzgs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:49:48.483568 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:48.483530    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13733f44-8aaa-4710-aa5f-be68364f5a14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-436f4927-hz94c\" (UID: \"13733f44-8aaa-4710-aa5f-be68364f5a14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-436f4927-hz94c"
Oct 30 18:49:48.483765 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:48.483674    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnt5g\" (UniqueName: \"kubernetes.io/projected/13733f44-8aaa-4710-aa5f-be68364f5a14-kube-api-access-lnt5g\") pod \"maas-api-revoke-sub-436f4927-hz94c\" (UID: \"13733f44-8aaa-4710-aa5f-be68364f5a14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-436f4927-hz94c"
Oct 30 18:49:48.483841 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:48.483826    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13733f44-8aaa-4710-aa5f-be68364f5a14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-436f4927-hz94c\" (UID: \"13733f44-8aaa-4710-aa5f-be68364f5a14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-436f4927-hz94c"
Oct 30 18:49:48.483926 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:48.483911    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5bf0f22-bbf6-4d98-ae18-51f107bd13d3-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:49:48.483985 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:48.483933    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5bf0f22-bbf6-4d98-ae18-51f107bd13d3-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:49:48.483985 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:48.483956    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svzgs\" (UniqueName: \"kubernetes.io/projected/a5bf0f22-bbf6-4d98-ae18-51f107bd13d3-kube-api-access-svzgs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:49:48.588300 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:48.588263    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13733f44-8aaa-4710-aa5f-be68364f5a14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-436f4927-hz94c\" (UID: \"13733f44-8aaa-4710-aa5f-be68364f5a14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-436f4927-hz94c"
Oct 30 18:49:48.588612 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:48.588592    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13733f44-8aaa-4710-aa5f-be68364f5a14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-436f4927-hz94c\" (UID: \"13733f44-8aaa-4710-aa5f-be68364f5a14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-436f4927-hz94c"
Oct 30 18:49:48.588869 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:48.588852    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnt5g\" (UniqueName: \"kubernetes.io/projected/13733f44-8aaa-4710-aa5f-be68364f5a14-kube-api-access-lnt5g\") pod \"maas-api-revoke-sub-436f4927-hz94c\" (UID: \"13733f44-8aaa-4710-aa5f-be68364f5a14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-436f4927-hz94c"
Oct 30 18:49:48.590398 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:48.590371    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13733f44-8aaa-4710-aa5f-be68364f5a14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-436f4927-hz94c\" (UID: \"13733f44-8aaa-4710-aa5f-be68364f5a14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-436f4927-hz94c"
Oct 30 18:49:48.595152 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:48.594076    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13733f44-8aaa-4710-aa5f-be68364f5a14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-436f4927-hz94c\" (UID: \"13733f44-8aaa-4710-aa5f-be68364f5a14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-436f4927-hz94c"
Oct 30 18:49:48.603281 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:48.603254    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnt5g\" (UniqueName: \"kubernetes.io/projected/13733f44-8aaa-4710-aa5f-be68364f5a14-kube-api-access-lnt5g\") pod \"maas-api-revoke-sub-436f4927-hz94c\" (UID: \"13733f44-8aaa-4710-aa5f-be68364f5a14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-436f4927-hz94c"
Oct 30 18:49:48.736754 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:48.735890    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-436f4927-hz94c"
Oct 30 18:49:49.145552 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:49.145528    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-436f4927-hz94c"]
Oct 30 18:49:49.145840 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:49.145813    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cce717cd-wxnlh" event={"ID":"a5bf0f22-bbf6-4d98-ae18-51f107bd13d3","Type":"ContainerDied","Data":"961aad8497442ee66503984e503f65456a1a73dbaadb31e810063de26bc24bf8"}
Oct 30 18:49:49.145921 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:49.145857    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="961aad8497442ee66503984e503f65456a1a73dbaadb31e810063de26bc24bf8"
Oct 30 18:49:49.145921 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:49.145669    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cce717cd-wxnlh"
Oct 30 18:49:49.433459 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:49.433424    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2fcdf464-jkxd9"]
Oct 30 18:49:49.439065 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:49.439044    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fcdf464-jkxd9"
Oct 30 18:49:49.444617 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:49.444597    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2fcdf464-jkxd9"]
Oct 30 18:49:49.522457 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:49.522432    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c09c6117-2835-4013-b4f8-9a4f6ef8064c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2fcdf464-jkxd9\" (UID: \"c09c6117-2835-4013-b4f8-9a4f6ef8064c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fcdf464-jkxd9"
Oct 30 18:49:49.522604 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:49.522468    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c09c6117-2835-4013-b4f8-9a4f6ef8064c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2fcdf464-jkxd9\" (UID: \"c09c6117-2835-4013-b4f8-9a4f6ef8064c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fcdf464-jkxd9"
Oct 30 18:49:49.522710 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:49.522666    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfd5j\" (UniqueName: \"kubernetes.io/projected/c09c6117-2835-4013-b4f8-9a4f6ef8064c-kube-api-access-xfd5j\") pod \"maas-api-revoke-sub-2fcdf464-jkxd9\" (UID: \"c09c6117-2835-4013-b4f8-9a4f6ef8064c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fcdf464-jkxd9"
Oct 30 18:49:49.624273 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:49.624238    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfd5j\" (UniqueName: \"kubernetes.io/projected/c09c6117-2835-4013-b4f8-9a4f6ef8064c-kube-api-access-xfd5j\") pod \"maas-api-revoke-sub-2fcdf464-jkxd9\" (UID: \"c09c6117-2835-4013-b4f8-9a4f6ef8064c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fcdf464-jkxd9"
Oct 30 18:49:49.624525 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:49.624500    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c09c6117-2835-4013-b4f8-9a4f6ef8064c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2fcdf464-jkxd9\" (UID: \"c09c6117-2835-4013-b4f8-9a4f6ef8064c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fcdf464-jkxd9"
Oct 30 18:49:49.624630 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:49.624545    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c09c6117-2835-4013-b4f8-9a4f6ef8064c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2fcdf464-jkxd9\" (UID: \"c09c6117-2835-4013-b4f8-9a4f6ef8064c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fcdf464-jkxd9"
Oct 30 18:49:49.625307 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:49.625285    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c09c6117-2835-4013-b4f8-9a4f6ef8064c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2fcdf464-jkxd9\" (UID: \"c09c6117-2835-4013-b4f8-9a4f6ef8064c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fcdf464-jkxd9"
Oct 30 18:49:49.626953 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:49.626936    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c09c6117-2835-4013-b4f8-9a4f6ef8064c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2fcdf464-jkxd9\" (UID: \"c09c6117-2835-4013-b4f8-9a4f6ef8064c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fcdf464-jkxd9"
Oct 30 18:49:49.633130 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:49.633101    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfd5j\" (UniqueName: \"kubernetes.io/projected/c09c6117-2835-4013-b4f8-9a4f6ef8064c-kube-api-access-xfd5j\") pod \"maas-api-revoke-sub-2fcdf464-jkxd9\" (UID: \"c09c6117-2835-4013-b4f8-9a4f6ef8064c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fcdf464-jkxd9"
Oct 30 18:49:49.756952 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:49.756873    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fcdf464-jkxd9"
Oct 30 18:49:50.041861 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:50.041829    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2fcdf464-jkxd9"]
Oct 30 18:49:50.042215 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:49:50.042192    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc09c6117_2835_4013_b4f8_9a4f6ef8064c.slice/crio-ea82d2ca3b47e289d7c15104428edffbc2a9a4c21101082b59ad95e52500f7d0 WatchSource:0}: Error finding container ea82d2ca3b47e289d7c15104428edffbc2a9a4c21101082b59ad95e52500f7d0: Status 404 returned error can't find the container with id ea82d2ca3b47e289d7c15104428edffbc2a9a4c21101082b59ad95e52500f7d0
Oct 30 18:49:50.153099 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:50.153066    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fcdf464-jkxd9" event={"ID":"c09c6117-2835-4013-b4f8-9a4f6ef8064c","Type":"ContainerStarted","Data":"ea82d2ca3b47e289d7c15104428edffbc2a9a4c21101082b59ad95e52500f7d0"}
Oct 30 18:49:50.154659 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:50.154632    4525 generic.go:356] "Generic (PLEG): container finished" podID="13733f44-8aaa-4710-aa5f-be68364f5a14" containerID="53cc1281188c36711e03d4f331e49fdd13802543589c56bfb9acdde49deea1cb" exitCode=0
Oct 30 18:49:50.154817 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:50.154715    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-436f4927-hz94c" event={"ID":"13733f44-8aaa-4710-aa5f-be68364f5a14","Type":"ContainerDied","Data":"53cc1281188c36711e03d4f331e49fdd13802543589c56bfb9acdde49deea1cb"}
Oct 30 18:49:50.154817 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:50.154768    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-436f4927-hz94c" event={"ID":"13733f44-8aaa-4710-aa5f-be68364f5a14","Type":"ContainerStarted","Data":"4825b9d4ab8f5adf84ee0c33b23f65954e1b69b511ebfd00b5ad51aa4690d9a2"}
Oct 30 18:49:51.162470 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:51.162438    4525 generic.go:356] "Generic (PLEG): container finished" podID="c09c6117-2835-4013-b4f8-9a4f6ef8064c" containerID="aca975120a4d12ccf001b26d9fa2f5e1690a4cb5bacfb89cd5e0b52d29da8c0b" exitCode=0
Oct 30 18:49:51.162884 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:51.162526    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fcdf464-jkxd9" event={"ID":"c09c6117-2835-4013-b4f8-9a4f6ef8064c","Type":"ContainerDied","Data":"aca975120a4d12ccf001b26d9fa2f5e1690a4cb5bacfb89cd5e0b52d29da8c0b"}
Oct 30 18:49:51.305564 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:51.305534    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-436f4927-hz94c"
Oct 30 18:49:51.448522 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:51.448493    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13733f44-8aaa-4710-aa5f-be68364f5a14-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13733f44-8aaa-4710-aa5f-be68364f5a14-maas-api-openshift-ca\") pod \"13733f44-8aaa-4710-aa5f-be68364f5a14\" (UID: \"13733f44-8aaa-4710-aa5f-be68364f5a14\") "
Oct 30 18:49:51.448712 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:51.448640    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13733f44-8aaa-4710-aa5f-be68364f5a14-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13733f44-8aaa-4710-aa5f-be68364f5a14-maas-api-xks-ca\") pod \"13733f44-8aaa-4710-aa5f-be68364f5a14\" (UID: \"13733f44-8aaa-4710-aa5f-be68364f5a14\") "
Oct 30 18:49:51.448761 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:51.448723    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13733f44-8aaa-4710-aa5f-be68364f5a14-kube-api-access-lnt5g\" (UniqueName: \"kubernetes.io/projected/13733f44-8aaa-4710-aa5f-be68364f5a14-kube-api-access-lnt5g\") pod \"13733f44-8aaa-4710-aa5f-be68364f5a14\" (UID: \"13733f44-8aaa-4710-aa5f-be68364f5a14\") "
Oct 30 18:49:51.448947 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:51.448918    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13733f44-8aaa-4710-aa5f-be68364f5a14-maas-api-openshift-ca" pod "13733f44-8aaa-4710-aa5f-be68364f5a14" (UID: "13733f44-8aaa-4710-aa5f-be68364f5a14"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:49:51.449054 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:51.449040    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13733f44-8aaa-4710-aa5f-be68364f5a14-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:49:51.450893 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:51.450873    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13733f44-8aaa-4710-aa5f-be68364f5a14-maas-api-xks-ca" pod "13733f44-8aaa-4710-aa5f-be68364f5a14" (UID: "13733f44-8aaa-4710-aa5f-be68364f5a14"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:49:51.450972 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:51.450893    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13733f44-8aaa-4710-aa5f-be68364f5a14-kube-api-access-lnt5g" pod "13733f44-8aaa-4710-aa5f-be68364f5a14" (UID: "13733f44-8aaa-4710-aa5f-be68364f5a14"). InnerVolumeSpecName "kube-api-access-lnt5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:49:51.550350 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:51.550108    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13733f44-8aaa-4710-aa5f-be68364f5a14-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:49:51.550350 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:51.550134    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnt5g\" (UniqueName: \"kubernetes.io/projected/13733f44-8aaa-4710-aa5f-be68364f5a14-kube-api-access-lnt5g\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:49:52.177557 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:52.177515    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-436f4927-hz94c"
Oct 30 18:49:52.178149 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:52.178127    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-436f4927-hz94c" event={"ID":"13733f44-8aaa-4710-aa5f-be68364f5a14","Type":"ContainerDied","Data":"4825b9d4ab8f5adf84ee0c33b23f65954e1b69b511ebfd00b5ad51aa4690d9a2"}
Oct 30 18:49:52.178197 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:52.178179    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4825b9d4ab8f5adf84ee0c33b23f65954e1b69b511ebfd00b5ad51aa4690d9a2"
Oct 30 18:49:52.330443 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:52.330413    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fcdf464-jkxd9"
Oct 30 18:49:52.460646 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:52.460555    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c09c6117-2835-4013-b4f8-9a4f6ef8064c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c09c6117-2835-4013-b4f8-9a4f6ef8064c-maas-api-openshift-ca\") pod \"c09c6117-2835-4013-b4f8-9a4f6ef8064c\" (UID: \"c09c6117-2835-4013-b4f8-9a4f6ef8064c\") "
Oct 30 18:49:52.460646 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:52.460600    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c09c6117-2835-4013-b4f8-9a4f6ef8064c-kube-api-access-xfd5j\" (UniqueName: \"kubernetes.io/projected/c09c6117-2835-4013-b4f8-9a4f6ef8064c-kube-api-access-xfd5j\") pod \"c09c6117-2835-4013-b4f8-9a4f6ef8064c\" (UID: \"c09c6117-2835-4013-b4f8-9a4f6ef8064c\") "
Oct 30 18:49:52.460872 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:52.460761    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c09c6117-2835-4013-b4f8-9a4f6ef8064c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c09c6117-2835-4013-b4f8-9a4f6ef8064c-maas-api-xks-ca\") pod \"c09c6117-2835-4013-b4f8-9a4f6ef8064c\" (UID: \"c09c6117-2835-4013-b4f8-9a4f6ef8064c\") "
Oct 30 18:49:52.460954 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:52.460929    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c09c6117-2835-4013-b4f8-9a4f6ef8064c-maas-api-openshift-ca" pod "c09c6117-2835-4013-b4f8-9a4f6ef8064c" (UID: "c09c6117-2835-4013-b4f8-9a4f6ef8064c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:49:52.461203 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:52.461141    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c09c6117-2835-4013-b4f8-9a4f6ef8064c-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:49:52.462882 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:52.462857    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c09c6117-2835-4013-b4f8-9a4f6ef8064c-kube-api-access-xfd5j" pod "c09c6117-2835-4013-b4f8-9a4f6ef8064c" (UID: "c09c6117-2835-4013-b4f8-9a4f6ef8064c"). InnerVolumeSpecName "kube-api-access-xfd5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:49:52.463061 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:52.463042    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c09c6117-2835-4013-b4f8-9a4f6ef8064c-maas-api-xks-ca" pod "c09c6117-2835-4013-b4f8-9a4f6ef8064c" (UID: "c09c6117-2835-4013-b4f8-9a4f6ef8064c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:49:52.561794 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:52.561755    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c09c6117-2835-4013-b4f8-9a4f6ef8064c-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:49:52.561794 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:52.561788    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfd5j\" (UniqueName: \"kubernetes.io/projected/c09c6117-2835-4013-b4f8-9a4f6ef8064c-kube-api-access-xfd5j\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:49:52.711481 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:52.711391    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60644dac-2k9w7"]
Oct 30 18:49:52.711927 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:52.711909    4525 cpu_manager.go:414] "containerMap: removing container" podUID="13733f44-8aaa-4710-aa5f-be68364f5a14" containerName="revoke-keys"
Oct 30 18:49:52.711994 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:52.711928    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13733f44-8aaa-4710-aa5f-be68364f5a14" containerName="revoke-keys"
Oct 30 18:49:52.711994 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:52.711962    4525 cpu_manager.go:414] "containerMap: removing container" podUID="c09c6117-2835-4013-b4f8-9a4f6ef8064c" containerName="revoke-keys"
Oct 30 18:49:52.711994 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:52.711967    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c09c6117-2835-4013-b4f8-9a4f6ef8064c" containerName="revoke-keys"
Oct 30 18:49:52.712103 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:52.712047    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="c09c6117-2835-4013-b4f8-9a4f6ef8064c" containerName="revoke-keys"
Oct 30 18:49:52.712103 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:52.712057    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="13733f44-8aaa-4710-aa5f-be68364f5a14" containerName="revoke-keys"
Oct 30 18:49:52.715353 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:52.715337    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60644dac-2k9w7"
Oct 30 18:49:52.722663 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:52.722635    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60644dac-2k9w7"]
Oct 30 18:49:52.864788 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:52.864754    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4adf49fc-7712-4a37-a27e-72500f89d9d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60644dac-2k9w7\" (UID: \"4adf49fc-7712-4a37-a27e-72500f89d9d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60644dac-2k9w7"
Oct 30 18:49:52.864933 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:52.864820    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4adf49fc-7712-4a37-a27e-72500f89d9d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60644dac-2k9w7\" (UID: \"4adf49fc-7712-4a37-a27e-72500f89d9d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60644dac-2k9w7"
Oct 30 18:49:52.864992 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:52.864941    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkwqh\" (UniqueName: \"kubernetes.io/projected/4adf49fc-7712-4a37-a27e-72500f89d9d0-kube-api-access-pkwqh\") pod \"maas-api-revoke-sub-60644dac-2k9w7\" (UID: \"4adf49fc-7712-4a37-a27e-72500f89d9d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60644dac-2k9w7"
Oct 30 18:49:52.965781 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:52.965661    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4adf49fc-7712-4a37-a27e-72500f89d9d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60644dac-2k9w7\" (UID: \"4adf49fc-7712-4a37-a27e-72500f89d9d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60644dac-2k9w7"
Oct 30 18:49:52.965781 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:52.965762    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkwqh\" (UniqueName: \"kubernetes.io/projected/4adf49fc-7712-4a37-a27e-72500f89d9d0-kube-api-access-pkwqh\") pod \"maas-api-revoke-sub-60644dac-2k9w7\" (UID: \"4adf49fc-7712-4a37-a27e-72500f89d9d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60644dac-2k9w7"
Oct 30 18:49:52.965966 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:52.965834    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4adf49fc-7712-4a37-a27e-72500f89d9d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60644dac-2k9w7\" (UID: \"4adf49fc-7712-4a37-a27e-72500f89d9d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60644dac-2k9w7"
Oct 30 18:49:52.966435 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:52.966383    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4adf49fc-7712-4a37-a27e-72500f89d9d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60644dac-2k9w7\" (UID: \"4adf49fc-7712-4a37-a27e-72500f89d9d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60644dac-2k9w7"
Oct 30 18:49:52.968234 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:52.968211    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4adf49fc-7712-4a37-a27e-72500f89d9d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60644dac-2k9w7\" (UID: \"4adf49fc-7712-4a37-a27e-72500f89d9d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60644dac-2k9w7"
Oct 30 18:49:52.974310 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:52.974290    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkwqh\" (UniqueName: \"kubernetes.io/projected/4adf49fc-7712-4a37-a27e-72500f89d9d0-kube-api-access-pkwqh\") pod \"maas-api-revoke-sub-60644dac-2k9w7\" (UID: \"4adf49fc-7712-4a37-a27e-72500f89d9d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60644dac-2k9w7"
Oct 30 18:49:53.032250 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:53.032211    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60644dac-2k9w7"
Oct 30 18:49:53.184987 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:53.184958    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fcdf464-jkxd9" event={"ID":"c09c6117-2835-4013-b4f8-9a4f6ef8064c","Type":"ContainerDied","Data":"ea82d2ca3b47e289d7c15104428edffbc2a9a4c21101082b59ad95e52500f7d0"}
Oct 30 18:49:53.184987 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:53.184984    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fcdf464-jkxd9"
Oct 30 18:49:53.185382 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:53.185004    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea82d2ca3b47e289d7c15104428edffbc2a9a4c21101082b59ad95e52500f7d0"
Oct 30 18:49:53.308351 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:53.308325    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60644dac-2k9w7"]
Oct 30 18:49:53.308806 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:49:53.308783    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4adf49fc_7712_4a37_a27e_72500f89d9d0.slice/crio-7770e8352b94241ce1c807746cb9b0ecc6703630a82925a52ae41640c7244eb6 WatchSource:0}: Error finding container 7770e8352b94241ce1c807746cb9b0ecc6703630a82925a52ae41640c7244eb6: Status 404 returned error can't find the container with id 7770e8352b94241ce1c807746cb9b0ecc6703630a82925a52ae41640c7244eb6
Oct 30 18:49:54.193472 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:54.193441    4525 generic.go:356] "Generic (PLEG): container finished" podID="4adf49fc-7712-4a37-a27e-72500f89d9d0" containerID="830230dce6c9c86f7b462b16d3a9eadfd73bfa01210f5150a4e441a1c9590659" exitCode=0
Oct 30 18:49:54.193873 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:54.193530    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60644dac-2k9w7" event={"ID":"4adf49fc-7712-4a37-a27e-72500f89d9d0","Type":"ContainerDied","Data":"830230dce6c9c86f7b462b16d3a9eadfd73bfa01210f5150a4e441a1c9590659"}
Oct 30 18:49:54.193873 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:54.193595    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60644dac-2k9w7" event={"ID":"4adf49fc-7712-4a37-a27e-72500f89d9d0","Type":"ContainerStarted","Data":"7770e8352b94241ce1c807746cb9b0ecc6703630a82925a52ae41640c7244eb6"}
Oct 30 18:49:55.341886 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:55.341860    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60644dac-2k9w7"
Oct 30 18:49:55.494164 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:55.494078    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4adf49fc-7712-4a37-a27e-72500f89d9d0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4adf49fc-7712-4a37-a27e-72500f89d9d0-maas-api-xks-ca\") pod \"4adf49fc-7712-4a37-a27e-72500f89d9d0\" (UID: \"4adf49fc-7712-4a37-a27e-72500f89d9d0\") "
Oct 30 18:49:55.494164 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:55.494149    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4adf49fc-7712-4a37-a27e-72500f89d9d0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4adf49fc-7712-4a37-a27e-72500f89d9d0-maas-api-openshift-ca\") pod \"4adf49fc-7712-4a37-a27e-72500f89d9d0\" (UID: \"4adf49fc-7712-4a37-a27e-72500f89d9d0\") "
Oct 30 18:49:55.494378 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:55.494189    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4adf49fc-7712-4a37-a27e-72500f89d9d0-kube-api-access-pkwqh\" (UniqueName: \"kubernetes.io/projected/4adf49fc-7712-4a37-a27e-72500f89d9d0-kube-api-access-pkwqh\") pod \"4adf49fc-7712-4a37-a27e-72500f89d9d0\" (UID: \"4adf49fc-7712-4a37-a27e-72500f89d9d0\") "
Oct 30 18:49:55.494541 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:55.494515    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4adf49fc-7712-4a37-a27e-72500f89d9d0-maas-api-openshift-ca" pod "4adf49fc-7712-4a37-a27e-72500f89d9d0" (UID: "4adf49fc-7712-4a37-a27e-72500f89d9d0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:49:55.496445 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:55.496414    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4adf49fc-7712-4a37-a27e-72500f89d9d0-maas-api-xks-ca" pod "4adf49fc-7712-4a37-a27e-72500f89d9d0" (UID: "4adf49fc-7712-4a37-a27e-72500f89d9d0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:49:55.496548 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:55.496486    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4adf49fc-7712-4a37-a27e-72500f89d9d0-kube-api-access-pkwqh" pod "4adf49fc-7712-4a37-a27e-72500f89d9d0" (UID: "4adf49fc-7712-4a37-a27e-72500f89d9d0"). InnerVolumeSpecName "kube-api-access-pkwqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:49:55.595919 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:55.595881    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4adf49fc-7712-4a37-a27e-72500f89d9d0-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:49:55.595919 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:55.595911    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4adf49fc-7712-4a37-a27e-72500f89d9d0-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:49:55.595919 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:55.595922    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkwqh\" (UniqueName: \"kubernetes.io/projected/4adf49fc-7712-4a37-a27e-72500f89d9d0-kube-api-access-pkwqh\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:49:56.207207 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:56.207178    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60644dac-2k9w7"
Oct 30 18:49:56.207391 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:56.207207    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60644dac-2k9w7" event={"ID":"4adf49fc-7712-4a37-a27e-72500f89d9d0","Type":"ContainerDied","Data":"7770e8352b94241ce1c807746cb9b0ecc6703630a82925a52ae41640c7244eb6"}
Oct 30 18:49:56.207391 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:56.207249    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7770e8352b94241ce1c807746cb9b0ecc6703630a82925a52ae41640c7244eb6"
Oct 30 18:49:56.729925 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:56.729893    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-baa9e441-l4dxz"]
Oct 30 18:49:56.730567 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:56.730548    4525 cpu_manager.go:414] "containerMap: removing container" podUID="4adf49fc-7712-4a37-a27e-72500f89d9d0" containerName="revoke-keys"
Oct 30 18:49:56.730619 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:56.730570    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4adf49fc-7712-4a37-a27e-72500f89d9d0" containerName="revoke-keys"
Oct 30 18:49:56.730703 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:56.730673    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="4adf49fc-7712-4a37-a27e-72500f89d9d0" containerName="revoke-keys"
Oct 30 18:49:56.734980 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:56.734958    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa9e441-l4dxz"
Oct 30 18:49:56.741470 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:56.741444    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:49:56.741759 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:56.741740    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-baa9e441-l4dxz"]
Oct 30 18:49:56.741759 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:56.741755    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94hcn\""
Oct 30 18:49:56.912786 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:56.912752    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4d5b030-8d02-4b18-9c8e-f47dacc8a72a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-baa9e441-l4dxz\" (UID: \"b4d5b030-8d02-4b18-9c8e-f47dacc8a72a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa9e441-l4dxz"
Oct 30 18:49:56.912971 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:56.912820    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4d5b030-8d02-4b18-9c8e-f47dacc8a72a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-baa9e441-l4dxz\" (UID: \"b4d5b030-8d02-4b18-9c8e-f47dacc8a72a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa9e441-l4dxz"
Oct 30 18:49:56.912971 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:56.912932    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krs7v\" (UniqueName: \"kubernetes.io/projected/b4d5b030-8d02-4b18-9c8e-f47dacc8a72a-kube-api-access-krs7v\") pod \"maas-api-revoke-sub-baa9e441-l4dxz\" (UID: \"b4d5b030-8d02-4b18-9c8e-f47dacc8a72a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa9e441-l4dxz"
Oct 30 18:49:57.014945 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:57.014835    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4d5b030-8d02-4b18-9c8e-f47dacc8a72a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-baa9e441-l4dxz\" (UID: \"b4d5b030-8d02-4b18-9c8e-f47dacc8a72a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa9e441-l4dxz"
Oct 30 18:49:57.015130 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:57.014955    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4d5b030-8d02-4b18-9c8e-f47dacc8a72a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-baa9e441-l4dxz\" (UID: \"b4d5b030-8d02-4b18-9c8e-f47dacc8a72a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa9e441-l4dxz"
Oct 30 18:49:57.015130 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:57.015004    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krs7v\" (UniqueName: \"kubernetes.io/projected/b4d5b030-8d02-4b18-9c8e-f47dacc8a72a-kube-api-access-krs7v\") pod \"maas-api-revoke-sub-baa9e441-l4dxz\" (UID: \"b4d5b030-8d02-4b18-9c8e-f47dacc8a72a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa9e441-l4dxz"
Oct 30 18:49:57.015767 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:57.015738    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4d5b030-8d02-4b18-9c8e-f47dacc8a72a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-baa9e441-l4dxz\" (UID: \"b4d5b030-8d02-4b18-9c8e-f47dacc8a72a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa9e441-l4dxz"
Oct 30 18:49:57.017613 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:57.017588    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4d5b030-8d02-4b18-9c8e-f47dacc8a72a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-baa9e441-l4dxz\" (UID: \"b4d5b030-8d02-4b18-9c8e-f47dacc8a72a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa9e441-l4dxz"
Oct 30 18:49:57.040244 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:57.040205    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krs7v\" (UniqueName: \"kubernetes.io/projected/b4d5b030-8d02-4b18-9c8e-f47dacc8a72a-kube-api-access-krs7v\") pod \"maas-api-revoke-sub-baa9e441-l4dxz\" (UID: \"b4d5b030-8d02-4b18-9c8e-f47dacc8a72a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa9e441-l4dxz"
Oct 30 18:49:57.051852 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:57.051821    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa9e441-l4dxz"
Oct 30 18:49:57.359537 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:57.359506    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-baa9e441-l4dxz"]
Oct 30 18:49:57.360954 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:49:57.360929    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4d5b030_8d02_4b18_9c8e_f47dacc8a72a.slice/crio-1b5a1c5bc6be3961f794af8599b7ce7eb88264a2dbde81e0b9f6f004bac80c20 WatchSource:0}: Error finding container 1b5a1c5bc6be3961f794af8599b7ce7eb88264a2dbde81e0b9f6f004bac80c20: Status 404 returned error can't find the container with id 1b5a1c5bc6be3961f794af8599b7ce7eb88264a2dbde81e0b9f6f004bac80c20
Oct 30 18:49:58.222011 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:58.221976    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa9e441-l4dxz" event={"ID":"b4d5b030-8d02-4b18-9c8e-f47dacc8a72a","Type":"ContainerStarted","Data":"a5f90c0a08b4da14c5207644a429b2991383f0cf426241b35803a8db4b92cf70"}
Oct 30 18:49:58.222409 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:58.222027    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa9e441-l4dxz" event={"ID":"b4d5b030-8d02-4b18-9c8e-f47dacc8a72a","Type":"ContainerStarted","Data":"1b5a1c5bc6be3961f794af8599b7ce7eb88264a2dbde81e0b9f6f004bac80c20"}
Oct 30 18:49:58.242434 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:58.242384    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa9e441-l4dxz" podStartSLOduration=2.242367996 podStartE2EDuration="2.242367996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:49:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:49:58.239541229 +0000 UTC m=+820.865211192" watchObservedRunningTime="2026-10-30 18:49:58.242367996 +0000 UTC m=+820.868037960"
Oct 30 18:49:59.229719 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:59.229671    4525 generic.go:356] "Generic (PLEG): container finished" podID="b4d5b030-8d02-4b18-9c8e-f47dacc8a72a" containerID="a5f90c0a08b4da14c5207644a429b2991383f0cf426241b35803a8db4b92cf70" exitCode=0
Oct 30 18:49:59.230126 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:49:59.229777    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa9e441-l4dxz" event={"ID":"b4d5b030-8d02-4b18-9c8e-f47dacc8a72a","Type":"ContainerDied","Data":"a5f90c0a08b4da14c5207644a429b2991383f0cf426241b35803a8db4b92cf70"}
Oct 30 18:50:00.374911 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:00.374877    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa9e441-l4dxz"
Oct 30 18:50:00.555189 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:00.555102    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4d5b030-8d02-4b18-9c8e-f47dacc8a72a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4d5b030-8d02-4b18-9c8e-f47dacc8a72a-maas-api-openshift-ca\") pod \"b4d5b030-8d02-4b18-9c8e-f47dacc8a72a\" (UID: \"b4d5b030-8d02-4b18-9c8e-f47dacc8a72a\") "
Oct 30 18:50:00.555338 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:00.555198    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4d5b030-8d02-4b18-9c8e-f47dacc8a72a-kube-api-access-krs7v\" (UniqueName: \"kubernetes.io/projected/b4d5b030-8d02-4b18-9c8e-f47dacc8a72a-kube-api-access-krs7v\") pod \"b4d5b030-8d02-4b18-9c8e-f47dacc8a72a\" (UID: \"b4d5b030-8d02-4b18-9c8e-f47dacc8a72a\") "
Oct 30 18:50:00.555338 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:00.555233    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4d5b030-8d02-4b18-9c8e-f47dacc8a72a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4d5b030-8d02-4b18-9c8e-f47dacc8a72a-maas-api-xks-ca\") pod \"b4d5b030-8d02-4b18-9c8e-f47dacc8a72a\" (UID: \"b4d5b030-8d02-4b18-9c8e-f47dacc8a72a\") "
Oct 30 18:50:00.555526 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:00.555498    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4d5b030-8d02-4b18-9c8e-f47dacc8a72a-maas-api-openshift-ca" pod "b4d5b030-8d02-4b18-9c8e-f47dacc8a72a" (UID: "b4d5b030-8d02-4b18-9c8e-f47dacc8a72a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:50:00.555655 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:00.555631    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4d5b030-8d02-4b18-9c8e-f47dacc8a72a-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:00.557408 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:00.557380    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4d5b030-8d02-4b18-9c8e-f47dacc8a72a-maas-api-xks-ca" pod "b4d5b030-8d02-4b18-9c8e-f47dacc8a72a" (UID: "b4d5b030-8d02-4b18-9c8e-f47dacc8a72a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:50:00.557544 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:00.557526    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4d5b030-8d02-4b18-9c8e-f47dacc8a72a-kube-api-access-krs7v" pod "b4d5b030-8d02-4b18-9c8e-f47dacc8a72a" (UID: "b4d5b030-8d02-4b18-9c8e-f47dacc8a72a"). InnerVolumeSpecName "kube-api-access-krs7v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:50:00.656459 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:00.656424    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4d5b030-8d02-4b18-9c8e-f47dacc8a72a-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:00.656459 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:00.656456    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krs7v\" (UniqueName: \"kubernetes.io/projected/b4d5b030-8d02-4b18-9c8e-f47dacc8a72a-kube-api-access-krs7v\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:01.243336 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:01.243308    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa9e441-l4dxz"
Oct 30 18:50:01.243336 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:01.243338    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa9e441-l4dxz" event={"ID":"b4d5b030-8d02-4b18-9c8e-f47dacc8a72a","Type":"ContainerDied","Data":"1b5a1c5bc6be3961f794af8599b7ce7eb88264a2dbde81e0b9f6f004bac80c20"}
Oct 30 18:50:01.243544 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:01.243392    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b5a1c5bc6be3961f794af8599b7ce7eb88264a2dbde81e0b9f6f004bac80c20"
Oct 30 18:50:03.335239 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:03.335202    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5c6fc54974-gq9dn"]
Oct 30 18:50:03.335647 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:03.335633    4525 cpu_manager.go:414] "containerMap: removing container" podUID="b4d5b030-8d02-4b18-9c8e-f47dacc8a72a" containerName="revoke-keys"
Oct 30 18:50:03.335727 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:03.335648    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4d5b030-8d02-4b18-9c8e-f47dacc8a72a" containerName="revoke-keys"
Oct 30 18:50:03.335763 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:03.335751    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4d5b030-8d02-4b18-9c8e-f47dacc8a72a" containerName="revoke-keys"
Oct 30 18:50:03.339038 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:03.339023    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5c6fc54974-gq9dn"
Oct 30 18:50:03.347019 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:03.346994    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-1cdb3fee4835\""
Oct 30 18:50:03.347153 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:03.347095    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Oct 30 18:50:03.347204 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:03.347182    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Oct 30 18:50:03.347389 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:03.347366    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-b2c1d1ef4964\""
Oct 30 18:50:03.347504 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:03.347398    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Oct 30 18:50:03.347504 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:03.347443    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Oct 30 18:50:03.350832 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:03.350812    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5c6fc54974-gq9dn"]
Oct 30 18:50:03.390284 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:03.390251    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/354c6c7c-df51-4817-8369-620c5a0c65ff-provider-credentials\") pod \"payload-processing-external-model-5c6fc54974-gq9dn\" (UID: \"354c6c7c-df51-4817-8369-620c5a0c65ff\") " pod="models-as-a-service/payload-processing-external-model-5c6fc54974-gq9dn"
Oct 30 18:50:03.390284 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:03.390289    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/354c6c7c-df51-4817-8369-620c5a0c65ff-routing-overlay-volume\") pod \"payload-processing-external-model-5c6fc54974-gq9dn\" (UID: \"354c6c7c-df51-4817-8369-620c5a0c65ff\") " pod="models-as-a-service/payload-processing-external-model-5c6fc54974-gq9dn"
Oct 30 18:50:03.390488 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:03.390322    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvxtb\" (UniqueName: \"kubernetes.io/projected/354c6c7c-df51-4817-8369-620c5a0c65ff-kube-api-access-wvxtb\") pod \"payload-processing-external-model-5c6fc54974-gq9dn\" (UID: \"354c6c7c-df51-4817-8369-620c5a0c65ff\") " pod="models-as-a-service/payload-processing-external-model-5c6fc54974-gq9dn"
Oct 30 18:50:03.390488 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:03.390449    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/354c6c7c-df51-4817-8369-620c5a0c65ff-plugins-config-volume\") pod \"payload-processing-external-model-5c6fc54974-gq9dn\" (UID: \"354c6c7c-df51-4817-8369-620c5a0c65ff\") " pod="models-as-a-service/payload-processing-external-model-5c6fc54974-gq9dn"
Oct 30 18:50:03.497719 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:03.495227    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/354c6c7c-df51-4817-8369-620c5a0c65ff-provider-credentials\") pod \"payload-processing-external-model-5c6fc54974-gq9dn\" (UID: \"354c6c7c-df51-4817-8369-620c5a0c65ff\") " pod="models-as-a-service/payload-processing-external-model-5c6fc54974-gq9dn"
Oct 30 18:50:03.497719 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:03.495281    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/354c6c7c-df51-4817-8369-620c5a0c65ff-routing-overlay-volume\") pod \"payload-processing-external-model-5c6fc54974-gq9dn\" (UID: \"354c6c7c-df51-4817-8369-620c5a0c65ff\") " pod="models-as-a-service/payload-processing-external-model-5c6fc54974-gq9dn"
Oct 30 18:50:03.497719 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:03.495326    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvxtb\" (UniqueName: \"kubernetes.io/projected/354c6c7c-df51-4817-8369-620c5a0c65ff-kube-api-access-wvxtb\") pod \"payload-processing-external-model-5c6fc54974-gq9dn\" (UID: \"354c6c7c-df51-4817-8369-620c5a0c65ff\") " pod="models-as-a-service/payload-processing-external-model-5c6fc54974-gq9dn"
Oct 30 18:50:03.497719 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:03.495366    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/354c6c7c-df51-4817-8369-620c5a0c65ff-plugins-config-volume\") pod \"payload-processing-external-model-5c6fc54974-gq9dn\" (UID: \"354c6c7c-df51-4817-8369-620c5a0c65ff\") " pod="models-as-a-service/payload-processing-external-model-5c6fc54974-gq9dn"
Oct 30 18:50:03.497719 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:03.496392    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/354c6c7c-df51-4817-8369-620c5a0c65ff-plugins-config-volume\") pod \"payload-processing-external-model-5c6fc54974-gq9dn\" (UID: \"354c6c7c-df51-4817-8369-620c5a0c65ff\") " pod="models-as-a-service/payload-processing-external-model-5c6fc54974-gq9dn"
Oct 30 18:50:03.497719 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:03.497503    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/354c6c7c-df51-4817-8369-620c5a0c65ff-routing-overlay-volume\") pod \"payload-processing-external-model-5c6fc54974-gq9dn\" (UID: \"354c6c7c-df51-4817-8369-620c5a0c65ff\") " pod="models-as-a-service/payload-processing-external-model-5c6fc54974-gq9dn"
Oct 30 18:50:03.503425 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:03.503253    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/354c6c7c-df51-4817-8369-620c5a0c65ff-provider-credentials\") pod \"payload-processing-external-model-5c6fc54974-gq9dn\" (UID: \"354c6c7c-df51-4817-8369-620c5a0c65ff\") " pod="models-as-a-service/payload-processing-external-model-5c6fc54974-gq9dn"
Oct 30 18:50:03.519365 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:03.519330    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvxtb\" (UniqueName: \"kubernetes.io/projected/354c6c7c-df51-4817-8369-620c5a0c65ff-kube-api-access-wvxtb\") pod \"payload-processing-external-model-5c6fc54974-gq9dn\" (UID: \"354c6c7c-df51-4817-8369-620c5a0c65ff\") " pod="models-as-a-service/payload-processing-external-model-5c6fc54974-gq9dn"
Oct 30 18:50:03.654219 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:03.654190    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5c6fc54974-gq9dn"
Oct 30 18:50:03.976085 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:03.975656    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5c6fc54974-gq9dn"]
Oct 30 18:50:04.267355 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:04.267258    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5c6fc54974-gq9dn" event={"ID":"354c6c7c-df51-4817-8369-620c5a0c65ff","Type":"ContainerStarted","Data":"03bc274bcd24c7a02548a47f3d1397e3cf48077b13b5b628029a635f5a43a1ff"}
Oct 30 18:50:04.267355 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:04.267322    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5c6fc54974-gq9dn" event={"ID":"354c6c7c-df51-4817-8369-620c5a0c65ff","Type":"ContainerStarted","Data":"29cd273d922d8ecbe0f9c7f01cf00e6b2d347400e8dc3fb30e61ddc5513347f8"}
Oct 30 18:50:04.267355 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:04.267354    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5c6fc54974-gq9dn"
Oct 30 18:50:04.289901 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:04.289853    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5c6fc54974-gq9dn" podStartSLOduration=1.289839679 podStartE2EDuration="1.289839679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:50:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:50:04.28722133 +0000 UTC m=+826.912891295" watchObservedRunningTime="2026-10-30 18:50:04.289839679 +0000 UTC m=+826.915509660"
Oct 30 18:50:05.055436 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:05.055405    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb66e9c4-9kb7p"]
Oct 30 18:50:05.059188 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:05.059172    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb66e9c4-9kb7p"
Oct 30 18:50:05.074102 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:05.074081    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:50:05.074257 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:05.074240    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94hcn\""
Oct 30 18:50:05.075113 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:05.075093    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb66e9c4-9kb7p"]
Oct 30 18:50:05.112622 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:05.112589    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d6c76d7-185c-49e5-8357-fbef4138d925-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb66e9c4-9kb7p\" (UID: \"4d6c76d7-185c-49e5-8357-fbef4138d925\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb66e9c4-9kb7p"
Oct 30 18:50:05.112817 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:05.112778    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d6c76d7-185c-49e5-8357-fbef4138d925-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb66e9c4-9kb7p\" (UID: \"4d6c76d7-185c-49e5-8357-fbef4138d925\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb66e9c4-9kb7p"
Oct 30 18:50:05.112859 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:05.112838    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmdgb\" (UniqueName: \"kubernetes.io/projected/4d6c76d7-185c-49e5-8357-fbef4138d925-kube-api-access-xmdgb\") pod \"maas-api-revoke-sub-bb66e9c4-9kb7p\" (UID: \"4d6c76d7-185c-49e5-8357-fbef4138d925\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb66e9c4-9kb7p"
Oct 30 18:50:05.214161 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:05.214118    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmdgb\" (UniqueName: \"kubernetes.io/projected/4d6c76d7-185c-49e5-8357-fbef4138d925-kube-api-access-xmdgb\") pod \"maas-api-revoke-sub-bb66e9c4-9kb7p\" (UID: \"4d6c76d7-185c-49e5-8357-fbef4138d925\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb66e9c4-9kb7p"
Oct 30 18:50:05.214425 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:05.214257    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d6c76d7-185c-49e5-8357-fbef4138d925-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb66e9c4-9kb7p\" (UID: \"4d6c76d7-185c-49e5-8357-fbef4138d925\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb66e9c4-9kb7p"
Oct 30 18:50:05.214425 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:05.214392    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d6c76d7-185c-49e5-8357-fbef4138d925-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb66e9c4-9kb7p\" (UID: \"4d6c76d7-185c-49e5-8357-fbef4138d925\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb66e9c4-9kb7p"
Oct 30 18:50:05.214955 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:05.214920    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d6c76d7-185c-49e5-8357-fbef4138d925-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb66e9c4-9kb7p\" (UID: \"4d6c76d7-185c-49e5-8357-fbef4138d925\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb66e9c4-9kb7p"
Oct 30 18:50:05.217110 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:05.217080    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d6c76d7-185c-49e5-8357-fbef4138d925-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb66e9c4-9kb7p\" (UID: \"4d6c76d7-185c-49e5-8357-fbef4138d925\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb66e9c4-9kb7p"
Oct 30 18:50:05.227328 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:05.227301    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmdgb\" (UniqueName: \"kubernetes.io/projected/4d6c76d7-185c-49e5-8357-fbef4138d925-kube-api-access-xmdgb\") pod \"maas-api-revoke-sub-bb66e9c4-9kb7p\" (UID: \"4d6c76d7-185c-49e5-8357-fbef4138d925\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb66e9c4-9kb7p"
Oct 30 18:50:05.372629 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:05.372598    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb66e9c4-9kb7p"
Oct 30 18:50:05.663543 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:05.663503    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb66e9c4-9kb7p"]
Oct 30 18:50:05.665203 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:50:05.665176    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d6c76d7_185c_49e5_8357_fbef4138d925.slice/crio-d7bee22f8ba4506648152708bfae677d41e9fb2fa8c6f6c3c27baa153560178a WatchSource:0}: Error finding container d7bee22f8ba4506648152708bfae677d41e9fb2fa8c6f6c3c27baa153560178a: Status 404 returned error can't find the container with id d7bee22f8ba4506648152708bfae677d41e9fb2fa8c6f6c3c27baa153560178a
Oct 30 18:50:06.281622 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:06.281528    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb66e9c4-9kb7p" event={"ID":"4d6c76d7-185c-49e5-8357-fbef4138d925","Type":"ContainerStarted","Data":"f83260bf5fd59e11d67dc243506405532a3e0cf85e908bc89c24db286fda49fd"}
Oct 30 18:50:06.281622 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:06.281580    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb66e9c4-9kb7p" event={"ID":"4d6c76d7-185c-49e5-8357-fbef4138d925","Type":"ContainerStarted","Data":"d7bee22f8ba4506648152708bfae677d41e9fb2fa8c6f6c3c27baa153560178a"}
Oct 30 18:50:07.289471 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:07.289432    4525 generic.go:356] "Generic (PLEG): container finished" podID="4d6c76d7-185c-49e5-8357-fbef4138d925" containerID="f83260bf5fd59e11d67dc243506405532a3e0cf85e908bc89c24db286fda49fd" exitCode=0
Oct 30 18:50:07.289979 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:07.289523    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb66e9c4-9kb7p" event={"ID":"4d6c76d7-185c-49e5-8357-fbef4138d925","Type":"ContainerDied","Data":"f83260bf5fd59e11d67dc243506405532a3e0cf85e908bc89c24db286fda49fd"}
Oct 30 18:50:08.433056 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:08.433028    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb66e9c4-9kb7p"
Oct 30 18:50:08.448180 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:08.448154    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d6c76d7-185c-49e5-8357-fbef4138d925-kube-api-access-xmdgb\" (UniqueName: \"kubernetes.io/projected/4d6c76d7-185c-49e5-8357-fbef4138d925-kube-api-access-xmdgb\") pod \"4d6c76d7-185c-49e5-8357-fbef4138d925\" (UID: \"4d6c76d7-185c-49e5-8357-fbef4138d925\") "
Oct 30 18:50:08.448327 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:08.448265    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d6c76d7-185c-49e5-8357-fbef4138d925-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d6c76d7-185c-49e5-8357-fbef4138d925-maas-api-openshift-ca\") pod \"4d6c76d7-185c-49e5-8357-fbef4138d925\" (UID: \"4d6c76d7-185c-49e5-8357-fbef4138d925\") "
Oct 30 18:50:08.448327 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:08.448310    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d6c76d7-185c-49e5-8357-fbef4138d925-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d6c76d7-185c-49e5-8357-fbef4138d925-maas-api-xks-ca\") pod \"4d6c76d7-185c-49e5-8357-fbef4138d925\" (UID: \"4d6c76d7-185c-49e5-8357-fbef4138d925\") "
Oct 30 18:50:08.448597 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:08.448574    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d6c76d7-185c-49e5-8357-fbef4138d925-maas-api-openshift-ca" pod "4d6c76d7-185c-49e5-8357-fbef4138d925" (UID: "4d6c76d7-185c-49e5-8357-fbef4138d925"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:50:08.450354 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:08.450331    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d6c76d7-185c-49e5-8357-fbef4138d925-kube-api-access-xmdgb" pod "4d6c76d7-185c-49e5-8357-fbef4138d925" (UID: "4d6c76d7-185c-49e5-8357-fbef4138d925"). InnerVolumeSpecName "kube-api-access-xmdgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:50:08.450464 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:08.450403    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d6c76d7-185c-49e5-8357-fbef4138d925-maas-api-xks-ca" pod "4d6c76d7-185c-49e5-8357-fbef4138d925" (UID: "4d6c76d7-185c-49e5-8357-fbef4138d925"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:50:08.549577 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:08.549496    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d6c76d7-185c-49e5-8357-fbef4138d925-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:08.549577 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:08.549524    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d6c76d7-185c-49e5-8357-fbef4138d925-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:08.549577 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:08.549535    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmdgb\" (UniqueName: \"kubernetes.io/projected/4d6c76d7-185c-49e5-8357-fbef4138d925-kube-api-access-xmdgb\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:09.302958 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:09.302930    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb66e9c4-9kb7p"
Oct 30 18:50:09.302958 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:09.302951    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb66e9c4-9kb7p" event={"ID":"4d6c76d7-185c-49e5-8357-fbef4138d925","Type":"ContainerDied","Data":"d7bee22f8ba4506648152708bfae677d41e9fb2fa8c6f6c3c27baa153560178a"}
Oct 30 18:50:09.303161 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:09.302998    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d7bee22f8ba4506648152708bfae677d41e9fb2fa8c6f6c3c27baa153560178a"
Oct 30 18:50:15.277398 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:15.277360    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5c6fc54974-gq9dn"
Oct 30 18:50:15.390971 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:15.390937    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10240f17-6wtp6"]
Oct 30 18:50:15.391734 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:15.391484    4525 cpu_manager.go:414] "containerMap: removing container" podUID="4d6c76d7-185c-49e5-8357-fbef4138d925" containerName="revoke-keys"
Oct 30 18:50:15.391734 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:15.391505    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d6c76d7-185c-49e5-8357-fbef4138d925" containerName="revoke-keys"
Oct 30 18:50:15.391734 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:15.391600    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d6c76d7-185c-49e5-8357-fbef4138d925" containerName="revoke-keys"
Oct 30 18:50:15.395109 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:15.395088    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10240f17-6wtp6"
Oct 30 18:50:15.407108 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:15.405873    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10240f17-6wtp6"]
Oct 30 18:50:15.408117 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:15.408035    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:50:15.408414 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:15.408334    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94hcn\""
Oct 30 18:50:15.525569 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:15.525533    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0df7654-ec74-41ac-adfc-bacec681ff36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10240f17-6wtp6\" (UID: \"d0df7654-ec74-41ac-adfc-bacec681ff36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10240f17-6wtp6"
Oct 30 18:50:15.525802 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:15.525589    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjc28\" (UniqueName: \"kubernetes.io/projected/d0df7654-ec74-41ac-adfc-bacec681ff36-kube-api-access-vjc28\") pod \"maas-api-revoke-sub-10240f17-6wtp6\" (UID: \"d0df7654-ec74-41ac-adfc-bacec681ff36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10240f17-6wtp6"
Oct 30 18:50:15.525802 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:15.525719    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0df7654-ec74-41ac-adfc-bacec681ff36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10240f17-6wtp6\" (UID: \"d0df7654-ec74-41ac-adfc-bacec681ff36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10240f17-6wtp6"
Oct 30 18:50:15.627003 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:15.626959    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0df7654-ec74-41ac-adfc-bacec681ff36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10240f17-6wtp6\" (UID: \"d0df7654-ec74-41ac-adfc-bacec681ff36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10240f17-6wtp6"
Oct 30 18:50:15.627326 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:15.627290    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjc28\" (UniqueName: \"kubernetes.io/projected/d0df7654-ec74-41ac-adfc-bacec681ff36-kube-api-access-vjc28\") pod \"maas-api-revoke-sub-10240f17-6wtp6\" (UID: \"d0df7654-ec74-41ac-adfc-bacec681ff36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10240f17-6wtp6"
Oct 30 18:50:15.627462 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:15.627387    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0df7654-ec74-41ac-adfc-bacec681ff36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10240f17-6wtp6\" (UID: \"d0df7654-ec74-41ac-adfc-bacec681ff36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10240f17-6wtp6"
Oct 30 18:50:15.627843 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:15.627792    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0df7654-ec74-41ac-adfc-bacec681ff36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10240f17-6wtp6\" (UID: \"d0df7654-ec74-41ac-adfc-bacec681ff36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10240f17-6wtp6"
Oct 30 18:50:15.630737 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:15.630704    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0df7654-ec74-41ac-adfc-bacec681ff36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10240f17-6wtp6\" (UID: \"d0df7654-ec74-41ac-adfc-bacec681ff36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10240f17-6wtp6"
Oct 30 18:50:15.647546 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:15.639405    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjc28\" (UniqueName: \"kubernetes.io/projected/d0df7654-ec74-41ac-adfc-bacec681ff36-kube-api-access-vjc28\") pod \"maas-api-revoke-sub-10240f17-6wtp6\" (UID: \"d0df7654-ec74-41ac-adfc-bacec681ff36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10240f17-6wtp6"
Oct 30 18:50:15.736705 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:15.736121    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10240f17-6wtp6"
Oct 30 18:50:16.061396 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:16.061365    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10240f17-6wtp6"]
Oct 30 18:50:16.062141 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:50:16.062115    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0df7654_ec74_41ac_adfc_bacec681ff36.slice/crio-9514507620a340f15c35151fa4c03e8477ac7a3b24d7ce137230625a7a3076fb WatchSource:0}: Error finding container 9514507620a340f15c35151fa4c03e8477ac7a3b24d7ce137230625a7a3076fb: Status 404 returned error can't find the container with id 9514507620a340f15c35151fa4c03e8477ac7a3b24d7ce137230625a7a3076fb
Oct 30 18:50:16.354099 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:16.354012    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10240f17-6wtp6" event={"ID":"d0df7654-ec74-41ac-adfc-bacec681ff36","Type":"ContainerStarted","Data":"9514507620a340f15c35151fa4c03e8477ac7a3b24d7ce137230625a7a3076fb"}
Oct 30 18:50:16.740783 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:16.740747    4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-68cb9b4b95-sgtll container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.24:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 18:50:16.741144 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:16.741114    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-sgtll" podUID="85ce0a7f-b7fb-4909-bfd6-affca6da6d49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.24:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:50:17.366300 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:17.366266    4525 generic.go:356] "Generic (PLEG): container finished" podID="d0df7654-ec74-41ac-adfc-bacec681ff36" containerID="0b2741fe94f37d0a61ff2ceadc18faaecf784f00a6fe34b161c9f97aaac991e6" exitCode=0
Oct 30 18:50:17.366694 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:17.366345    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10240f17-6wtp6" event={"ID":"d0df7654-ec74-41ac-adfc-bacec681ff36","Type":"ContainerDied","Data":"0b2741fe94f37d0a61ff2ceadc18faaecf784f00a6fe34b161c9f97aaac991e6"}
Oct 30 18:50:17.403472 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:50:17.403430    4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d49l" containerName="extractor"
Oct 30 18:50:18.517883 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:18.517858    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10240f17-6wtp6"
Oct 30 18:50:18.582136 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:18.582098    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0df7654-ec74-41ac-adfc-bacec681ff36-kube-api-access-vjc28\" (UniqueName: \"kubernetes.io/projected/d0df7654-ec74-41ac-adfc-bacec681ff36-kube-api-access-vjc28\") pod \"d0df7654-ec74-41ac-adfc-bacec681ff36\" (UID: \"d0df7654-ec74-41ac-adfc-bacec681ff36\") "
Oct 30 18:50:18.582543 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:18.582522    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0df7654-ec74-41ac-adfc-bacec681ff36-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0df7654-ec74-41ac-adfc-bacec681ff36-maas-api-xks-ca\") pod \"d0df7654-ec74-41ac-adfc-bacec681ff36\" (UID: \"d0df7654-ec74-41ac-adfc-bacec681ff36\") "
Oct 30 18:50:18.582708 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:18.582672    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0df7654-ec74-41ac-adfc-bacec681ff36-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0df7654-ec74-41ac-adfc-bacec681ff36-maas-api-openshift-ca\") pod \"d0df7654-ec74-41ac-adfc-bacec681ff36\" (UID: \"d0df7654-ec74-41ac-adfc-bacec681ff36\") "
Oct 30 18:50:18.583719 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:18.583672    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0df7654-ec74-41ac-adfc-bacec681ff36-maas-api-openshift-ca" pod "d0df7654-ec74-41ac-adfc-bacec681ff36" (UID: "d0df7654-ec74-41ac-adfc-bacec681ff36"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:50:18.589130 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:18.589098    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0df7654-ec74-41ac-adfc-bacec681ff36-kube-api-access-vjc28" pod "d0df7654-ec74-41ac-adfc-bacec681ff36" (UID: "d0df7654-ec74-41ac-adfc-bacec681ff36"). InnerVolumeSpecName "kube-api-access-vjc28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:50:18.589897 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:18.589866    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0df7654-ec74-41ac-adfc-bacec681ff36-maas-api-xks-ca" pod "d0df7654-ec74-41ac-adfc-bacec681ff36" (UID: "d0df7654-ec74-41ac-adfc-bacec681ff36"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:50:18.684045 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:18.683970    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjc28\" (UniqueName: \"kubernetes.io/projected/d0df7654-ec74-41ac-adfc-bacec681ff36-kube-api-access-vjc28\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:18.684045 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:18.683999    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0df7654-ec74-41ac-adfc-bacec681ff36-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:18.684045 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:18.684009    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0df7654-ec74-41ac-adfc-bacec681ff36-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:19.381317 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:19.381288    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10240f17-6wtp6"
Oct 30 18:50:19.381470 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:19.381322    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10240f17-6wtp6" event={"ID":"d0df7654-ec74-41ac-adfc-bacec681ff36","Type":"ContainerDied","Data":"9514507620a340f15c35151fa4c03e8477ac7a3b24d7ce137230625a7a3076fb"}
Oct 30 18:50:19.381470 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:19.381391    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9514507620a340f15c35151fa4c03e8477ac7a3b24d7ce137230625a7a3076fb"
Oct 30 18:50:21.422252 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:21.422217    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6bc5d5dd85-4mgzj"]
Oct 30 18:50:21.422658 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:21.422644    4525 cpu_manager.go:414] "containerMap: removing container" podUID="d0df7654-ec74-41ac-adfc-bacec681ff36" containerName="revoke-keys"
Oct 30 18:50:21.422733 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:21.422658    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0df7654-ec74-41ac-adfc-bacec681ff36" containerName="revoke-keys"
Oct 30 18:50:21.422795 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:21.422784    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0df7654-ec74-41ac-adfc-bacec681ff36" containerName="revoke-keys"
Oct 30 18:50:21.426166 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:21.426151    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6bc5d5dd85-4mgzj"
Oct 30 18:50:21.432714 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:21.432657    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-qv9w2\""
Oct 30 18:50:21.435487 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:21.435466    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6bc5d5dd85-4mgzj"]
Oct 30 18:50:21.515224 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:21.515194    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/510b0958-2710-413f-8da2-410728868b76-plugins-config-volume\") pod \"payload-processing-external-model-6bc5d5dd85-4mgzj\" (UID: \"510b0958-2710-413f-8da2-410728868b76\") " pod="models-as-a-service/payload-processing-external-model-6bc5d5dd85-4mgzj"
Oct 30 18:50:21.515356 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:21.515337    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82j8l\" (UniqueName: \"kubernetes.io/projected/510b0958-2710-413f-8da2-410728868b76-kube-api-access-82j8l\") pod \"payload-processing-external-model-6bc5d5dd85-4mgzj\" (UID: \"510b0958-2710-413f-8da2-410728868b76\") " pod="models-as-a-service/payload-processing-external-model-6bc5d5dd85-4mgzj"
Oct 30 18:50:21.515404 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:21.515372    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/510b0958-2710-413f-8da2-410728868b76-routing-overlay-volume\") pod \"payload-processing-external-model-6bc5d5dd85-4mgzj\" (UID: \"510b0958-2710-413f-8da2-410728868b76\") " pod="models-as-a-service/payload-processing-external-model-6bc5d5dd85-4mgzj"
Oct 30 18:50:21.515436 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:21.515401    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/510b0958-2710-413f-8da2-410728868b76-provider-credentials\") pod \"payload-processing-external-model-6bc5d5dd85-4mgzj\" (UID: \"510b0958-2710-413f-8da2-410728868b76\") " pod="models-as-a-service/payload-processing-external-model-6bc5d5dd85-4mgzj"
Oct 30 18:50:21.616096 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:21.616064    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82j8l\" (UniqueName: \"kubernetes.io/projected/510b0958-2710-413f-8da2-410728868b76-kube-api-access-82j8l\") pod \"payload-processing-external-model-6bc5d5dd85-4mgzj\" (UID: \"510b0958-2710-413f-8da2-410728868b76\") " pod="models-as-a-service/payload-processing-external-model-6bc5d5dd85-4mgzj"
Oct 30 18:50:21.616096 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:21.616101    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/510b0958-2710-413f-8da2-410728868b76-routing-overlay-volume\") pod \"payload-processing-external-model-6bc5d5dd85-4mgzj\" (UID: \"510b0958-2710-413f-8da2-410728868b76\") " pod="models-as-a-service/payload-processing-external-model-6bc5d5dd85-4mgzj"
Oct 30 18:50:21.616393 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:21.616126    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/510b0958-2710-413f-8da2-410728868b76-provider-credentials\") pod \"payload-processing-external-model-6bc5d5dd85-4mgzj\" (UID: \"510b0958-2710-413f-8da2-410728868b76\") " pod="models-as-a-service/payload-processing-external-model-6bc5d5dd85-4mgzj"
Oct 30 18:50:21.616393 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:21.616192    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/510b0958-2710-413f-8da2-410728868b76-plugins-config-volume\") pod \"payload-processing-external-model-6bc5d5dd85-4mgzj\" (UID: \"510b0958-2710-413f-8da2-410728868b76\") " pod="models-as-a-service/payload-processing-external-model-6bc5d5dd85-4mgzj"
Oct 30 18:50:21.616818 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:21.616796    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/510b0958-2710-413f-8da2-410728868b76-routing-overlay-volume\") pod \"payload-processing-external-model-6bc5d5dd85-4mgzj\" (UID: \"510b0958-2710-413f-8da2-410728868b76\") " pod="models-as-a-service/payload-processing-external-model-6bc5d5dd85-4mgzj"
Oct 30 18:50:21.616941 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:21.616922    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/510b0958-2710-413f-8da2-410728868b76-plugins-config-volume\") pod \"payload-processing-external-model-6bc5d5dd85-4mgzj\" (UID: \"510b0958-2710-413f-8da2-410728868b76\") " pod="models-as-a-service/payload-processing-external-model-6bc5d5dd85-4mgzj"
Oct 30 18:50:21.618892 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:21.618875    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/510b0958-2710-413f-8da2-410728868b76-provider-credentials\") pod \"payload-processing-external-model-6bc5d5dd85-4mgzj\" (UID: \"510b0958-2710-413f-8da2-410728868b76\") " pod="models-as-a-service/payload-processing-external-model-6bc5d5dd85-4mgzj"
Oct 30 18:50:21.624374 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:21.624348    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82j8l\" (UniqueName: \"kubernetes.io/projected/510b0958-2710-413f-8da2-410728868b76-kube-api-access-82j8l\") pod \"payload-processing-external-model-6bc5d5dd85-4mgzj\" (UID: \"510b0958-2710-413f-8da2-410728868b76\") " pod="models-as-a-service/payload-processing-external-model-6bc5d5dd85-4mgzj"
Oct 30 18:50:21.761835 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:21.761750    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6bc5d5dd85-4mgzj"
Oct 30 18:50:22.039334 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:22.039281    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6bc5d5dd85-4mgzj"]
Oct 30 18:50:22.043159 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:50:22.043124    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod510b0958_2710_413f_8da2_410728868b76.slice/crio-98f8d1930ed61c30bfe3ba8ac561c4b1a9b0d59b5b2a43743e93c0d0eb624db6 WatchSource:0}: Error finding container 98f8d1930ed61c30bfe3ba8ac561c4b1a9b0d59b5b2a43743e93c0d0eb624db6: Status 404 returned error can't find the container with id 98f8d1930ed61c30bfe3ba8ac561c4b1a9b0d59b5b2a43743e93c0d0eb624db6
Oct 30 18:50:22.401556 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:22.401516    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6bc5d5dd85-4mgzj" event={"ID":"510b0958-2710-413f-8da2-410728868b76","Type":"ContainerStarted","Data":"db7e7cd7d9b18d8acbb8a838942dc807cb70d0c7ccf08e68f2f2292a4af4bdcc"}
Oct 30 18:50:22.401769 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:22.401582    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6bc5d5dd85-4mgzj" event={"ID":"510b0958-2710-413f-8da2-410728868b76","Type":"ContainerStarted","Data":"98f8d1930ed61c30bfe3ba8ac561c4b1a9b0d59b5b2a43743e93c0d0eb624db6"}
Oct 30 18:50:22.401769 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:22.401608    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6bc5d5dd85-4mgzj"
Oct 30 18:50:22.419341 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:22.419293    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6bc5d5dd85-4mgzj" podStartSLOduration=1.4192803330000001 podStartE2EDuration="1.419280333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:50:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:50:22.417906625 +0000 UTC m=+845.043576590" watchObservedRunningTime="2026-10-30 18:50:22.419280333 +0000 UTC m=+845.044950297"
Oct 30 18:50:25.991930 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:25.991896    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-39bde3/e2e-conflict-b-8eedd2a0-kserve-84fcfc5594-x4szz"]
Oct 30 18:50:25.995920 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:25.995901    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-39bde3/e2e-conflict-b-8eedd2a0-kserve-84fcfc5594-x4szz"
Oct 30 18:50:26.001719 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:26.001655    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-39bde3\"/\"e2e-conflict-b-8eedd2a0-kserve-self-signed-certs\""
Oct 30 18:50:26.018723 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:26.018667    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-39bde3/e2e-conflict-b-8eedd2a0-kserve-84fcfc5594-x4szz"]
Oct 30 18:50:26.077344 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:26.077311    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a-dshm\") pod \"e2e-conflict-b-8eedd2a0-kserve-84fcfc5594-x4szz\" (UID: \"5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a\") " pod="e2e-models-e2e-worker-w2-39bde3/e2e-conflict-b-8eedd2a0-kserve-84fcfc5594-x4szz"
Oct 30 18:50:26.077527 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:26.077460    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a-tmp-dir\") pod \"e2e-conflict-b-8eedd2a0-kserve-84fcfc5594-x4szz\" (UID: \"5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a\") " pod="e2e-models-e2e-worker-w2-39bde3/e2e-conflict-b-8eedd2a0-kserve-84fcfc5594-x4szz"
Oct 30 18:50:26.077527 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:26.077512    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a-tls-certs\") pod \"e2e-conflict-b-8eedd2a0-kserve-84fcfc5594-x4szz\" (UID: \"5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a\") " pod="e2e-models-e2e-worker-w2-39bde3/e2e-conflict-b-8eedd2a0-kserve-84fcfc5594-x4szz"
Oct 30 18:50:26.077643 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:26.077532    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdc5p\" (UniqueName: \"kubernetes.io/projected/5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a-kube-api-access-rdc5p\") pod \"e2e-conflict-b-8eedd2a0-kserve-84fcfc5594-x4szz\" (UID: \"5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a\") " pod="e2e-models-e2e-worker-w2-39bde3/e2e-conflict-b-8eedd2a0-kserve-84fcfc5594-x4szz"
Oct 30 18:50:26.077643 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:26.077600    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a-model-cache\") pod \"e2e-conflict-b-8eedd2a0-kserve-84fcfc5594-x4szz\" (UID: \"5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a\") " pod="e2e-models-e2e-worker-w2-39bde3/e2e-conflict-b-8eedd2a0-kserve-84fcfc5594-x4szz"
Oct 30 18:50:26.077769 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:26.077738    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a-home\") pod \"e2e-conflict-b-8eedd2a0-kserve-84fcfc5594-x4szz\" (UID: \"5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a\") " pod="e2e-models-e2e-worker-w2-39bde3/e2e-conflict-b-8eedd2a0-kserve-84fcfc5594-x4szz"
Oct 30 18:50:26.178370 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:26.178329    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a-tmp-dir\") pod \"e2e-conflict-b-8eedd2a0-kserve-84fcfc5594-x4szz\" (UID: \"5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a\") " pod="e2e-models-e2e-worker-w2-39bde3/e2e-conflict-b-8eedd2a0-kserve-84fcfc5594-x4szz"
Oct 30 18:50:26.178543 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:26.178399    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a-tls-certs\") pod \"e2e-conflict-b-8eedd2a0-kserve-84fcfc5594-x4szz\" (UID: \"5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a\") " pod="e2e-models-e2e-worker-w2-39bde3/e2e-conflict-b-8eedd2a0-kserve-84fcfc5594-x4szz"
Oct 30 18:50:26.178543 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:26.178428    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdc5p\" (UniqueName: \"kubernetes.io/projected/5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a-kube-api-access-rdc5p\") pod \"e2e-conflict-b-8eedd2a0-kserve-84fcfc5594-x4szz\" (UID: \"5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a\") " pod="e2e-models-e2e-worker-w2-39bde3/e2e-conflict-b-8eedd2a0-kserve-84fcfc5594-x4szz"
Oct 30 18:50:26.178543 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:26.178477    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a-model-cache\") pod \"e2e-conflict-b-8eedd2a0-kserve-84fcfc5594-x4szz\" (UID: \"5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a\") " pod="e2e-models-e2e-worker-w2-39bde3/e2e-conflict-b-8eedd2a0-kserve-84fcfc5594-x4szz"
Oct 30 18:50:26.178543 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:26.178506    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a-home\") pod \"e2e-conflict-b-8eedd2a0-kserve-84fcfc5594-x4szz\" (UID: \"5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a\") " pod="e2e-models-e2e-worker-w2-39bde3/e2e-conflict-b-8eedd2a0-kserve-84fcfc5594-x4szz"
Oct 30 18:50:26.178820 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:26.178550    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a-dshm\") pod \"e2e-conflict-b-8eedd2a0-kserve-84fcfc5594-x4szz\" (UID: \"5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a\") " pod="e2e-models-e2e-worker-w2-39bde3/e2e-conflict-b-8eedd2a0-kserve-84fcfc5594-x4szz"
Oct 30 18:50:26.178880 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:26.178837    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a-tmp-dir\") pod \"e2e-conflict-b-8eedd2a0-kserve-84fcfc5594-x4szz\" (UID: \"5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a\") " pod="e2e-models-e2e-worker-w2-39bde3/e2e-conflict-b-8eedd2a0-kserve-84fcfc5594-x4szz"
Oct 30 18:50:26.178880 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:26.178854    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a-model-cache\") pod \"e2e-conflict-b-8eedd2a0-kserve-84fcfc5594-x4szz\" (UID: \"5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a\") " pod="e2e-models-e2e-worker-w2-39bde3/e2e-conflict-b-8eedd2a0-kserve-84fcfc5594-x4szz"
Oct 30 18:50:26.178981 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:26.178914    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a-home\") pod \"e2e-conflict-b-8eedd2a0-kserve-84fcfc5594-x4szz\" (UID: \"5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a\") " pod="e2e-models-e2e-worker-w2-39bde3/e2e-conflict-b-8eedd2a0-kserve-84fcfc5594-x4szz"
Oct 30 18:50:26.180897 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:26.180870    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a-dshm\") pod \"e2e-conflict-b-8eedd2a0-kserve-84fcfc5594-x4szz\" (UID: \"5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a\") " pod="e2e-models-e2e-worker-w2-39bde3/e2e-conflict-b-8eedd2a0-kserve-84fcfc5594-x4szz"
Oct 30 18:50:26.181134 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:26.181117    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a-tls-certs\") pod \"e2e-conflict-b-8eedd2a0-kserve-84fcfc5594-x4szz\" (UID: \"5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a\") " pod="e2e-models-e2e-worker-w2-39bde3/e2e-conflict-b-8eedd2a0-kserve-84fcfc5594-x4szz"
Oct 30 18:50:26.186780 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:26.186761    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdc5p\" (UniqueName: \"kubernetes.io/projected/5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a-kube-api-access-rdc5p\") pod \"e2e-conflict-b-8eedd2a0-kserve-84fcfc5594-x4szz\" (UID: \"5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a\") " pod="e2e-models-e2e-worker-w2-39bde3/e2e-conflict-b-8eedd2a0-kserve-84fcfc5594-x4szz"
Oct 30 18:50:26.312452 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:26.312354    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-39bde3/e2e-conflict-b-8eedd2a0-kserve-84fcfc5594-x4szz"
Oct 30 18:50:26.644096 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:26.644058    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-39bde3/e2e-conflict-b-8eedd2a0-kserve-84fcfc5594-x4szz"]
Oct 30 18:50:27.113706 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:27.113054    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-tckhm"]
Oct 30 18:50:27.156022 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:27.155306    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-6pb9d"]
Oct 30 18:50:27.156022 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:27.155433    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-tckhm"
Oct 30 18:50:27.166841 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:27.166815    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-tckhm"]
Oct 30 18:50:27.167043 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:27.167027    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-6pb9d"]
Oct 30 18:50:27.167172 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:27.167152    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-6pb9d"
Oct 30 18:50:27.173081 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:27.173066    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-39bde3-openshift-default-dockercfg-vwm86\""
Oct 30 18:50:27.311298 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:27.311253    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21d3419d-d29b-4e2b-8598-fe844028161e-credential-socket\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-6pb9d\" (UID: \"21d3419d-d29b-4e2b-8598-fe844028161e\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-6pb9d"
Oct 30 18:50:27.311776 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:27.311582    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpz8v\" (UniqueName: \"kubernetes.io/projected/23ad1447-7842-4b64-b99f-adf61a803291-kube-api-access-hpz8v\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-tckhm\" (UID: \"23ad1447-7842-4b64-b99f-adf61a803291\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-tckhm"
Oct 30 18:50:27.311910 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:27.311803    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvpks\" (UniqueName: \"kubernetes.io/projected/21d3419d-d29b-4e2b-8598-fe844028161e-kube-api-access-jvpks\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-6pb9d\" (UID: \"21d3419d-d29b-4e2b-8598-fe844028161e\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-6pb9d"
Oct 30 18:50:27.311910 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:27.311837    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21d3419d-d29b-4e2b-8598-fe844028161e-istio-token\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-6pb9d\" (UID: \"21d3419d-d29b-4e2b-8598-fe844028161e\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-6pb9d"
Oct 30 18:50:27.311910 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:27.311862    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23ad1447-7842-4b64-b99f-adf61a803291-credential-socket\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-tckhm\" (UID: \"23ad1447-7842-4b64-b99f-adf61a803291\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-tckhm"
Oct 30 18:50:27.312085 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:27.311917    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23ad1447-7842-4b64-b99f-adf61a803291-istio-envoy\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-tckhm\" (UID: \"23ad1447-7842-4b64-b99f-adf61a803291\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-tckhm"
Oct 30 18:50:27.312085 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:27.312052    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23ad1447-7842-4b64-b99f-adf61a803291-workload-certs\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-tckhm\" (UID: \"23ad1447-7842-4b64-b99f-adf61a803291\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-tckhm"
Oct 30 18:50:27.312189 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:27.312094    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23ad1447-7842-4b64-b99f-adf61a803291-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-tckhm\" (UID: \"23ad1447-7842-4b64-b99f-adf61a803291\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-tckhm"
Oct 30 18:50:27.312189 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:27.312143    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21d3419d-d29b-4e2b-8598-fe844028161e-istiod-ca-cert\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-6pb9d\" (UID: \"21d3419d-d29b-4e2b-8598-fe844028161e\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-6pb9d"
Oct 30 18:50:27.312295 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:27.312207    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21d3419d-d29b-4e2b-8598-fe844028161e-istio-envoy\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-6pb9d\" (UID: \"21d3419d-d29b-4e2b-8598-fe844028161e\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-6pb9d"
Oct 30 18:50:27.312295 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:27.312239    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23ad1447-7842-4b64-b99f-adf61a803291-workload-socket\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-tckhm\" (UID: \"23ad1447-7842-4b64-b99f-adf61a803291\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-tckhm"
Oct 30 18:50:27.312295 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:27.312269    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23ad1447-7842-4b64-b99f-adf61a803291-istio-token\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-tckhm\" (UID: \"23ad1447-7842-4b64-b99f-adf61a803291\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-tckhm"
Oct 30 18:50:27.312440 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:27.312315    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21d3419d-d29b-4e2b-8598-fe844028161e-workload-socket\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-6pb9d\" (UID: \"21d3419d-d29b-4e2b-8598-fe844028161e\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-6pb9d"
Oct 30 18:50:27.312440 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:27.312342    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21d3419d-d29b-4e2b-8598-fe844028161e-istio-data\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-6pb9d\" (UID: \"21d3419d-d29b-4e2b-8598-fe844028161e\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-6pb9d"
Oct 30 18:50:27.312440 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:27.312407    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23ad1447-7842-4b64-b99f-adf61a803291-istio-podinfo\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-tckhm\" (UID: \"23ad1447-7842-4b64-b99f-adf61a803291\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-tckhm"
Oct 30 18:50:27.312595 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:27.312441    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21d3419d-d29b-4e2b-8598-fe844028161e-workload-certs\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-6pb9d\" (UID: \"21d3419d-d29b-4e2b-8598-fe844028161e\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-6pb9d"
Oct 30 18:50:27.312595 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:27.312487    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23ad1447-7842-4b64-b99f-adf61a803291-istio-data\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-tckhm\" (UID: \"23ad1447-7842-4b64-b99f-adf61a803291\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-tckhm"
Oct 30 18:50:27.312595 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:27.312518    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21d3419d-d29b-4e2b-8598-fe844028161e-istio-podinfo\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-6pb9d\" (UID: \"21d3419d-d29b-4e2b-8598-fe844028161e\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-6pb9d"
Oct 30 18:50:27.427770 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:27.427355    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23ad1447-7842-4b64-b99f-adf61a803291-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-tckhm\" (UID: \"23ad1447-7842-4b64-b99f-adf61a803291\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-tckhm"
Oct 30 18:50:27.427770 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:27.427435    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21d3419d-d29b-4e2b-8598-fe844028161e-istiod-ca-cert\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-6pb9d\" (UID: \"21d3419d-d29b-4e2b-8598-fe844028161e\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-6pb9d"
Oct 30 18:50:27.427770 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:27.427499    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21d3419d-d29b-4e2b-8598-fe844028161e-istio-envoy\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-6pb9d\" (UID: \"21d3419d-d29b-4e2b-8598-fe844028161e\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-6pb9d"
Oct 30 18:50:27.427770 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:27.427533    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23ad1447-7842-4b64-b99f-adf61a803291-workload-socket\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-tckhm\" (UID: \"23ad1447-7842-4b64-b99f-adf61a803291\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-tckhm"
Oct 30 18:50:27.427770 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:27.427559    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23ad1447-7842-4b64-b99f-adf61a803291-istio-token\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-tckhm\" (UID: \"23ad1447-7842-4b64-b99f-adf61a803291\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-tckhm"
Oct 30 18:50:27.427770 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:27.427602    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21d3419d-d29b-4e2b-8598-fe844028161e-workload-socket\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-6pb9d\" (UID: \"21d3419d-d29b-4e2b-8598-fe844028161e\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-6pb9d"
Oct 30 18:50:27.427770 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:27.427625    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21d3419d-d29b-4e2b-8598-fe844028161e-istio-data\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-6pb9d\" (UID: \"21d3419d-d29b-4e2b-8598-fe844028161e\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-6pb9d"
Oct 30 18:50:27.427770 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:27.427730    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23ad1447-7842-4b64-b99f-adf61a803291-istio-podinfo\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-tckhm\" (UID: \"23ad1447-7842-4b64-b99f-adf61a803291\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-tckhm"
Oct 30 18:50:27.427770 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:27.427766    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21d3419d-d29b-4e2b-8598-fe844028161e-workload-certs\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-6pb9d\" (UID: \"21d3419d-d29b-4e2b-8598-fe844028161e\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-6pb9d"
Oct 30 18:50:27.428393 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:27.427813    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23ad1447-7842-4b64-b99f-adf61a803291-istio-data\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-tckhm\" (UID: \"23ad1447-7842-4b64-b99f-adf61a803291\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-tckhm"
Oct 30 18:50:27.428393 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:27.427844    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21d3419d-d29b-4e2b-8598-fe844028161e-istio-podinfo\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-6pb9d\" (UID: \"21d3419d-d29b-4e2b-8598-fe844028161e\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-6pb9d"
Oct 30 18:50:27.428393 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:27.427893    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21d3419d-d29b-4e2b-8598-fe844028161e-credential-socket\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-6pb9d\" (UID: \"21d3419d-d29b-4e2b-8598-fe844028161e\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-6pb9d"
Oct 30 18:50:27.428393 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:27.427946    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpz8v\" (UniqueName: \"kubernetes.io/projected/23ad1447-7842-4b64-b99f-adf61a803291-kube-api-access-hpz8v\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-tckhm\" (UID: \"23ad1447-7842-4b64-b99f-adf61a803291\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-tckhm"
Oct 30 18:50:27.428393 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:27.427984    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvpks\" (UniqueName: \"kubernetes.io/projected/21d3419d-d29b-4e2b-8598-fe844028161e-kube-api-access-jvpks\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-6pb9d\" (UID: \"21d3419d-d29b-4e2b-8598-fe844028161e\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-6pb9d"
Oct 30 18:50:27.428393 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:27.428013    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21d3419d-d29b-4e2b-8598-fe844028161e-istio-token\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-6pb9d\" (UID: \"21d3419d-d29b-4e2b-8598-fe844028161e\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-6pb9d"
Oct 30 18:50:27.428393 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:27.428038    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23ad1447-7842-4b64-b99f-adf61a803291-credential-socket\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-tckhm\" (UID: \"23ad1447-7842-4b64-b99f-adf61a803291\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-tckhm"
Oct 30 18:50:27.428393 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:27.428062    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23ad1447-7842-4b64-b99f-adf61a803291-istio-envoy\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-tckhm\" (UID: \"23ad1447-7842-4b64-b99f-adf61a803291\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-tckhm"
Oct 30 18:50:27.428393 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:27.428116    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23ad1447-7842-4b64-b99f-adf61a803291-workload-certs\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-tckhm\" (UID: \"23ad1447-7842-4b64-b99f-adf61a803291\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-tckhm"
Oct 30 18:50:27.428960 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:27.428588    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23ad1447-7842-4b64-b99f-adf61a803291-workload-certs\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-tckhm\" (UID: \"23ad1447-7842-4b64-b99f-adf61a803291\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-tckhm"
Oct 30 18:50:27.435516 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:27.429265    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23ad1447-7842-4b64-b99f-adf61a803291-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-tckhm\" (UID: \"23ad1447-7842-4b64-b99f-adf61a803291\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-tckhm"
Oct 30 18:50:27.435516 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:27.429920    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21d3419d-d29b-4e2b-8598-fe844028161e-istiod-ca-cert\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-6pb9d\" (UID: \"21d3419d-d29b-4e2b-8598-fe844028161e\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-6pb9d"
Oct 30 18:50:27.435516 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:27.430790    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23ad1447-7842-4b64-b99f-adf61a803291-istio-data\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-tckhm\" (UID: \"23ad1447-7842-4b64-b99f-adf61a803291\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-tckhm"
Oct 30 18:50:27.435516 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:27.431037    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23ad1447-7842-4b64-b99f-adf61a803291-workload-socket\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-tckhm\" (UID: \"23ad1447-7842-4b64-b99f-adf61a803291\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-tckhm"
Oct 30 18:50:27.435516 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:27.431596    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21d3419d-d29b-4e2b-8598-fe844028161e-workload-socket\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-6pb9d\" (UID: \"21d3419d-d29b-4e2b-8598-fe844028161e\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-6pb9d"
Oct 30 18:50:27.435516 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:27.431855    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21d3419d-d29b-4e2b-8598-fe844028161e-istio-data\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-6pb9d\" (UID: \"21d3419d-d29b-4e2b-8598-fe844028161e\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-6pb9d"
Oct 30 18:50:27.435516 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:27.435162    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21d3419d-d29b-4e2b-8598-fe844028161e-workload-certs\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-6pb9d\" (UID: \"21d3419d-d29b-4e2b-8598-fe844028161e\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-6pb9d"
Oct 30 18:50:27.435516 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:27.435415    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23ad1447-7842-4b64-b99f-adf61a803291-credential-socket\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-tckhm\" (UID: \"23ad1447-7842-4b64-b99f-adf61a803291\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-tckhm"
Oct 30 18:50:27.438020 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:27.435887    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21d3419d-d29b-4e2b-8598-fe844028161e-credential-socket\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-6pb9d\" (UID: \"21d3419d-d29b-4e2b-8598-fe844028161e\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-6pb9d"
Oct 30 18:50:27.438020 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:27.435926    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21d3419d-d29b-4e2b-8598-fe844028161e-istio-envoy\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-6pb9d\" (UID: \"21d3419d-d29b-4e2b-8598-fe844028161e\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-6pb9d"
Oct 30 18:50:27.438020 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:27.436518    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23ad1447-7842-4b64-b99f-adf61a803291-istio-podinfo\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-tckhm\" (UID: \"23ad1447-7842-4b64-b99f-adf61a803291\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-tckhm"
Oct 30 18:50:27.455151 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:27.455090    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvpks\" (UniqueName: \"kubernetes.io/projected/21d3419d-d29b-4e2b-8598-fe844028161e-kube-api-access-jvpks\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-6pb9d\" (UID: \"21d3419d-d29b-4e2b-8598-fe844028161e\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-6pb9d"
Oct 30 18:50:27.456176 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:27.455796    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23ad1447-7842-4b64-b99f-adf61a803291-istio-token\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-tckhm\" (UID: \"23ad1447-7842-4b64-b99f-adf61a803291\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-tckhm"
Oct 30 18:50:27.456176 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:27.456081    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23ad1447-7842-4b64-b99f-adf61a803291-istio-envoy\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-tckhm\" (UID: \"23ad1447-7842-4b64-b99f-adf61a803291\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-tckhm"
Oct 30 18:50:27.456176 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:27.456131    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21d3419d-d29b-4e2b-8598-fe844028161e-istio-podinfo\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-6pb9d\" (UID: \"21d3419d-d29b-4e2b-8598-fe844028161e\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-6pb9d"
Oct 30 18:50:27.456470 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:27.456450    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21d3419d-d29b-4e2b-8598-fe844028161e-istio-token\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-6pb9d\" (UID: \"21d3419d-d29b-4e2b-8598-fe844028161e\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-6pb9d"
Oct 30 18:50:27.459179 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:27.459106    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpz8v\" (UniqueName: \"kubernetes.io/projected/23ad1447-7842-4b64-b99f-adf61a803291-kube-api-access-hpz8v\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-tckhm\" (UID: \"23ad1447-7842-4b64-b99f-adf61a803291\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-tckhm"
Oct 30 18:50:27.465090 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:27.465063    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-39bde3/e2e-conflict-b-8eedd2a0-kserve-84fcfc5594-x4szz" event={"ID":"5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a","Type":"ContainerStarted","Data":"b7894f7423b5a1f4d8133c0ed7694e8d30d07a11d9718a673d625821b1b04709"}
Oct 30 18:50:27.465203 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:27.465108    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-39bde3/e2e-conflict-b-8eedd2a0-kserve-84fcfc5594-x4szz" event={"ID":"5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a","Type":"ContainerStarted","Data":"318412c9b2f4629e4718fdf8c3f013650ceb1d5378580ecd0bcb5bfea963efa1"}
Oct 30 18:50:27.492895 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:27.489106    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-tckhm"
Oct 30 18:50:27.525645 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:27.524402    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-6pb9d"
Oct 30 18:50:27.845741 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:27.817918    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-39bde3/e2e-conflict-b-8eedd2a0-kserve-84fcfc5594-x4szz" podStartSLOduration=2.817866982 podStartE2EDuration="2.817866982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:50:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:50:27.488938657 +0000 UTC m=+850.114608623" watchObservedRunningTime="2026-10-30 18:50:27.817866982 +0000 UTC m=+850.443536946"
Oct 30 18:50:27.845741 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:27.833868    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-tckhm"]
Oct 30 18:50:27.853345 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:27.853305    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:50:27.853434 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:27.853401    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:50:27.853472 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:27.853443    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:50:27.930320 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:27.929188    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:50:27.930320 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:27.929279    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:50:27.930320 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:27.929323    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:50:27.931242 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:27.931214    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-6pb9d"]
Oct 30 18:50:28.324088 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:28.323963    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39bde3/e2e-conflict-b-8eedd2a0-kserve-84fcfc5594-x4szz"
Oct 30 18:50:28.474405 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:28.474359    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-6pb9d" event={"ID":"21d3419d-d29b-4e2b-8598-fe844028161e","Type":"ContainerStarted","Data":"a4836020c1f29976f62b485d0adb521ab0a7fddc503f4cbb0259bae8cdd1da30"}
Oct 30 18:50:28.474592 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:28.474458    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-6pb9d" event={"ID":"21d3419d-d29b-4e2b-8598-fe844028161e","Type":"ContainerStarted","Data":"c1de833ef5efcb0650d72119599f9ca9c4363e2c900cf1db9932b1f1941105b2"}
Oct 30 18:50:28.476326 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:28.476294    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-tckhm" event={"ID":"23ad1447-7842-4b64-b99f-adf61a803291","Type":"ContainerStarted","Data":"525658dbf81810aa40953b1b7f429e101ebc5295b76dac02844b42dc442e64f7"}
Oct 30 18:50:28.476503 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:28.476487    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-tckhm" event={"ID":"23ad1447-7842-4b64-b99f-adf61a803291","Type":"ContainerStarted","Data":"03a2e1e58464edc9df7dea36a60e484644c7a9fc3ad9cdde4a4ec3e8170f4bc5"}
Oct 30 18:50:28.491411 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:28.491384    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-tckhm"
Oct 30 18:50:28.491713 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:28.491670    4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-68cb9b4b95-tckhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 30 18:50:28.491878 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:28.491853    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-tckhm" podUID="23ad1447-7842-4b64-b99f-adf61a803291" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 30 18:50:28.511099 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:28.511044    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-6pb9d" podStartSLOduration=1.51102641 podStartE2EDuration="1.51102641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:50:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:50:28.495102014 +0000 UTC m=+851.120771979" watchObservedRunningTime="2026-10-30 18:50:28.51102641 +0000 UTC m=+851.136696376"
Oct 30 18:50:28.530437 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:28.530406    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-6pb9d"
Oct 30 18:50:28.540870 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:28.540811    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-tckhm" podStartSLOduration=1.5407906900000001 podStartE2EDuration="1.54079069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:50:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:50:28.530202811 +0000 UTC m=+851.155872775" watchObservedRunningTime="2026-10-30 18:50:28.54079069 +0000 UTC m=+851.166460658"
Oct 30 18:50:29.495709 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:29.493806    4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-68cb9b4b95-tckhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 30 18:50:29.495709 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:29.493876    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-tckhm" podUID="23ad1447-7842-4b64-b99f-adf61a803291" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 30 18:50:29.530825 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:29.530792    4525 patch_prober.go:29] interesting pod/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-6pb9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 18:50:29.531114 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:29.531085    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-6pb9d" podUID="21d3419d-d29b-4e2b-8598-fe844028161e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 18:50:30.511706 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:30.511126    4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-68cb9b4b95-tckhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 30 18:50:30.511706 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:30.511198    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-tckhm" podUID="23ad1447-7842-4b64-b99f-adf61a803291" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 30 18:50:30.528212 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:30.528164    4525 patch_prober.go:29] interesting pod/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-6pb9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 18:50:30.528455 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:30.528424    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-6pb9d" podUID="21d3419d-d29b-4e2b-8598-fe844028161e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 18:50:31.541634 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:31.522835    4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-68cb9b4b95-tckhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 30 18:50:31.541634 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:31.522904    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-tckhm" podUID="23ad1447-7842-4b64-b99f-adf61a803291" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 30 18:50:31.541634 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:31.540793    4525 patch_prober.go:29] interesting pod/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-6pb9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 18:50:31.541634 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:31.540849    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-6pb9d" podUID="21d3419d-d29b-4e2b-8598-fe844028161e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 18:50:32.498708 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:32.496134    4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-68cb9b4b95-tckhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 30 18:50:32.498708 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:32.496211    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-tckhm" podUID="23ad1447-7842-4b64-b99f-adf61a803291" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 30 18:50:32.526176 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:32.526147    4525 patch_prober.go:29] interesting pod/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-6pb9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 18:50:32.526444 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:32.526407    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-6pb9d" podUID="21d3419d-d29b-4e2b-8598-fe844028161e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 18:50:33.423706 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:33.423386    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-6bc5d5dd85-4mgzj"
Oct 30 18:50:33.520708 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:33.520507    4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-68cb9b4b95-tckhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 30 18:50:33.520708 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:33.520571    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-tckhm" podUID="23ad1447-7842-4b64-b99f-adf61a803291" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 30 18:50:33.530708 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:33.530567    4525 patch_prober.go:29] interesting pod/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-6pb9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 18:50:33.530708 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:33.530621    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-6pb9d" podUID="21d3419d-d29b-4e2b-8598-fe844028161e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 18:50:33.535000 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:33.534903    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5c6fc54974-gq9dn"]
Oct 30 18:50:34.495187 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:34.495146    4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-68cb9b4b95-tckhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 30 18:50:34.495847 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:34.495813    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-tckhm" podUID="23ad1447-7842-4b64-b99f-adf61a803291" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 30 18:50:34.525894 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:34.525842    4525 patch_prober.go:29] interesting pod/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-6pb9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 18:50:34.526052 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:34.525918    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-6pb9d" podUID="21d3419d-d29b-4e2b-8598-fe844028161e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 18:50:35.489879 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:35.489835    4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-68cb9b4b95-tckhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 30 18:50:35.490066 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:35.489907    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-tckhm" podUID="23ad1447-7842-4b64-b99f-adf61a803291" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 30 18:50:35.562105 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:35.562066    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-6pb9d"
Oct 30 18:50:35.562574 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:35.562512    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-6pb9d"
Oct 30 18:50:35.563972 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:35.563949    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-6pb9d"
Oct 30 18:50:36.313534 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:36.313494    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-39bde3/e2e-conflict-b-8eedd2a0-kserve-84fcfc5594-x4szz"
Oct 30 18:50:36.328020 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:36.327988    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-39bde3/e2e-conflict-b-8eedd2a0-kserve-84fcfc5594-x4szz"
Oct 30 18:50:36.611415 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:36.611393    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39bde3/e2e-conflict-b-8eedd2a0-kserve-84fcfc5594-x4szz"
Oct 30 18:50:36.756025 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:36.755994    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj"]
Oct 30 18:50:36.761863 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:36.761837    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj"]
Oct 30 18:50:36.762071 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:36.761967    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj"
Oct 30 18:50:36.780279 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:36.780212    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-2889dd\"/\"distinct-w3-2889dd-kserve-self-signed-certs\""
Oct 30 18:50:36.830479 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:36.830446    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84c99c49-1d60-4176-bda4-cad29c7d801c-tmp-dir\") pod \"distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj\" (UID: \"84c99c49-1d60-4176-bda4-cad29c7d801c\") " pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj"
Oct 30 18:50:36.830658 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:36.830489    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84c99c49-1d60-4176-bda4-cad29c7d801c-tls-certs\") pod \"distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj\" (UID: \"84c99c49-1d60-4176-bda4-cad29c7d801c\") " pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj"
Oct 30 18:50:36.830764 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:36.830659    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84c99c49-1d60-4176-bda4-cad29c7d801c-model-cache\") pod \"distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj\" (UID: \"84c99c49-1d60-4176-bda4-cad29c7d801c\") " pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj"
Oct 30 18:50:36.830837 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:36.830770    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84c99c49-1d60-4176-bda4-cad29c7d801c-home\") pod \"distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj\" (UID: \"84c99c49-1d60-4176-bda4-cad29c7d801c\") " pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj"
Oct 30 18:50:36.830900 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:36.830837    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xctzt\" (UniqueName: \"kubernetes.io/projected/84c99c49-1d60-4176-bda4-cad29c7d801c-kube-api-access-xctzt\") pod \"distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj\" (UID: \"84c99c49-1d60-4176-bda4-cad29c7d801c\") " pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj"
Oct 30 18:50:36.830900 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:36.830886    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84c99c49-1d60-4176-bda4-cad29c7d801c-dshm\") pod \"distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj\" (UID: \"84c99c49-1d60-4176-bda4-cad29c7d801c\") " pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj"
Oct 30 18:50:36.933485 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:36.933347    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84c99c49-1d60-4176-bda4-cad29c7d801c-dshm\") pod \"distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj\" (UID: \"84c99c49-1d60-4176-bda4-cad29c7d801c\") " pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj"
Oct 30 18:50:36.933669 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:36.933509    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84c99c49-1d60-4176-bda4-cad29c7d801c-tmp-dir\") pod \"distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj\" (UID: \"84c99c49-1d60-4176-bda4-cad29c7d801c\") " pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj"
Oct 30 18:50:36.933669 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:36.933553    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84c99c49-1d60-4176-bda4-cad29c7d801c-tls-certs\") pod \"distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj\" (UID: \"84c99c49-1d60-4176-bda4-cad29c7d801c\") " pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj"
Oct 30 18:50:36.933895 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:36.933719    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84c99c49-1d60-4176-bda4-cad29c7d801c-model-cache\") pod \"distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj\" (UID: \"84c99c49-1d60-4176-bda4-cad29c7d801c\") " pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj"
Oct 30 18:50:36.933895 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:36.933769    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84c99c49-1d60-4176-bda4-cad29c7d801c-home\") pod \"distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj\" (UID: \"84c99c49-1d60-4176-bda4-cad29c7d801c\") " pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj"
Oct 30 18:50:36.933895 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:36.933804    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xctzt\" (UniqueName: \"kubernetes.io/projected/84c99c49-1d60-4176-bda4-cad29c7d801c-kube-api-access-xctzt\") pod \"distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj\" (UID: \"84c99c49-1d60-4176-bda4-cad29c7d801c\") " pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj"
Oct 30 18:50:36.934344 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:36.934317    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84c99c49-1d60-4176-bda4-cad29c7d801c-model-cache\") pod \"distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj\" (UID: \"84c99c49-1d60-4176-bda4-cad29c7d801c\") " pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj"
Oct 30 18:50:36.934528 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:36.934339    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84c99c49-1d60-4176-bda4-cad29c7d801c-home\") pod \"distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj\" (UID: \"84c99c49-1d60-4176-bda4-cad29c7d801c\") " pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj"
Oct 30 18:50:36.934657 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:36.934546    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84c99c49-1d60-4176-bda4-cad29c7d801c-tmp-dir\") pod \"distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj\" (UID: \"84c99c49-1d60-4176-bda4-cad29c7d801c\") " pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj"
Oct 30 18:50:36.937532 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:36.937511    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84c99c49-1d60-4176-bda4-cad29c7d801c-tls-certs\") pod \"distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj\" (UID: \"84c99c49-1d60-4176-bda4-cad29c7d801c\") " pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj"
Oct 30 18:50:36.942088 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:36.942052    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xctzt\" (UniqueName: \"kubernetes.io/projected/84c99c49-1d60-4176-bda4-cad29c7d801c-kube-api-access-xctzt\") pod \"distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj\" (UID: \"84c99c49-1d60-4176-bda4-cad29c7d801c\") " pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj"
Oct 30 18:50:36.942323 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:36.942302    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84c99c49-1d60-4176-bda4-cad29c7d801c-dshm\") pod \"distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj\" (UID: \"84c99c49-1d60-4176-bda4-cad29c7d801c\") " pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj"
Oct 30 18:50:37.032355 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:37.032323    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-tckhm"
Oct 30 18:50:37.032896 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:37.032877    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-tckhm"
Oct 30 18:50:37.033798 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:37.033776    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-tckhm"
Oct 30 18:50:37.109148 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:37.109116    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj"
Oct 30 18:50:37.395313 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:37.395281    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj"]
Oct 30 18:50:37.396525 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:50:37.396490    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod84c99c49_1d60_4176_bda4_cad29c7d801c.slice/crio-f152ac13cb8664c977bbf2508f57a16be644adc8b8935ee2eae71cc30724ca2d WatchSource:0}: Error finding container f152ac13cb8664c977bbf2508f57a16be644adc8b8935ee2eae71cc30724ca2d: Status 404 returned error can't find the container with id f152ac13cb8664c977bbf2508f57a16be644adc8b8935ee2eae71cc30724ca2d
Oct 30 18:50:37.602480 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:37.602442    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj" event={"ID":"84c99c49-1d60-4176-bda4-cad29c7d801c","Type":"ContainerStarted","Data":"4e5700a3ef9944f99c7b66b54ee07727051d81b2a5be20c6653b3dd34b9c3a9a"}
Oct 30 18:50:37.602664 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:37.602492    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj" event={"ID":"84c99c49-1d60-4176-bda4-cad29c7d801c","Type":"ContainerStarted","Data":"f152ac13cb8664c977bbf2508f57a16be644adc8b8935ee2eae71cc30724ca2d"}
Oct 30 18:50:37.620978 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:37.620934    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj" podStartSLOduration=1.6209193750000002 podStartE2EDuration="1.620919375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:50:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:50:37.618318116 +0000 UTC m=+860.243988080" watchObservedRunningTime="2026-10-30 18:50:37.620919375 +0000 UTC m=+860.246589338"
Oct 30 18:50:38.109520 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:38.109488    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj"
Oct 30 18:50:38.535928 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:38.535836    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5c6fc54974-gq9dn" podUID="354c6c7c-df51-4817-8369-620c5a0c65ff" containerName="payload-processing" containerID="cri-o://03bc274bcd24c7a02548a47f3d1397e3cf48077b13b5b628029a635f5a43a1ff" gracePeriod=25
Oct 30 18:50:38.809023 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:38.808993    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5c6fc54974-gq9dn"
Oct 30 18:50:38.856803 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:38.856771    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/354c6c7c-df51-4817-8369-620c5a0c65ff-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/354c6c7c-df51-4817-8369-620c5a0c65ff-routing-overlay-volume\") pod \"354c6c7c-df51-4817-8369-620c5a0c65ff\" (UID: \"354c6c7c-df51-4817-8369-620c5a0c65ff\") "
Oct 30 18:50:38.857007 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:38.856867    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/354c6c7c-df51-4817-8369-620c5a0c65ff-kube-api-access-wvxtb\" (UniqueName: \"kubernetes.io/projected/354c6c7c-df51-4817-8369-620c5a0c65ff-kube-api-access-wvxtb\") pod \"354c6c7c-df51-4817-8369-620c5a0c65ff\" (UID: \"354c6c7c-df51-4817-8369-620c5a0c65ff\") "
Oct 30 18:50:38.857007 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:38.856935    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/354c6c7c-df51-4817-8369-620c5a0c65ff-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/354c6c7c-df51-4817-8369-620c5a0c65ff-plugins-config-volume\") pod \"354c6c7c-df51-4817-8369-620c5a0c65ff\" (UID: \"354c6c7c-df51-4817-8369-620c5a0c65ff\") "
Oct 30 18:50:38.857007 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:38.856995    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/354c6c7c-df51-4817-8369-620c5a0c65ff-provider-credentials\" (UniqueName: \"kubernetes.io/projected/354c6c7c-df51-4817-8369-620c5a0c65ff-provider-credentials\") pod \"354c6c7c-df51-4817-8369-620c5a0c65ff\" (UID: \"354c6c7c-df51-4817-8369-620c5a0c65ff\") "
Oct 30 18:50:38.857334 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:38.857306    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/354c6c7c-df51-4817-8369-620c5a0c65ff-routing-overlay-volume" pod "354c6c7c-df51-4817-8369-620c5a0c65ff" (UID: "354c6c7c-df51-4817-8369-620c5a0c65ff"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:50:38.857506 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:38.857371    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/354c6c7c-df51-4817-8369-620c5a0c65ff-plugins-config-volume" pod "354c6c7c-df51-4817-8369-620c5a0c65ff" (UID: "354c6c7c-df51-4817-8369-620c5a0c65ff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:50:38.859436 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:38.859409    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/354c6c7c-df51-4817-8369-620c5a0c65ff-kube-api-access-wvxtb" pod "354c6c7c-df51-4817-8369-620c5a0c65ff" (UID: "354c6c7c-df51-4817-8369-620c5a0c65ff"). InnerVolumeSpecName "kube-api-access-wvxtb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:50:38.859712 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:38.859669    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/354c6c7c-df51-4817-8369-620c5a0c65ff-provider-credentials" pod "354c6c7c-df51-4817-8369-620c5a0c65ff" (UID: "354c6c7c-df51-4817-8369-620c5a0c65ff"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:50:38.958295 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:38.958255    4525 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/354c6c7c-df51-4817-8369-620c5a0c65ff-plugins-config-volume\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:38.958295 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:38.958295    4525 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/354c6c7c-df51-4817-8369-620c5a0c65ff-provider-credentials\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:38.958533 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:38.958307    4525 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/354c6c7c-df51-4817-8369-620c5a0c65ff-routing-overlay-volume\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:38.958533 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:38.958323    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvxtb\" (UniqueName: \"kubernetes.io/projected/354c6c7c-df51-4817-8369-620c5a0c65ff-kube-api-access-wvxtb\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:39.617535 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:39.617501    4525 generic.go:356] "Generic (PLEG): container finished" podID="354c6c7c-df51-4817-8369-620c5a0c65ff" containerID="03bc274bcd24c7a02548a47f3d1397e3cf48077b13b5b628029a635f5a43a1ff" exitCode=0
Oct 30 18:50:39.617736 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:39.617622    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5c6fc54974-gq9dn"
Oct 30 18:50:39.617736 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:39.617641    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5c6fc54974-gq9dn" event={"ID":"354c6c7c-df51-4817-8369-620c5a0c65ff","Type":"ContainerDied","Data":"03bc274bcd24c7a02548a47f3d1397e3cf48077b13b5b628029a635f5a43a1ff"}
Oct 30 18:50:39.617736 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:39.617714    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5c6fc54974-gq9dn" event={"ID":"354c6c7c-df51-4817-8369-620c5a0c65ff","Type":"ContainerDied","Data":"29cd273d922d8ecbe0f9c7f01cf00e6b2d347400e8dc3fb30e61ddc5513347f8"}
Oct 30 18:50:39.617864 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:39.617752    4525 scope.go:122] "RemoveContainer" containerID="03bc274bcd24c7a02548a47f3d1397e3cf48077b13b5b628029a635f5a43a1ff"
Oct 30 18:50:39.627878 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:39.627858    4525 scope.go:122] "RemoveContainer" containerID="03bc274bcd24c7a02548a47f3d1397e3cf48077b13b5b628029a635f5a43a1ff"
Oct 30 18:50:39.628186 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:50:39.628165    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03bc274bcd24c7a02548a47f3d1397e3cf48077b13b5b628029a635f5a43a1ff\": container with ID starting with 03bc274bcd24c7a02548a47f3d1397e3cf48077b13b5b628029a635f5a43a1ff not found: ID does not exist" containerID="03bc274bcd24c7a02548a47f3d1397e3cf48077b13b5b628029a635f5a43a1ff"
Oct 30 18:50:39.628236 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:39.628196    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03bc274bcd24c7a02548a47f3d1397e3cf48077b13b5b628029a635f5a43a1ff"} err="failed to get container status \"03bc274bcd24c7a02548a47f3d1397e3cf48077b13b5b628029a635f5a43a1ff\": rpc error: code = NotFound desc = could not find container \"03bc274bcd24c7a02548a47f3d1397e3cf48077b13b5b628029a635f5a43a1ff\": container with ID starting with 03bc274bcd24c7a02548a47f3d1397e3cf48077b13b5b628029a635f5a43a1ff not found: ID does not exist"
Oct 30 18:50:39.638654 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:39.638629    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5c6fc54974-gq9dn"]
Oct 30 18:50:39.646381 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:39.646356    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5c6fc54974-gq9dn"]
Oct 30 18:50:41.546210 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:41.546180    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="354c6c7c-df51-4817-8369-620c5a0c65ff" path="/var/lib/kubelet/pods/354c6c7c-df51-4817-8369-620c5a0c65ff/volumes"
Oct 30 18:50:42.123989 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:42.123953    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-h74bt"]
Oct 30 18:50:42.124754 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:42.124725    4525 cpu_manager.go:414] "containerMap: removing container" podUID="354c6c7c-df51-4817-8369-620c5a0c65ff" containerName="payload-processing"
Oct 30 18:50:42.124754 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:42.124752    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="354c6c7c-df51-4817-8369-620c5a0c65ff" containerName="payload-processing"
Oct 30 18:50:42.124940 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:42.124883    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="354c6c7c-df51-4817-8369-620c5a0c65ff" containerName="payload-processing"
Oct 30 18:50:42.129669 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:42.129647    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-h74bt"
Oct 30 18:50:42.141408 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:42.141073    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-h74bt"]
Oct 30 18:50:42.148273 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:42.148243    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-k7bpd"]
Oct 30 18:50:42.154143 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:42.154116    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-k7bpd"
Oct 30 18:50:42.167182 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:42.166939    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-k7bpd"]
Oct 30 18:50:42.221773 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:42.221733    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/050bdbd2-11b3-41bc-92e1-e816bb463c9b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-h74bt\" (UID: \"050bdbd2-11b3-41bc-92e1-e816bb463c9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-h74bt"
Oct 30 18:50:42.221971 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:42.221800    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/050bdbd2-11b3-41bc-92e1-e816bb463c9b-istio-envoy\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-h74bt\" (UID: \"050bdbd2-11b3-41bc-92e1-e816bb463c9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-h74bt"
Oct 30 18:50:42.221971 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:42.221892    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/050bdbd2-11b3-41bc-92e1-e816bb463c9b-workload-certs\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-h74bt\" (UID: \"050bdbd2-11b3-41bc-92e1-e816bb463c9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-h74bt"
Oct 30 18:50:42.221971 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:42.221963    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a57dc1f-182b-4a20-9ad7-bb38b9552e4d-istio-data\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-k7bpd\" (UID: \"7a57dc1f-182b-4a20-9ad7-bb38b9552e4d\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-k7bpd"
Oct 30 18:50:42.222121 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:42.222020    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a57dc1f-182b-4a20-9ad7-bb38b9552e4d-workload-socket\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-k7bpd\" (UID: \"7a57dc1f-182b-4a20-9ad7-bb38b9552e4d\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-k7bpd"
Oct 30 18:50:42.222121 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:42.222096    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/050bdbd2-11b3-41bc-92e1-e816bb463c9b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-h74bt\" (UID: \"050bdbd2-11b3-41bc-92e1-e816bb463c9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-h74bt"
Oct 30 18:50:42.222214 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:42.222148    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a57dc1f-182b-4a20-9ad7-bb38b9552e4d-istio-envoy\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-k7bpd\" (UID: \"7a57dc1f-182b-4a20-9ad7-bb38b9552e4d\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-k7bpd"
Oct 30 18:50:42.222214 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:42.222199    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4d8p\" (UniqueName: \"kubernetes.io/projected/050bdbd2-11b3-41bc-92e1-e816bb463c9b-kube-api-access-m4d8p\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-h74bt\" (UID: \"050bdbd2-11b3-41bc-92e1-e816bb463c9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-h74bt"
Oct 30 18:50:42.222304 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:42.222223    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/050bdbd2-11b3-41bc-92e1-e816bb463c9b-istio-token\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-h74bt\" (UID: \"050bdbd2-11b3-41bc-92e1-e816bb463c9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-h74bt"
Oct 30 18:50:42.222345 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:42.222304    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a57dc1f-182b-4a20-9ad7-bb38b9552e4d-istio-podinfo\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-k7bpd\" (UID: \"7a57dc1f-182b-4a20-9ad7-bb38b9552e4d\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-k7bpd"
Oct 30 18:50:42.222396 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:42.222340    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/050bdbd2-11b3-41bc-92e1-e816bb463c9b-credential-socket\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-h74bt\" (UID: \"050bdbd2-11b3-41bc-92e1-e816bb463c9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-h74bt"
Oct 30 18:50:42.222396 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:42.222368    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a57dc1f-182b-4a20-9ad7-bb38b9552e4d-istio-token\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-k7bpd\" (UID: \"7a57dc1f-182b-4a20-9ad7-bb38b9552e4d\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-k7bpd"
Oct 30 18:50:42.222483 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:42.222414    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/050bdbd2-11b3-41bc-92e1-e816bb463c9b-istio-data\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-h74bt\" (UID: \"050bdbd2-11b3-41bc-92e1-e816bb463c9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-h74bt"
Oct 30 18:50:42.222483 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:42.222445    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mszfs\" (UniqueName: \"kubernetes.io/projected/7a57dc1f-182b-4a20-9ad7-bb38b9552e4d-kube-api-access-mszfs\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-k7bpd\" (UID: \"7a57dc1f-182b-4a20-9ad7-bb38b9552e4d\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-k7bpd"
Oct 30 18:50:42.222552 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:42.222483    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a57dc1f-182b-4a20-9ad7-bb38b9552e4d-istiod-ca-cert\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-k7bpd\" (UID: \"7a57dc1f-182b-4a20-9ad7-bb38b9552e4d\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-k7bpd"
Oct 30 18:50:42.222552 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:42.222525    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/050bdbd2-11b3-41bc-92e1-e816bb463c9b-workload-socket\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-h74bt\" (UID: \"050bdbd2-11b3-41bc-92e1-e816bb463c9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-h74bt"
Oct 30 18:50:42.222654 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:42.222581    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a57dc1f-182b-4a20-9ad7-bb38b9552e4d-credential-socket\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-k7bpd\" (UID: \"7a57dc1f-182b-4a20-9ad7-bb38b9552e4d\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-k7bpd"
Oct 30 18:50:42.222654 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:42.222631    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a57dc1f-182b-4a20-9ad7-bb38b9552e4d-workload-certs\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-k7bpd\" (UID: \"7a57dc1f-182b-4a20-9ad7-bb38b9552e4d\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-k7bpd"
Oct 30 18:50:42.324113 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:42.324074    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/050bdbd2-11b3-41bc-92e1-e816bb463c9b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-h74bt\" (UID: \"050bdbd2-11b3-41bc-92e1-e816bb463c9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-h74bt"
Oct 30 18:50:42.324113 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:42.324115    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a57dc1f-182b-4a20-9ad7-bb38b9552e4d-istio-envoy\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-k7bpd\" (UID: \"7a57dc1f-182b-4a20-9ad7-bb38b9552e4d\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-k7bpd"
Oct 30 18:50:42.324372 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:42.324144    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4d8p\" (UniqueName: \"kubernetes.io/projected/050bdbd2-11b3-41bc-92e1-e816bb463c9b-kube-api-access-m4d8p\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-h74bt\" (UID: \"050bdbd2-11b3-41bc-92e1-e816bb463c9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-h74bt"
Oct 30 18:50:42.324372 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:42.324177    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/050bdbd2-11b3-41bc-92e1-e816bb463c9b-istio-token\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-h74bt\" (UID: \"050bdbd2-11b3-41bc-92e1-e816bb463c9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-h74bt"
Oct 30 18:50:42.324372 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:42.324218    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a57dc1f-182b-4a20-9ad7-bb38b9552e4d-istio-podinfo\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-k7bpd\" (UID: \"7a57dc1f-182b-4a20-9ad7-bb38b9552e4d\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-k7bpd"
Oct 30 18:50:42.324372 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:42.324273    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/050bdbd2-11b3-41bc-92e1-e816bb463c9b-credential-socket\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-h74bt\" (UID: \"050bdbd2-11b3-41bc-92e1-e816bb463c9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-h74bt"
Oct 30 18:50:42.324372 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:42.324309    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a57dc1f-182b-4a20-9ad7-bb38b9552e4d-istio-token\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-k7bpd\" (UID: \"7a57dc1f-182b-4a20-9ad7-bb38b9552e4d\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-k7bpd"
Oct 30 18:50:42.324372 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:42.324344    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/050bdbd2-11b3-41bc-92e1-e816bb463c9b-istio-data\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-h74bt\" (UID: \"050bdbd2-11b3-41bc-92e1-e816bb463c9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-h74bt"
Oct 30 18:50:42.324661 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:42.324371    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mszfs\" (UniqueName: \"kubernetes.io/projected/7a57dc1f-182b-4a20-9ad7-bb38b9552e4d-kube-api-access-mszfs\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-k7bpd\" (UID: \"7a57dc1f-182b-4a20-9ad7-bb38b9552e4d\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-k7bpd"
Oct 30 18:50:42.324661 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:42.324402    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a57dc1f-182b-4a20-9ad7-bb38b9552e4d-istiod-ca-cert\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-k7bpd\" (UID: \"7a57dc1f-182b-4a20-9ad7-bb38b9552e4d\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-k7bpd"
Oct 30 18:50:42.324661 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:42.324474    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/050bdbd2-11b3-41bc-92e1-e816bb463c9b-workload-socket\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-h74bt\" (UID: \"050bdbd2-11b3-41bc-92e1-e816bb463c9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-h74bt"
Oct 30 18:50:42.324661 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:42.324537    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a57dc1f-182b-4a20-9ad7-bb38b9552e4d-credential-socket\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-k7bpd\" (UID: \"7a57dc1f-182b-4a20-9ad7-bb38b9552e4d\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-k7bpd"
Oct 30 18:50:42.324661 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:42.324587    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a57dc1f-182b-4a20-9ad7-bb38b9552e4d-workload-certs\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-k7bpd\" (UID: \"7a57dc1f-182b-4a20-9ad7-bb38b9552e4d\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-k7bpd"
Oct 30 18:50:42.324661 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:42.324633    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/050bdbd2-11b3-41bc-92e1-e816bb463c9b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-h74bt\" (UID: \"050bdbd2-11b3-41bc-92e1-e816bb463c9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-h74bt"
Oct 30 18:50:42.325014 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:42.324702    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/050bdbd2-11b3-41bc-92e1-e816bb463c9b-istio-envoy\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-h74bt\" (UID: \"050bdbd2-11b3-41bc-92e1-e816bb463c9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-h74bt"
Oct 30 18:50:42.325014 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:42.324732    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/050bdbd2-11b3-41bc-92e1-e816bb463c9b-workload-certs\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-h74bt\" (UID: \"050bdbd2-11b3-41bc-92e1-e816bb463c9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-h74bt"
Oct 30 18:50:42.325014 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:42.324763    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a57dc1f-182b-4a20-9ad7-bb38b9552e4d-istio-data\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-k7bpd\" (UID: \"7a57dc1f-182b-4a20-9ad7-bb38b9552e4d\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-k7bpd"
Oct 30 18:50:42.325014 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:42.324824    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a57dc1f-182b-4a20-9ad7-bb38b9552e4d-workload-socket\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-k7bpd\" (UID: \"7a57dc1f-182b-4a20-9ad7-bb38b9552e4d\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-k7bpd"
Oct 30 18:50:42.325014 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:42.324957    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/050bdbd2-11b3-41bc-92e1-e816bb463c9b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-h74bt\" (UID: \"050bdbd2-11b3-41bc-92e1-e816bb463c9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-h74bt"
Oct 30 18:50:42.325282 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:42.325140    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a57dc1f-182b-4a20-9ad7-bb38b9552e4d-credential-socket\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-k7bpd\" (UID: \"7a57dc1f-182b-4a20-9ad7-bb38b9552e4d\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-k7bpd"
Oct 30 18:50:42.325282 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:42.325194    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a57dc1f-182b-4a20-9ad7-bb38b9552e4d-workload-socket\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-k7bpd\" (UID: \"7a57dc1f-182b-4a20-9ad7-bb38b9552e4d\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-k7bpd"
Oct 30 18:50:42.325460 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:42.325414    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/050bdbd2-11b3-41bc-92e1-e816bb463c9b-workload-certs\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-h74bt\" (UID: \"050bdbd2-11b3-41bc-92e1-e816bb463c9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-h74bt"
Oct 30 18:50:42.325587 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:42.325514    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a57dc1f-182b-4a20-9ad7-bb38b9552e4d-workload-certs\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-k7bpd\" (UID: \"7a57dc1f-182b-4a20-9ad7-bb38b9552e4d\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-k7bpd"
Oct 30 18:50:42.325587 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:42.325538    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/050bdbd2-11b3-41bc-92e1-e816bb463c9b-workload-socket\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-h74bt\" (UID: \"050bdbd2-11b3-41bc-92e1-e816bb463c9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-h74bt"
Oct 30 18:50:42.325760 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:42.325667    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a57dc1f-182b-4a20-9ad7-bb38b9552e4d-istiod-ca-cert\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-k7bpd\" (UID: \"7a57dc1f-182b-4a20-9ad7-bb38b9552e4d\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-k7bpd"
Oct 30 18:50:42.325877 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:42.325852    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a57dc1f-182b-4a20-9ad7-bb38b9552e4d-istio-data\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-k7bpd\" (UID: \"7a57dc1f-182b-4a20-9ad7-bb38b9552e4d\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-k7bpd"
Oct 30 18:50:42.325964 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:42.325923    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/050bdbd2-11b3-41bc-92e1-e816bb463c9b-credential-socket\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-h74bt\" (UID: \"050bdbd2-11b3-41bc-92e1-e816bb463c9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-h74bt"
Oct 30 18:50:42.326018 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:42.325998    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/050bdbd2-11b3-41bc-92e1-e816bb463c9b-istio-data\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-h74bt\" (UID: \"050bdbd2-11b3-41bc-92e1-e816bb463c9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-h74bt"
Oct 30 18:50:42.327214 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:42.327178    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a57dc1f-182b-4a20-9ad7-bb38b9552e4d-istio-podinfo\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-k7bpd\" (UID: \"7a57dc1f-182b-4a20-9ad7-bb38b9552e4d\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-k7bpd"
Oct 30 18:50:42.327318 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:42.327274    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a57dc1f-182b-4a20-9ad7-bb38b9552e4d-istio-envoy\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-k7bpd\" (UID: \"7a57dc1f-182b-4a20-9ad7-bb38b9552e4d\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-k7bpd"
Oct 30 18:50:42.327585 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:42.327565    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/050bdbd2-11b3-41bc-92e1-e816bb463c9b-istio-envoy\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-h74bt\" (UID: \"050bdbd2-11b3-41bc-92e1-e816bb463c9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-h74bt"
Oct 30 18:50:42.328131 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:42.328109    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/050bdbd2-11b3-41bc-92e1-e816bb463c9b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-h74bt\" (UID: \"050bdbd2-11b3-41bc-92e1-e816bb463c9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-h74bt"
Oct 30 18:50:42.332556 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:42.332489    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/050bdbd2-11b3-41bc-92e1-e816bb463c9b-istio-token\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-h74bt\" (UID: \"050bdbd2-11b3-41bc-92e1-e816bb463c9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-h74bt"
Oct 30 18:50:42.334170 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:42.334146    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4d8p\" (UniqueName: \"kubernetes.io/projected/050bdbd2-11b3-41bc-92e1-e816bb463c9b-kube-api-access-m4d8p\") pod \"maas-default-gateway-openshift-default-68cb9b4b95-h74bt\" (UID: \"050bdbd2-11b3-41bc-92e1-e816bb463c9b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-h74bt"
Oct 30 18:50:42.334810 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:42.334789    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a57dc1f-182b-4a20-9ad7-bb38b9552e4d-istio-token\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-k7bpd\" (UID: \"7a57dc1f-182b-4a20-9ad7-bb38b9552e4d\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-k7bpd"
Oct 30 18:50:42.334906 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:42.334892    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mszfs\" (UniqueName: \"kubernetes.io/projected/7a57dc1f-182b-4a20-9ad7-bb38b9552e4d-kube-api-access-mszfs\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-k7bpd\" (UID: \"7a57dc1f-182b-4a20-9ad7-bb38b9552e4d\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-k7bpd"
Oct 30 18:50:42.449844 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:42.449754    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-h74bt"
Oct 30 18:50:42.474707 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:42.474623    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-k7bpd"
Oct 30 18:50:42.754612 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:42.754581    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-h74bt"]
Oct 30 18:50:42.755896 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:42.755867    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:50:42.755985 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:42.755928    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:50:42.755985 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:42.755979    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:50:42.775372 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:50:42.775341    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a57dc1f_182b_4a20_9ad7_bb38b9552e4d.slice/crio-e2a62632942103e867e6e00131f5048dc55f4dda8d7202beb316a15a9a7b0348 WatchSource:0}: Error finding container e2a62632942103e867e6e00131f5048dc55f4dda8d7202beb316a15a9a7b0348: Status 404 returned error can't find the container with id e2a62632942103e867e6e00131f5048dc55f4dda8d7202beb316a15a9a7b0348
Oct 30 18:50:42.776082 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:42.776061    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-k7bpd"]
Oct 30 18:50:42.778465 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:42.778436    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:50:42.778537 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:42.778500    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:50:42.778537 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:42.778528    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:50:43.670928 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:43.670895    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-k7bpd" event={"ID":"7a57dc1f-182b-4a20-9ad7-bb38b9552e4d","Type":"ContainerStarted","Data":"b61cb9e0d01a0e8a8b1f03182ddce95b4b5ea018efd5df3974ffad103dcf5ddb"}
Oct 30 18:50:43.671111 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:43.670956    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-k7bpd" event={"ID":"7a57dc1f-182b-4a20-9ad7-bb38b9552e4d","Type":"ContainerStarted","Data":"e2a62632942103e867e6e00131f5048dc55f4dda8d7202beb316a15a9a7b0348"}
Oct 30 18:50:43.672608 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:43.672583    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-h74bt" event={"ID":"050bdbd2-11b3-41bc-92e1-e816bb463c9b","Type":"ContainerStarted","Data":"2266671aecaa481be24266f033803e4b8ad1b41e5f3d9cc884fe123c7ccad05e"}
Oct 30 18:50:43.672761 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:43.672624    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-h74bt" event={"ID":"050bdbd2-11b3-41bc-92e1-e816bb463c9b","Type":"ContainerStarted","Data":"079d21fe64a6a2a209495724ea1322285a7b378be75da4ca44e7fb0d2da6d0a1"}
Oct 30 18:50:43.730463 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:43.730402    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-k7bpd" podStartSLOduration=1.730382496 podStartE2EDuration="1.730382496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:50:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:50:43.707207404 +0000 UTC m=+866.332877369" watchObservedRunningTime="2026-10-30 18:50:43.730382496 +0000 UTC m=+866.356052461"
Oct 30 18:50:43.753031 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:43.752971    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-h74bt" podStartSLOduration=1.7529313119999999 podStartE2EDuration="1.752931312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:50:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:50:43.751540565 +0000 UTC m=+866.377210529" watchObservedRunningTime="2026-10-30 18:50:43.752931312 +0000 UTC m=+866.378601277"
Oct 30 18:50:44.450720 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:44.450691    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-h74bt"
Oct 30 18:50:44.451745 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:44.451723    4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-68cb9b4b95-h74bt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 18:50:44.451821 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:44.451769    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-h74bt" podUID="050bdbd2-11b3-41bc-92e1-e816bb463c9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 18:50:44.499707 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:44.475671    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-k7bpd"
Oct 30 18:50:44.524784 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:44.521814    4525 patch_prober.go:29] interesting pod/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-k7bpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 18:50:44.524784 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:44.521881    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-k7bpd" podUID="7a57dc1f-182b-4a20-9ad7-bb38b9552e4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 18:50:44.597706 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:44.577453    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-bc6ff8476-tfpc7"]
Oct 30 18:50:44.628709 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:44.626415    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-bc6ff8476-tfpc7"
Oct 30 18:50:44.632705 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:44.630581    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-bc6ff8476-tfpc7"]
Oct 30 18:50:44.635715 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:44.633436    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-fb196975096e\""
Oct 30 18:50:44.635715 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:44.633502    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-4ed9bb7efb08\""
Oct 30 18:50:44.660949 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:44.660916    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2svdg\" (UniqueName: \"kubernetes.io/projected/1da1c465-0d66-484c-a6de-393667a61a86-kube-api-access-2svdg\") pod \"payload-processing-external-model-bc6ff8476-tfpc7\" (UID: \"1da1c465-0d66-484c-a6de-393667a61a86\") " pod="models-as-a-service/payload-processing-external-model-bc6ff8476-tfpc7"
Oct 30 18:50:44.661098 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:44.661037    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1da1c465-0d66-484c-a6de-393667a61a86-routing-overlay-volume\") pod \"payload-processing-external-model-bc6ff8476-tfpc7\" (UID: \"1da1c465-0d66-484c-a6de-393667a61a86\") " pod="models-as-a-service/payload-processing-external-model-bc6ff8476-tfpc7"
Oct 30 18:50:44.661098 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:44.661081    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1da1c465-0d66-484c-a6de-393667a61a86-plugins-config-volume\") pod \"payload-processing-external-model-bc6ff8476-tfpc7\" (UID: \"1da1c465-0d66-484c-a6de-393667a61a86\") " pod="models-as-a-service/payload-processing-external-model-bc6ff8476-tfpc7"
Oct 30 18:50:44.661287 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:44.661269    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1da1c465-0d66-484c-a6de-393667a61a86-provider-credentials\") pod \"payload-processing-external-model-bc6ff8476-tfpc7\" (UID: \"1da1c465-0d66-484c-a6de-393667a61a86\") " pod="models-as-a-service/payload-processing-external-model-bc6ff8476-tfpc7"
Oct 30 18:50:44.762605 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:44.762313    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1da1c465-0d66-484c-a6de-393667a61a86-routing-overlay-volume\") pod \"payload-processing-external-model-bc6ff8476-tfpc7\" (UID: \"1da1c465-0d66-484c-a6de-393667a61a86\") " pod="models-as-a-service/payload-processing-external-model-bc6ff8476-tfpc7"
Oct 30 18:50:44.762605 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:44.762363    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1da1c465-0d66-484c-a6de-393667a61a86-plugins-config-volume\") pod \"payload-processing-external-model-bc6ff8476-tfpc7\" (UID: \"1da1c465-0d66-484c-a6de-393667a61a86\") " pod="models-as-a-service/payload-processing-external-model-bc6ff8476-tfpc7"
Oct 30 18:50:44.762841 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:44.762663    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1da1c465-0d66-484c-a6de-393667a61a86-provider-credentials\") pod \"payload-processing-external-model-bc6ff8476-tfpc7\" (UID: \"1da1c465-0d66-484c-a6de-393667a61a86\") " pod="models-as-a-service/payload-processing-external-model-bc6ff8476-tfpc7"
Oct 30 18:50:44.762841 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:44.762827    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2svdg\" (UniqueName: \"kubernetes.io/projected/1da1c465-0d66-484c-a6de-393667a61a86-kube-api-access-2svdg\") pod \"payload-processing-external-model-bc6ff8476-tfpc7\" (UID: \"1da1c465-0d66-484c-a6de-393667a61a86\") " pod="models-as-a-service/payload-processing-external-model-bc6ff8476-tfpc7"
Oct 30 18:50:44.763868 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:44.763837    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1da1c465-0d66-484c-a6de-393667a61a86-routing-overlay-volume\") pod \"payload-processing-external-model-bc6ff8476-tfpc7\" (UID: \"1da1c465-0d66-484c-a6de-393667a61a86\") " pod="models-as-a-service/payload-processing-external-model-bc6ff8476-tfpc7"
Oct 30 18:50:44.764151 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:44.764096    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1da1c465-0d66-484c-a6de-393667a61a86-plugins-config-volume\") pod \"payload-processing-external-model-bc6ff8476-tfpc7\" (UID: \"1da1c465-0d66-484c-a6de-393667a61a86\") " pod="models-as-a-service/payload-processing-external-model-bc6ff8476-tfpc7"
Oct 30 18:50:44.767474 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:44.767450    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1da1c465-0d66-484c-a6de-393667a61a86-provider-credentials\") pod \"payload-processing-external-model-bc6ff8476-tfpc7\" (UID: \"1da1c465-0d66-484c-a6de-393667a61a86\") " pod="models-as-a-service/payload-processing-external-model-bc6ff8476-tfpc7"
Oct 30 18:50:44.770810 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:44.770789    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2svdg\" (UniqueName: \"kubernetes.io/projected/1da1c465-0d66-484c-a6de-393667a61a86-kube-api-access-2svdg\") pod \"payload-processing-external-model-bc6ff8476-tfpc7\" (UID: \"1da1c465-0d66-484c-a6de-393667a61a86\") " pod="models-as-a-service/payload-processing-external-model-bc6ff8476-tfpc7"
Oct 30 18:50:44.956262 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:44.956231    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-bc6ff8476-tfpc7"
Oct 30 18:50:45.331469 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:45.331434    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-bc6ff8476-tfpc7"]
Oct 30 18:50:45.450915 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:45.450883    4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-68cb9b4b95-h74bt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 18:50:45.451372 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:45.450943    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-h74bt" podUID="050bdbd2-11b3-41bc-92e1-e816bb463c9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 18:50:45.479720 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:45.475912    4525 patch_prober.go:29] interesting pod/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-k7bpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 18:50:45.479720 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:45.475967    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-k7bpd" podUID="7a57dc1f-182b-4a20-9ad7-bb38b9552e4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 18:50:45.738305 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:45.737731    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-bc6ff8476-tfpc7" event={"ID":"1da1c465-0d66-484c-a6de-393667a61a86","Type":"ContainerStarted","Data":"f7c6bcb80dc8f9bd0242a227b294b8e9306d02698646af27293dde8b2e86b656"}
Oct 30 18:50:45.738305 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:45.737782    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-bc6ff8476-tfpc7" event={"ID":"1da1c465-0d66-484c-a6de-393667a61a86","Type":"ContainerStarted","Data":"6108819b5c440dc0591bdb2ab3b1fa0ee6d4284615f880984b20c70f635b925b"}
Oct 30 18:50:45.738305 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:45.738003    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-bc6ff8476-tfpc7"
Oct 30 18:50:45.761531 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:45.761486    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-bc6ff8476-tfpc7" podStartSLOduration=1.761473152 podStartE2EDuration="1.761473152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:50:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:50:45.75835803 +0000 UTC m=+868.384027995" watchObservedRunningTime="2026-10-30 18:50:45.761473152 +0000 UTC m=+868.387143116"
Oct 30 18:50:45.935274 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:45.934436    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-39bde3/e2e-conflict-b-8eedd2a0-kserve-84fcfc5594-x4szz"]
Oct 30 18:50:46.450264 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:46.450231    4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-68cb9b4b95-h74bt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 18:50:46.450451 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:46.450279    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-h74bt" podUID="050bdbd2-11b3-41bc-92e1-e816bb463c9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 18:50:46.474990 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:46.474962    4525 patch_prober.go:29] interesting pod/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-k7bpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 18:50:46.475344 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:46.475002    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-k7bpd" podUID="7a57dc1f-182b-4a20-9ad7-bb38b9552e4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 18:50:47.120706 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:47.120011    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj"
Oct 30 18:50:47.151778 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:47.151748    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj"
Oct 30 18:50:47.449461 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:50:47.449385    4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d49l" containerName="extractor"
Oct 30 18:50:47.450827 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:47.450804    4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-68cb9b4b95-h74bt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 18:50:47.450887 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:47.450848    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-h74bt" podUID="050bdbd2-11b3-41bc-92e1-e816bb463c9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 18:50:47.474932 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:47.474905    4525 patch_prober.go:29] interesting pod/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-k7bpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 18:50:47.475039 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:47.474941    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-k7bpd" podUID="7a57dc1f-182b-4a20-9ad7-bb38b9552e4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 18:50:47.764176 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:47.764097    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj"
Oct 30 18:50:48.450731 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:48.450663    4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-68cb9b4b95-h74bt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 18:50:48.450942 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:48.450747    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-h74bt" podUID="050bdbd2-11b3-41bc-92e1-e816bb463c9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 18:50:48.475055 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:48.475010    4525 patch_prober.go:29] interesting pod/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-k7bpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 18:50:48.475528 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:48.475490    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-k7bpd" podUID="7a57dc1f-182b-4a20-9ad7-bb38b9552e4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 18:50:49.453290 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:49.451225    4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-68cb9b4b95-h74bt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 18:50:49.453290 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:49.451307    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-h74bt" podUID="050bdbd2-11b3-41bc-92e1-e816bb463c9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 18:50:49.475138 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:49.475099    4525 patch_prober.go:29] interesting pod/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-k7bpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 18:50:49.475506 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:49.475172    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-k7bpd" podUID="7a57dc1f-182b-4a20-9ad7-bb38b9552e4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 18:50:50.453428 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:50.453387    4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-68cb9b4b95-h74bt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 18:50:50.453792 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:50.453761    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-h74bt" podUID="050bdbd2-11b3-41bc-92e1-e816bb463c9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 18:50:50.896624 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:50.895019    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-k7bpd"
Oct 30 18:50:50.896624 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:50.896465    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-k7bpd"
Oct 30 18:50:50.898948 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:50.898888    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-k7bpd"
Oct 30 18:50:52.503707 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:52.502846    4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-68cb9b4b95-h74bt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 18:50:52.503707 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:52.502932    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-h74bt" podUID="050bdbd2-11b3-41bc-92e1-e816bb463c9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:50:52.674807 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:52.674758    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb" podUID="12ebc8b9-c794-47b5-b87b-a63c4bd993e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:50:52.766897 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:52.766792    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr" podUID="82bbbf85-ed93-48d2-9f17-49ac1f31c0a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:50:52.953316 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:52.953274    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-h74bt"
Oct 30 18:50:52.954750 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:52.954729    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-h74bt"
Oct 30 18:50:52.958954 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:52.958937    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-h74bt"
Oct 30 18:50:53.210707 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:53.210640    4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-68cb9b4b95-tckhm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 18:50:53.211029 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:53.210998    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-tckhm" podUID="23ad1447-7842-4b64-b99f-adf61a803291" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:50:56.748931 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:56.748904    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-bc6ff8476-tfpc7"
Oct 30 18:50:56.806699 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:50:56.806654    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6bc5d5dd85-4mgzj"]
Oct 30 18:51:00.958503 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:00.958465    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-39bde3/e2e-conflict-b-8eedd2a0-kserve-84fcfc5594-x4szz" podUID="5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a" containerName="main" containerID="cri-o://b7894f7423b5a1f4d8133c0ed7694e8d30d07a11d9718a673d625821b1b04709" gracePeriod=45
Oct 30 18:51:01.225372 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:01.225311    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-39bde3/e2e-conflict-b-8eedd2a0-kserve-84fcfc5594-x4szz"
Oct 30 18:51:01.413360 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:01.413318    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a-kube-api-access-rdc5p\" (UniqueName: \"kubernetes.io/projected/5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a-kube-api-access-rdc5p\") pod \"5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a\" (UID: \"5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a\") "
Oct 30 18:51:01.413538 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:01.413400    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a-tmp-dir\") pod \"5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a\" (UID: \"5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a\") "
Oct 30 18:51:01.413538 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:01.413483    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a-home\" (UniqueName: \"kubernetes.io/empty-dir/5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a-home\") pod \"5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a\" (UID: \"5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a\") "
Oct 30 18:51:01.413612 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:01.413598    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a-tls-certs\" (UniqueName: \"kubernetes.io/secret/5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a-tls-certs\") pod \"5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a\" (UID: \"5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a\") "
Oct 30 18:51:01.413732 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:01.413715    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a-dshm\") pod \"5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a\" (UID: \"5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a\") "
Oct 30 18:51:01.413795 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:01.413718    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a-tmp-dir" pod "5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a" (UID: "5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:51:01.413795 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:01.413744    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a-home" pod "5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a" (UID: "5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:51:01.413795 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:01.413749    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a-model-cache\") pod \"5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a\" (UID: \"5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a\") "
Oct 30 18:51:01.413948 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:01.413929    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a-model-cache" pod "5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a" (UID: "5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:51:01.414205 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:01.414184    4525 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a-home\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:01.414205 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:01.414206    4525 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a-model-cache\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:01.414375 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:01.414219    4525 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a-tmp-dir\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:01.415879 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:01.415855    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a-kube-api-access-rdc5p" pod "5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a" (UID: "5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a"). InnerVolumeSpecName "kube-api-access-rdc5p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:51:01.415967 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:01.415952    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a-tls-certs" pod "5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a" (UID: "5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:51:01.416028 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:01.416009    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a-dshm" pod "5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a" (UID: "5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:51:01.515395 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:01.515310    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdc5p\" (UniqueName: \"kubernetes.io/projected/5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a-kube-api-access-rdc5p\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:01.515395 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:01.515341    4525 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a-tls-certs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:01.515395 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:01.515351    4525 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a-dshm\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:01.807438 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:01.807341    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6bc5d5dd85-4mgzj" podUID="510b0958-2710-413f-8da2-410728868b76" containerName="payload-processing" containerID="cri-o://db7e7cd7d9b18d8acbb8a838942dc807cb70d0c7ccf08e68f2f2292a4af4bdcc" gracePeriod=25
Oct 30 18:51:02.071280 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:02.071206    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6bc5d5dd85-4mgzj"
Oct 30 18:51:02.071652 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:02.071394    4525 generic.go:356] "Generic (PLEG): container finished" podID="5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a" containerID="b7894f7423b5a1f4d8133c0ed7694e8d30d07a11d9718a673d625821b1b04709" exitCode=0
Oct 30 18:51:02.071652 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:02.071457    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-39bde3/e2e-conflict-b-8eedd2a0-kserve-84fcfc5594-x4szz"
Oct 30 18:51:02.071652 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:02.071479    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-39bde3/e2e-conflict-b-8eedd2a0-kserve-84fcfc5594-x4szz" event={"ID":"5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a","Type":"ContainerDied","Data":"b7894f7423b5a1f4d8133c0ed7694e8d30d07a11d9718a673d625821b1b04709"}
Oct 30 18:51:02.071652 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:02.071520    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-39bde3/e2e-conflict-b-8eedd2a0-kserve-84fcfc5594-x4szz" event={"ID":"5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a","Type":"ContainerDied","Data":"318412c9b2f4629e4718fdf8c3f013650ceb1d5378580ecd0bcb5bfea963efa1"}
Oct 30 18:51:02.071652 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:02.071537    4525 scope.go:122] "RemoveContainer" containerID="b7894f7423b5a1f4d8133c0ed7694e8d30d07a11d9718a673d625821b1b04709"
Oct 30 18:51:02.073287 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:02.073260    4525 generic.go:356] "Generic (PLEG): container finished" podID="510b0958-2710-413f-8da2-410728868b76" containerID="db7e7cd7d9b18d8acbb8a838942dc807cb70d0c7ccf08e68f2f2292a4af4bdcc" exitCode=0
Oct 30 18:51:02.073407 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:02.073306    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6bc5d5dd85-4mgzj"
Oct 30 18:51:02.073465 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:02.073309    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6bc5d5dd85-4mgzj" event={"ID":"510b0958-2710-413f-8da2-410728868b76","Type":"ContainerDied","Data":"db7e7cd7d9b18d8acbb8a838942dc807cb70d0c7ccf08e68f2f2292a4af4bdcc"}
Oct 30 18:51:02.073513 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:02.073462    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6bc5d5dd85-4mgzj" event={"ID":"510b0958-2710-413f-8da2-410728868b76","Type":"ContainerDied","Data":"98f8d1930ed61c30bfe3ba8ac561c4b1a9b0d59b5b2a43743e93c0d0eb624db6"}
Oct 30 18:51:02.081299 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:02.081277    4525 scope.go:122] "RemoveContainer" containerID="b7894f7423b5a1f4d8133c0ed7694e8d30d07a11d9718a673d625821b1b04709"
Oct 30 18:51:02.081551 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:51:02.081534    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7894f7423b5a1f4d8133c0ed7694e8d30d07a11d9718a673d625821b1b04709\": container with ID starting with b7894f7423b5a1f4d8133c0ed7694e8d30d07a11d9718a673d625821b1b04709 not found: ID does not exist" containerID="b7894f7423b5a1f4d8133c0ed7694e8d30d07a11d9718a673d625821b1b04709"
Oct 30 18:51:02.081615 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:02.081560    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7894f7423b5a1f4d8133c0ed7694e8d30d07a11d9718a673d625821b1b04709"} err="failed to get container status \"b7894f7423b5a1f4d8133c0ed7694e8d30d07a11d9718a673d625821b1b04709\": rpc error: code = NotFound desc = could not find container \"b7894f7423b5a1f4d8133c0ed7694e8d30d07a11d9718a673d625821b1b04709\": container with ID starting with b7894f7423b5a1f4d8133c0ed7694e8d30d07a11d9718a673d625821b1b04709 not found: ID does not exist"
Oct 30 18:51:02.081615 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:02.081577    4525 scope.go:122] "RemoveContainer" containerID="db7e7cd7d9b18d8acbb8a838942dc807cb70d0c7ccf08e68f2f2292a4af4bdcc"
Oct 30 18:51:02.090923 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:02.090900    4525 scope.go:122] "RemoveContainer" containerID="db7e7cd7d9b18d8acbb8a838942dc807cb70d0c7ccf08e68f2f2292a4af4bdcc"
Oct 30 18:51:02.092060 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:51:02.091241    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db7e7cd7d9b18d8acbb8a838942dc807cb70d0c7ccf08e68f2f2292a4af4bdcc\": container with ID starting with db7e7cd7d9b18d8acbb8a838942dc807cb70d0c7ccf08e68f2f2292a4af4bdcc not found: ID does not exist" containerID="db7e7cd7d9b18d8acbb8a838942dc807cb70d0c7ccf08e68f2f2292a4af4bdcc"
Oct 30 18:51:02.092060 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:02.091278    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db7e7cd7d9b18d8acbb8a838942dc807cb70d0c7ccf08e68f2f2292a4af4bdcc"} err="failed to get container status \"db7e7cd7d9b18d8acbb8a838942dc807cb70d0c7ccf08e68f2f2292a4af4bdcc\": rpc error: code = NotFound desc = could not find container \"db7e7cd7d9b18d8acbb8a838942dc807cb70d0c7ccf08e68f2f2292a4af4bdcc\": container with ID starting with db7e7cd7d9b18d8acbb8a838942dc807cb70d0c7ccf08e68f2f2292a4af4bdcc not found: ID does not exist"
Oct 30 18:51:02.110972 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:02.110945    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-39bde3/e2e-conflict-b-8eedd2a0-kserve-84fcfc5594-x4szz"]
Oct 30 18:51:02.119580 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:02.119556    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-39bde3/e2e-conflict-b-8eedd2a0-kserve-84fcfc5594-x4szz"]
Oct 30 18:51:02.225348 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:02.225309    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/510b0958-2710-413f-8da2-410728868b76-provider-credentials\" (UniqueName: \"kubernetes.io/projected/510b0958-2710-413f-8da2-410728868b76-provider-credentials\") pod \"510b0958-2710-413f-8da2-410728868b76\" (UID: \"510b0958-2710-413f-8da2-410728868b76\") "
Oct 30 18:51:02.225525 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:02.225363    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/510b0958-2710-413f-8da2-410728868b76-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/510b0958-2710-413f-8da2-410728868b76-routing-overlay-volume\") pod \"510b0958-2710-413f-8da2-410728868b76\" (UID: \"510b0958-2710-413f-8da2-410728868b76\") "
Oct 30 18:51:02.225525 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:02.225464    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/510b0958-2710-413f-8da2-410728868b76-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/510b0958-2710-413f-8da2-410728868b76-plugins-config-volume\") pod \"510b0958-2710-413f-8da2-410728868b76\" (UID: \"510b0958-2710-413f-8da2-410728868b76\") "
Oct 30 18:51:02.225525 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:02.225503    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/510b0958-2710-413f-8da2-410728868b76-kube-api-access-82j8l\" (UniqueName: \"kubernetes.io/projected/510b0958-2710-413f-8da2-410728868b76-kube-api-access-82j8l\") pod \"510b0958-2710-413f-8da2-410728868b76\" (UID: \"510b0958-2710-413f-8da2-410728868b76\") "
Oct 30 18:51:02.225793 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:02.225770    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/510b0958-2710-413f-8da2-410728868b76-routing-overlay-volume" pod "510b0958-2710-413f-8da2-410728868b76" (UID: "510b0958-2710-413f-8da2-410728868b76"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:51:02.225962 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:02.225941    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/510b0958-2710-413f-8da2-410728868b76-plugins-config-volume" pod "510b0958-2710-413f-8da2-410728868b76" (UID: "510b0958-2710-413f-8da2-410728868b76"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:51:02.226138 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:02.226122    4525 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/510b0958-2710-413f-8da2-410728868b76-routing-overlay-volume\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:02.226214 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:02.226144    4525 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/510b0958-2710-413f-8da2-410728868b76-plugins-config-volume\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:02.228736 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:02.228709    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/510b0958-2710-413f-8da2-410728868b76-provider-credentials" pod "510b0958-2710-413f-8da2-410728868b76" (UID: "510b0958-2710-413f-8da2-410728868b76"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:51:02.228884 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:02.228859    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/510b0958-2710-413f-8da2-410728868b76-kube-api-access-82j8l" pod "510b0958-2710-413f-8da2-410728868b76" (UID: "510b0958-2710-413f-8da2-410728868b76"). InnerVolumeSpecName "kube-api-access-82j8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:51:02.327026 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:02.326931    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82j8l\" (UniqueName: \"kubernetes.io/projected/510b0958-2710-413f-8da2-410728868b76-kube-api-access-82j8l\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:02.327026 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:02.326965    4525 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/510b0958-2710-413f-8da2-410728868b76-provider-credentials\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:02.396625 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:02.396589    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6bc5d5dd85-4mgzj"]
Oct 30 18:51:02.399752 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:02.399730    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6bc5d5dd85-4mgzj"]
Oct 30 18:51:03.556030 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:03.555992    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="510b0958-2710-413f-8da2-410728868b76" path="/var/lib/kubelet/pods/510b0958-2710-413f-8da2-410728868b76/volumes"
Oct 30 18:51:03.556436 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:03.556383    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a" path="/var/lib/kubelet/pods/5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a/volumes"
Oct 30 18:51:06.678706 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:06.678339    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw" podUID="ac6852fd-01be-4d69-8c9b-0c591e50b889" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:51:06.787406 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:06.786809    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv" podUID="595230e3-55f3-43eb-9a7b-6d89250af9b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:51:06.883882 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:06.883844    4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-68cb9b4b95-dhclt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 18:51:06.884174 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:06.884123    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-dhclt" podUID="0e0636e7-6f61-4843-844d-fb9d7bb527da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:51:07.685881 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:07.685849    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hncld"]
Oct 30 18:51:07.686353 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:07.686325    4525 cpu_manager.go:414] "containerMap: removing container" podUID="5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a" containerName="main"
Oct 30 18:51:07.686353 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:07.686342    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a" containerName="main"
Oct 30 18:51:07.686427 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:07.686369    4525 cpu_manager.go:414] "containerMap: removing container" podUID="510b0958-2710-413f-8da2-410728868b76" containerName="payload-processing"
Oct 30 18:51:07.686427 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:07.686376    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="510b0958-2710-413f-8da2-410728868b76" containerName="payload-processing"
Oct 30 18:51:07.686488 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:07.686481    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bcd8d9d-5274-43bf-86bf-d0aa7b298a3a" containerName="main"
Oct 30 18:51:07.686526 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:07.686492    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="510b0958-2710-413f-8da2-410728868b76" containerName="payload-processing"
Oct 30 18:51:07.689785 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:07.689766    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hncld"
Oct 30 18:51:07.698953 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:07.698923    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 30 18:51:07.717614 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:07.717582    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hncld"]
Oct 30 18:51:07.736726 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:07.729277    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-gtdr6"]
Oct 30 18:51:07.736726 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:07.734163    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-gtdr6"
Oct 30 18:51:07.747641 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:07.747612    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-gtdr6"]
Oct 30 18:51:07.799403 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:07.799372    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 30 18:51:07.799725 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:07.799709    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 30 18:51:07.800024 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:07.799992    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-dqnb2\""
Oct 30 18:51:07.880992 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:07.880956    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ee5fed99-83c8-46d3-a07c-b1db0da31534-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hncld\" (UID: \"ee5fed99-83c8-46d3-a07c-b1db0da31534\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hncld"
Oct 30 18:51:07.881193 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:07.881056    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee5fed99-83c8-46d3-a07c-b1db0da31534-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hncld\" (UID: \"ee5fed99-83c8-46d3-a07c-b1db0da31534\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hncld"
Oct 30 18:51:07.881193 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:07.881081    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ee5fed99-83c8-46d3-a07c-b1db0da31534-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hncld\" (UID: \"ee5fed99-83c8-46d3-a07c-b1db0da31534\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hncld"
Oct 30 18:51:07.881307 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:07.881188    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01a5c107-dcf4-4ca5-8c9d-57c40814206d-istio-token\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-gtdr6\" (UID: \"01a5c107-dcf4-4ca5-8c9d-57c40814206d\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-gtdr6"
Oct 30 18:51:07.881307 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:07.881222    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ee5fed99-83c8-46d3-a07c-b1db0da31534-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hncld\" (UID: \"ee5fed99-83c8-46d3-a07c-b1db0da31534\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hncld"
Oct 30 18:51:07.881402 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:07.881342    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ee5fed99-83c8-46d3-a07c-b1db0da31534-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hncld\" (UID: \"ee5fed99-83c8-46d3-a07c-b1db0da31534\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hncld"
Oct 30 18:51:07.881402 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:07.881391    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee5fed99-83c8-46d3-a07c-b1db0da31534-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hncld\" (UID: \"ee5fed99-83c8-46d3-a07c-b1db0da31534\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hncld"
Oct 30 18:51:07.881532 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:07.881508    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trs94\" (UniqueName: \"kubernetes.io/projected/01a5c107-dcf4-4ca5-8c9d-57c40814206d-kube-api-access-trs94\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-gtdr6\" (UID: \"01a5c107-dcf4-4ca5-8c9d-57c40814206d\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-gtdr6"
Oct 30 18:51:07.881585 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:07.881557    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a5c107-dcf4-4ca5-8c9d-57c40814206d-workload-socket\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-gtdr6\" (UID: \"01a5c107-dcf4-4ca5-8c9d-57c40814206d\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-gtdr6"
Oct 30 18:51:07.881665 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:07.881646    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a5c107-dcf4-4ca5-8c9d-57c40814206d-credential-socket\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-gtdr6\" (UID: \"01a5c107-dcf4-4ca5-8c9d-57c40814206d\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-gtdr6"
Oct 30 18:51:07.881758 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:07.881733    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01a5c107-dcf4-4ca5-8c9d-57c40814206d-workload-certs\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-gtdr6\" (UID: \"01a5c107-dcf4-4ca5-8c9d-57c40814206d\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-gtdr6"
Oct 30 18:51:07.881912 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:07.881868    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dnb5\" (UniqueName: \"kubernetes.io/projected/ee5fed99-83c8-46d3-a07c-b1db0da31534-kube-api-access-6dnb5\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hncld\" (UID: \"ee5fed99-83c8-46d3-a07c-b1db0da31534\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hncld"
Oct 30 18:51:07.881986 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:07.881915    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01a5c107-dcf4-4ca5-8c9d-57c40814206d-istio-data\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-gtdr6\" (UID: \"01a5c107-dcf4-4ca5-8c9d-57c40814206d\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-gtdr6"
Oct 30 18:51:07.881986 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:07.881968    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01a5c107-dcf4-4ca5-8c9d-57c40814206d-istio-podinfo\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-gtdr6\" (UID: \"01a5c107-dcf4-4ca5-8c9d-57c40814206d\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-gtdr6"
Oct 30 18:51:07.882107 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:07.882007    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01a5c107-dcf4-4ca5-8c9d-57c40814206d-istiod-ca-cert\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-gtdr6\" (UID: \"01a5c107-dcf4-4ca5-8c9d-57c40814206d\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-gtdr6"
Oct 30 18:51:07.882107 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:07.882057    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01a5c107-dcf4-4ca5-8c9d-57c40814206d-istio-envoy\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-gtdr6\" (UID: \"01a5c107-dcf4-4ca5-8c9d-57c40814206d\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-gtdr6"
Oct 30 18:51:07.983836 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:07.983751    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ee5fed99-83c8-46d3-a07c-b1db0da31534-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hncld\" (UID: \"ee5fed99-83c8-46d3-a07c-b1db0da31534\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hncld"
Oct 30 18:51:07.983836 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:07.983813    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee5fed99-83c8-46d3-a07c-b1db0da31534-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hncld\" (UID: \"ee5fed99-83c8-46d3-a07c-b1db0da31534\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hncld"
Oct 30 18:51:07.984035 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:07.983927    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ee5fed99-83c8-46d3-a07c-b1db0da31534-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hncld\" (UID: \"ee5fed99-83c8-46d3-a07c-b1db0da31534\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hncld"
Oct 30 18:51:07.984035 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:07.983973    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01a5c107-dcf4-4ca5-8c9d-57c40814206d-istio-token\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-gtdr6\" (UID: \"01a5c107-dcf4-4ca5-8c9d-57c40814206d\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-gtdr6"
Oct 30 18:51:07.984035 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:07.984008    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ee5fed99-83c8-46d3-a07c-b1db0da31534-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hncld\" (UID: \"ee5fed99-83c8-46d3-a07c-b1db0da31534\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hncld"
Oct 30 18:51:07.984169 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:07.984063    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ee5fed99-83c8-46d3-a07c-b1db0da31534-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hncld\" (UID: \"ee5fed99-83c8-46d3-a07c-b1db0da31534\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hncld"
Oct 30 18:51:07.984169 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:07.984106    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee5fed99-83c8-46d3-a07c-b1db0da31534-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hncld\" (UID: \"ee5fed99-83c8-46d3-a07c-b1db0da31534\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hncld"
Oct 30 18:51:07.984277 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:07.984175    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trs94\" (UniqueName: \"kubernetes.io/projected/01a5c107-dcf4-4ca5-8c9d-57c40814206d-kube-api-access-trs94\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-gtdr6\" (UID: \"01a5c107-dcf4-4ca5-8c9d-57c40814206d\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-gtdr6"
Oct 30 18:51:07.984277 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:07.984219    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a5c107-dcf4-4ca5-8c9d-57c40814206d-workload-socket\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-gtdr6\" (UID: \"01a5c107-dcf4-4ca5-8c9d-57c40814206d\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-gtdr6"
Oct 30 18:51:07.984277 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:07.984270    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a5c107-dcf4-4ca5-8c9d-57c40814206d-credential-socket\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-gtdr6\" (UID: \"01a5c107-dcf4-4ca5-8c9d-57c40814206d\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-gtdr6"
Oct 30 18:51:07.984447 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:07.984319    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01a5c107-dcf4-4ca5-8c9d-57c40814206d-workload-certs\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-gtdr6\" (UID: \"01a5c107-dcf4-4ca5-8c9d-57c40814206d\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-gtdr6"
Oct 30 18:51:07.984447 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:07.984431    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dnb5\" (UniqueName: \"kubernetes.io/projected/ee5fed99-83c8-46d3-a07c-b1db0da31534-kube-api-access-6dnb5\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hncld\" (UID: \"ee5fed99-83c8-46d3-a07c-b1db0da31534\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hncld"
Oct 30 18:51:07.984559 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:07.984476    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01a5c107-dcf4-4ca5-8c9d-57c40814206d-istio-data\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-gtdr6\" (UID: \"01a5c107-dcf4-4ca5-8c9d-57c40814206d\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-gtdr6"
Oct 30 18:51:07.984559 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:07.984502    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01a5c107-dcf4-4ca5-8c9d-57c40814206d-istio-podinfo\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-gtdr6\" (UID: \"01a5c107-dcf4-4ca5-8c9d-57c40814206d\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-gtdr6"
Oct 30 18:51:07.984559 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:07.984531    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01a5c107-dcf4-4ca5-8c9d-57c40814206d-istiod-ca-cert\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-gtdr6\" (UID: \"01a5c107-dcf4-4ca5-8c9d-57c40814206d\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-gtdr6"
Oct 30 18:51:07.984768 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:07.984563    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01a5c107-dcf4-4ca5-8c9d-57c40814206d-istio-envoy\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-gtdr6\" (UID: \"01a5c107-dcf4-4ca5-8c9d-57c40814206d\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-gtdr6"
Oct 30 18:51:07.984948 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:07.984924    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ee5fed99-83c8-46d3-a07c-b1db0da31534-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hncld\" (UID: \"ee5fed99-83c8-46d3-a07c-b1db0da31534\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hncld"
Oct 30 18:51:07.985424 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:07.985395    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a5c107-dcf4-4ca5-8c9d-57c40814206d-credential-socket\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-gtdr6\" (UID: \"01a5c107-dcf4-4ca5-8c9d-57c40814206d\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-gtdr6"
Oct 30 18:51:07.985737 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:07.985671    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01a5c107-dcf4-4ca5-8c9d-57c40814206d-workload-certs\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-gtdr6\" (UID: \"01a5c107-dcf4-4ca5-8c9d-57c40814206d\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-gtdr6"
Oct 30 18:51:07.986016 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:07.985975    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01a5c107-dcf4-4ca5-8c9d-57c40814206d-istio-data\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-gtdr6\" (UID: \"01a5c107-dcf4-4ca5-8c9d-57c40814206d\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-gtdr6"
Oct 30 18:51:07.986117 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:07.986031    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01a5c107-dcf4-4ca5-8c9d-57c40814206d-istiod-ca-cert\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-gtdr6\" (UID: \"01a5c107-dcf4-4ca5-8c9d-57c40814206d\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-gtdr6"
Oct 30 18:51:07.986117 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:07.986048    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a5c107-dcf4-4ca5-8c9d-57c40814206d-workload-socket\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-gtdr6\" (UID: \"01a5c107-dcf4-4ca5-8c9d-57c40814206d\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-gtdr6"
Oct 30 18:51:07.987295 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:07.987213    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ee5fed99-83c8-46d3-a07c-b1db0da31534-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hncld\" (UID: \"ee5fed99-83c8-46d3-a07c-b1db0da31534\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hncld"
Oct 30 18:51:07.987295 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:07.987274    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ee5fed99-83c8-46d3-a07c-b1db0da31534-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hncld\" (UID: \"ee5fed99-83c8-46d3-a07c-b1db0da31534\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hncld"
Oct 30 18:51:07.987591 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:07.987567    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee5fed99-83c8-46d3-a07c-b1db0da31534-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hncld\" (UID: \"ee5fed99-83c8-46d3-a07c-b1db0da31534\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hncld"
Oct 30 18:51:07.987856 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:07.987835    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01a5c107-dcf4-4ca5-8c9d-57c40814206d-istio-envoy\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-gtdr6\" (UID: \"01a5c107-dcf4-4ca5-8c9d-57c40814206d\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-gtdr6"
Oct 30 18:51:07.988013 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:07.987993    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ee5fed99-83c8-46d3-a07c-b1db0da31534-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hncld\" (UID: \"ee5fed99-83c8-46d3-a07c-b1db0da31534\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hncld"
Oct 30 18:51:07.988544 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:07.988524    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01a5c107-dcf4-4ca5-8c9d-57c40814206d-istio-podinfo\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-gtdr6\" (UID: \"01a5c107-dcf4-4ca5-8c9d-57c40814206d\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-gtdr6"
Oct 30 18:51:07.992894 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:07.992869    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01a5c107-dcf4-4ca5-8c9d-57c40814206d-istio-token\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-gtdr6\" (UID: \"01a5c107-dcf4-4ca5-8c9d-57c40814206d\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-gtdr6"
Oct 30 18:51:07.992894 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:07.992888    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trs94\" (UniqueName: \"kubernetes.io/projected/01a5c107-dcf4-4ca5-8c9d-57c40814206d-kube-api-access-trs94\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-gtdr6\" (UID: \"01a5c107-dcf4-4ca5-8c9d-57c40814206d\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-gtdr6"
Oct 30 18:51:07.993485 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:07.993460    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dnb5\" (UniqueName: \"kubernetes.io/projected/ee5fed99-83c8-46d3-a07c-b1db0da31534-kube-api-access-6dnb5\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hncld\" (UID: \"ee5fed99-83c8-46d3-a07c-b1db0da31534\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hncld"
Oct 30 18:51:07.994230 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:07.994212    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee5fed99-83c8-46d3-a07c-b1db0da31534-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hncld\" (UID: \"ee5fed99-83c8-46d3-a07c-b1db0da31534\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hncld"
Oct 30 18:51:08.002255 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:08.002229    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hncld"
Oct 30 18:51:08.058746 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:08.058664    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-gtdr6"
Oct 30 18:51:08.311383 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:08.311355    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hncld"]
Oct 30 18:51:08.313866 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:51:08.313835    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee5fed99_83c8_46d3_a07c_b1db0da31534.slice/crio-5d6037f47c794411625b92cabc6f3f5a6f9406c270ae9dc460232ff68643babf WatchSource:0}: Error finding container 5d6037f47c794411625b92cabc6f3f5a6f9406c270ae9dc460232ff68643babf: Status 404 returned error can't find the container with id 5d6037f47c794411625b92cabc6f3f5a6f9406c270ae9dc460232ff68643babf
Oct 30 18:51:08.375275 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:08.375243    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-gtdr6"]
Oct 30 18:51:08.375765 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:51:08.375732    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01a5c107_dcf4_4ca5_8c9d_57c40814206d.slice/crio-55649931f8faa01eae62c9cefcf93ab82bf1b4b3fe384727c91f82c3219b771f WatchSource:0}: Error finding container 55649931f8faa01eae62c9cefcf93ab82bf1b4b3fe384727c91f82c3219b771f: Status 404 returned error can't find the container with id 55649931f8faa01eae62c9cefcf93ab82bf1b4b3fe384727c91f82c3219b771f
Oct 30 18:51:08.378300 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:08.378263    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:51:08.378408 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:08.378331    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:51:08.378408 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:08.378359    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:51:09.199703 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:09.196004    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-gtdr6" event={"ID":"01a5c107-dcf4-4ca5-8c9d-57c40814206d","Type":"ContainerStarted","Data":"d06820e7a9f2c469f042b89ef9b369d6078f46df9a49292147b2e8e81b1c55d4"}
Oct 30 18:51:09.199703 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:09.196078    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-gtdr6" event={"ID":"01a5c107-dcf4-4ca5-8c9d-57c40814206d","Type":"ContainerStarted","Data":"55649931f8faa01eae62c9cefcf93ab82bf1b4b3fe384727c91f82c3219b771f"}
Oct 30 18:51:09.220272 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:09.220235    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hncld" event={"ID":"ee5fed99-83c8-46d3-a07c-b1db0da31534","Type":"ContainerStarted","Data":"5d6037f47c794411625b92cabc6f3f5a6f9406c270ae9dc460232ff68643babf"}
Oct 30 18:51:09.230363 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:09.230306    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-gtdr6" podStartSLOduration=2.230289905 podStartE2EDuration="2.230289905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:51:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:51:09.226554217 +0000 UTC m=+891.852224181" watchObservedRunningTime="2026-10-30 18:51:09.230289905 +0000 UTC m=+891.855959869"
Oct 30 18:51:10.059313 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:10.059232    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-gtdr6"
Oct 30 18:51:10.061494 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:10.061459    4525 patch_prober.go:29] interesting pod/e2e-worker-w3-2889dd-openshift-default-84768d69f7-gtdr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 18:51:10.061716 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:10.061524    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-gtdr6" podUID="01a5c107-dcf4-4ca5-8c9d-57c40814206d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 18:51:11.059547 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:11.059514    4525 patch_prober.go:29] interesting pod/e2e-worker-w3-2889dd-openshift-default-84768d69f7-gtdr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 18:51:11.060085 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:11.059573    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-gtdr6" podUID="01a5c107-dcf4-4ca5-8c9d-57c40814206d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 18:51:12.024878 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:12.024845    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:51:12.024971 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:12.024906    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:51:12.059864 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:12.059831    4525 patch_prober.go:29] interesting pod/e2e-worker-w3-2889dd-openshift-default-84768d69f7-gtdr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 18:51:12.060273 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:12.059895    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-gtdr6" podUID="01a5c107-dcf4-4ca5-8c9d-57c40814206d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 18:51:12.246166 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:12.246126    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hncld" event={"ID":"ee5fed99-83c8-46d3-a07c-b1db0da31534","Type":"ContainerStarted","Data":"9b961518a9533c405586dc32f8a69ea172918fef5320e297d267976ccd72b5fa"}
Oct 30 18:51:12.246363 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:12.246197    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hncld"
Oct 30 18:51:12.273934 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:12.273819    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hncld" podStartSLOduration=1.566423419 podStartE2EDuration="5.273801014s" totalImagesPullingTime="3.707377595s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:51:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:51:12.266767306 +0000 UTC m=+894.892437269" watchObservedRunningTime="2026-10-30 18:51:12.273801014 +0000 UTC m=+894.899470978"
Oct 30 18:51:13.059823 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:13.059777    4525 patch_prober.go:29] interesting pod/e2e-worker-w3-2889dd-openshift-default-84768d69f7-gtdr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 18:51:13.059995 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:13.059854    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-gtdr6" podUID="01a5c107-dcf4-4ca5-8c9d-57c40814206d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 18:51:13.263906 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:13.263859    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hncld"
Oct 30 18:51:14.146728 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:14.121272    4525 patch_prober.go:29] interesting pod/e2e-worker-w3-2889dd-openshift-default-84768d69f7-gtdr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 18:51:14.146728 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:14.122317    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-gtdr6" podUID="01a5c107-dcf4-4ca5-8c9d-57c40814206d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 18:51:14.366775 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:14.366737    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-zvv2m"]
Oct 30 18:51:14.394926 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:14.394887    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-zvv2m"]
Oct 30 18:51:14.395267 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:14.395246    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-zvv2m"
Oct 30 18:51:14.489940 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:14.488150    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c2697b3-2f09-4c2e-a6b6-abd10d93835c-workload-socket\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-zvv2m\" (UID: \"7c2697b3-2f09-4c2e-a6b6-abd10d93835c\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-zvv2m"
Oct 30 18:51:14.489940 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:14.488277    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c2697b3-2f09-4c2e-a6b6-abd10d93835c-credential-socket\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-zvv2m\" (UID: \"7c2697b3-2f09-4c2e-a6b6-abd10d93835c\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-zvv2m"
Oct 30 18:51:14.489940 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:14.488309    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxfnp\" (UniqueName: \"kubernetes.io/projected/7c2697b3-2f09-4c2e-a6b6-abd10d93835c-kube-api-access-gxfnp\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-zvv2m\" (UID: \"7c2697b3-2f09-4c2e-a6b6-abd10d93835c\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-zvv2m"
Oct 30 18:51:14.489940 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:14.488370    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c2697b3-2f09-4c2e-a6b6-abd10d93835c-istio-envoy\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-zvv2m\" (UID: \"7c2697b3-2f09-4c2e-a6b6-abd10d93835c\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-zvv2m"
Oct 30 18:51:14.489940 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:14.488406    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c2697b3-2f09-4c2e-a6b6-abd10d93835c-istio-token\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-zvv2m\" (UID: \"7c2697b3-2f09-4c2e-a6b6-abd10d93835c\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-zvv2m"
Oct 30 18:51:14.489940 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:14.488549    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c2697b3-2f09-4c2e-a6b6-abd10d93835c-istiod-ca-cert\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-zvv2m\" (UID: \"7c2697b3-2f09-4c2e-a6b6-abd10d93835c\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-zvv2m"
Oct 30 18:51:14.489940 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:14.488590    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c2697b3-2f09-4c2e-a6b6-abd10d93835c-workload-certs\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-zvv2m\" (UID: \"7c2697b3-2f09-4c2e-a6b6-abd10d93835c\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-zvv2m"
Oct 30 18:51:14.489940 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:14.488621    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c2697b3-2f09-4c2e-a6b6-abd10d93835c-istio-data\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-zvv2m\" (UID: \"7c2697b3-2f09-4c2e-a6b6-abd10d93835c\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-zvv2m"
Oct 30 18:51:14.489940 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:14.488644    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c2697b3-2f09-4c2e-a6b6-abd10d93835c-istio-podinfo\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-zvv2m\" (UID: \"7c2697b3-2f09-4c2e-a6b6-abd10d93835c\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-zvv2m"
Oct 30 18:51:14.590434 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:14.590390    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c2697b3-2f09-4c2e-a6b6-abd10d93835c-workload-socket\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-zvv2m\" (UID: \"7c2697b3-2f09-4c2e-a6b6-abd10d93835c\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-zvv2m"
Oct 30 18:51:14.590637 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:14.590494    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c2697b3-2f09-4c2e-a6b6-abd10d93835c-credential-socket\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-zvv2m\" (UID: \"7c2697b3-2f09-4c2e-a6b6-abd10d93835c\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-zvv2m"
Oct 30 18:51:14.590637 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:14.590525    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxfnp\" (UniqueName: \"kubernetes.io/projected/7c2697b3-2f09-4c2e-a6b6-abd10d93835c-kube-api-access-gxfnp\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-zvv2m\" (UID: \"7c2697b3-2f09-4c2e-a6b6-abd10d93835c\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-zvv2m"
Oct 30 18:51:14.590637 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:14.590585    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c2697b3-2f09-4c2e-a6b6-abd10d93835c-istio-envoy\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-zvv2m\" (UID: \"7c2697b3-2f09-4c2e-a6b6-abd10d93835c\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-zvv2m"
Oct 30 18:51:14.590637 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:14.590619    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c2697b3-2f09-4c2e-a6b6-abd10d93835c-istio-token\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-zvv2m\" (UID: \"7c2697b3-2f09-4c2e-a6b6-abd10d93835c\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-zvv2m"
Oct 30 18:51:14.590894 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:14.590753    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c2697b3-2f09-4c2e-a6b6-abd10d93835c-istiod-ca-cert\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-zvv2m\" (UID: \"7c2697b3-2f09-4c2e-a6b6-abd10d93835c\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-zvv2m"
Oct 30 18:51:14.590894 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:14.590785    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c2697b3-2f09-4c2e-a6b6-abd10d93835c-workload-certs\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-zvv2m\" (UID: \"7c2697b3-2f09-4c2e-a6b6-abd10d93835c\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-zvv2m"
Oct 30 18:51:14.590894 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:14.590818    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c2697b3-2f09-4c2e-a6b6-abd10d93835c-istio-data\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-zvv2m\" (UID: \"7c2697b3-2f09-4c2e-a6b6-abd10d93835c\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-zvv2m"
Oct 30 18:51:14.590894 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:14.590846    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c2697b3-2f09-4c2e-a6b6-abd10d93835c-istio-podinfo\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-zvv2m\" (UID: \"7c2697b3-2f09-4c2e-a6b6-abd10d93835c\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-zvv2m"
Oct 30 18:51:14.600384 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:14.599399    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c2697b3-2f09-4c2e-a6b6-abd10d93835c-workload-socket\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-zvv2m\" (UID: \"7c2697b3-2f09-4c2e-a6b6-abd10d93835c\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-zvv2m"
Oct 30 18:51:14.600384 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:14.599634    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c2697b3-2f09-4c2e-a6b6-abd10d93835c-credential-socket\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-zvv2m\" (UID: \"7c2697b3-2f09-4c2e-a6b6-abd10d93835c\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-zvv2m"
Oct 30 18:51:14.600384 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:14.599922    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c2697b3-2f09-4c2e-a6b6-abd10d93835c-workload-certs\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-zvv2m\" (UID: \"7c2697b3-2f09-4c2e-a6b6-abd10d93835c\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-zvv2m"
Oct 30 18:51:14.600384 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:14.600121    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c2697b3-2f09-4c2e-a6b6-abd10d93835c-istiod-ca-cert\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-zvv2m\" (UID: \"7c2697b3-2f09-4c2e-a6b6-abd10d93835c\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-zvv2m"
Oct 30 18:51:14.600384 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:14.600174    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c2697b3-2f09-4c2e-a6b6-abd10d93835c-istio-data\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-zvv2m\" (UID: \"7c2697b3-2f09-4c2e-a6b6-abd10d93835c\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-zvv2m"
Oct 30 18:51:14.608369 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:14.608338    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c2697b3-2f09-4c2e-a6b6-abd10d93835c-istio-podinfo\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-zvv2m\" (UID: \"7c2697b3-2f09-4c2e-a6b6-abd10d93835c\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-zvv2m"
Oct 30 18:51:14.609192 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:14.609155    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxfnp\" (UniqueName: \"kubernetes.io/projected/7c2697b3-2f09-4c2e-a6b6-abd10d93835c-kube-api-access-gxfnp\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-zvv2m\" (UID: \"7c2697b3-2f09-4c2e-a6b6-abd10d93835c\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-zvv2m"
Oct 30 18:51:14.609456 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:14.609431    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c2697b3-2f09-4c2e-a6b6-abd10d93835c-istio-envoy\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-zvv2m\" (UID: \"7c2697b3-2f09-4c2e-a6b6-abd10d93835c\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-zvv2m"
Oct 30 18:51:14.610531 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:14.610497    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c2697b3-2f09-4c2e-a6b6-abd10d93835c-istio-token\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-zvv2m\" (UID: \"7c2697b3-2f09-4c2e-a6b6-abd10d93835c\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-zvv2m"
Oct 30 18:51:14.761609 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:14.761495    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-zvv2m"
Oct 30 18:51:14.975174 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:14.975146    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b473ede5-lndt8"]
Oct 30 18:51:14.980045 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:14.980028    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b473ede5-lndt8"
Oct 30 18:51:14.983812 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:14.983758    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:51:14.984251 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:14.984233    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94hcn\""
Oct 30 18:51:14.988274 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:14.988245    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b473ede5-lndt8"]
Oct 30 18:51:15.010992 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:15.010970    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3d06734-31e4-4da4-9618-f8aadfad076a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b473ede5-lndt8\" (UID: \"e3d06734-31e4-4da4-9618-f8aadfad076a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b473ede5-lndt8"
Oct 30 18:51:15.011111 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:15.011045    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3d06734-31e4-4da4-9618-f8aadfad076a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b473ede5-lndt8\" (UID: \"e3d06734-31e4-4da4-9618-f8aadfad076a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b473ede5-lndt8"
Oct 30 18:51:15.011111 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:15.011103    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj2w7\" (UniqueName: \"kubernetes.io/projected/e3d06734-31e4-4da4-9618-f8aadfad076a-kube-api-access-rj2w7\") pod \"maas-api-revoke-sub-b473ede5-lndt8\" (UID: \"e3d06734-31e4-4da4-9618-f8aadfad076a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b473ede5-lndt8"
Oct 30 18:51:15.060090 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:15.060009    4525 patch_prober.go:29] interesting pod/e2e-worker-w3-2889dd-openshift-default-84768d69f7-gtdr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 30 18:51:15.060233 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:15.060077    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-gtdr6" podUID="01a5c107-dcf4-4ca5-8c9d-57c40814206d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 30 18:51:15.071180 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:15.071112    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-zvv2m"]
Oct 30 18:51:15.072200 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:51:15.071561    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c2697b3_2f09_4c2e_a6b6_abd10d93835c.slice/crio-b3133f42a09fd34ff6c4909079014495b217d3b5fdb1e61124b9392d0a57af12 WatchSource:0}: Error finding container b3133f42a09fd34ff6c4909079014495b217d3b5fdb1e61124b9392d0a57af12: Status 404 returned error can't find the container with id b3133f42a09fd34ff6c4909079014495b217d3b5fdb1e61124b9392d0a57af12
Oct 30 18:51:15.074423 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:15.074389    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:51:15.074517 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:15.074458    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:51:15.074517 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:15.074487    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:51:15.111999 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:15.111973    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3d06734-31e4-4da4-9618-f8aadfad076a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b473ede5-lndt8\" (UID: \"e3d06734-31e4-4da4-9618-f8aadfad076a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b473ede5-lndt8"
Oct 30 18:51:15.112107 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:15.112021    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj2w7\" (UniqueName: \"kubernetes.io/projected/e3d06734-31e4-4da4-9618-f8aadfad076a-kube-api-access-rj2w7\") pod \"maas-api-revoke-sub-b473ede5-lndt8\" (UID: \"e3d06734-31e4-4da4-9618-f8aadfad076a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b473ede5-lndt8"
Oct 30 18:51:15.112149 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:15.112125    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3d06734-31e4-4da4-9618-f8aadfad076a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b473ede5-lndt8\" (UID: \"e3d06734-31e4-4da4-9618-f8aadfad076a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b473ede5-lndt8"
Oct 30 18:51:15.112643 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:15.112616    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3d06734-31e4-4da4-9618-f8aadfad076a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b473ede5-lndt8\" (UID: \"e3d06734-31e4-4da4-9618-f8aadfad076a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b473ede5-lndt8"
Oct 30 18:51:15.114449 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:15.114429    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3d06734-31e4-4da4-9618-f8aadfad076a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b473ede5-lndt8\" (UID: \"e3d06734-31e4-4da4-9618-f8aadfad076a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b473ede5-lndt8"
Oct 30 18:51:15.120310 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:15.120291    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj2w7\" (UniqueName: \"kubernetes.io/projected/e3d06734-31e4-4da4-9618-f8aadfad076a-kube-api-access-rj2w7\") pod \"maas-api-revoke-sub-b473ede5-lndt8\" (UID: \"e3d06734-31e4-4da4-9618-f8aadfad076a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b473ede5-lndt8"
Oct 30 18:51:15.303358 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:15.303321    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b473ede5-lndt8"
Oct 30 18:51:15.303787 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:15.303567    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51fe5e9b-pskxl"]
Oct 30 18:51:15.308581 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:15.308561    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51fe5e9b-pskxl"
Oct 30 18:51:15.313648 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:15.313589    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51fe5e9b-pskxl"]
Oct 30 18:51:15.313946 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:15.313750    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ps9w\" (UniqueName: \"kubernetes.io/projected/412a1669-045f-4356-a0b4-773b3988ee04-kube-api-access-5ps9w\") pod \"maas-api-revoke-sub-51fe5e9b-pskxl\" (UID: \"412a1669-045f-4356-a0b4-773b3988ee04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51fe5e9b-pskxl"
Oct 30 18:51:15.314816 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:15.314793    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/412a1669-045f-4356-a0b4-773b3988ee04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51fe5e9b-pskxl\" (UID: \"412a1669-045f-4356-a0b4-773b3988ee04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51fe5e9b-pskxl"
Oct 30 18:51:15.314950 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:15.314925    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/412a1669-045f-4356-a0b4-773b3988ee04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51fe5e9b-pskxl\" (UID: \"412a1669-045f-4356-a0b4-773b3988ee04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51fe5e9b-pskxl"
Oct 30 18:51:15.397910 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:15.397657    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-zvv2m" event={"ID":"7c2697b3-2f09-4c2e-a6b6-abd10d93835c","Type":"ContainerStarted","Data":"d79e73b4d3461f461b871b6ad2c34e85d4c522c2dc7fe1a6dca1e2f8a61dbd6b"}
Oct 30 18:51:15.397910 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:15.397735    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-zvv2m" event={"ID":"7c2697b3-2f09-4c2e-a6b6-abd10d93835c","Type":"ContainerStarted","Data":"b3133f42a09fd34ff6c4909079014495b217d3b5fdb1e61124b9392d0a57af12"}
Oct 30 18:51:15.417652 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:15.415868    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/412a1669-045f-4356-a0b4-773b3988ee04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51fe5e9b-pskxl\" (UID: \"412a1669-045f-4356-a0b4-773b3988ee04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51fe5e9b-pskxl"
Oct 30 18:51:15.417652 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:15.416016    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ps9w\" (UniqueName: \"kubernetes.io/projected/412a1669-045f-4356-a0b4-773b3988ee04-kube-api-access-5ps9w\") pod \"maas-api-revoke-sub-51fe5e9b-pskxl\" (UID: \"412a1669-045f-4356-a0b4-773b3988ee04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51fe5e9b-pskxl"
Oct 30 18:51:15.417652 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:15.416136    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/412a1669-045f-4356-a0b4-773b3988ee04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51fe5e9b-pskxl\" (UID: \"412a1669-045f-4356-a0b4-773b3988ee04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51fe5e9b-pskxl"
Oct 30 18:51:15.422913 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:15.418753    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/412a1669-045f-4356-a0b4-773b3988ee04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51fe5e9b-pskxl\" (UID: \"412a1669-045f-4356-a0b4-773b3988ee04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51fe5e9b-pskxl"
Oct 30 18:51:15.431983 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:15.429773    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/412a1669-045f-4356-a0b4-773b3988ee04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51fe5e9b-pskxl\" (UID: \"412a1669-045f-4356-a0b4-773b3988ee04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51fe5e9b-pskxl"
Oct 30 18:51:15.431983 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:15.430361    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ps9w\" (UniqueName: \"kubernetes.io/projected/412a1669-045f-4356-a0b4-773b3988ee04-kube-api-access-5ps9w\") pod \"maas-api-revoke-sub-51fe5e9b-pskxl\" (UID: \"412a1669-045f-4356-a0b4-773b3988ee04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51fe5e9b-pskxl"
Oct 30 18:51:15.431983 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:15.431042    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-zvv2m" podStartSLOduration=2.431025294 podStartE2EDuration="2.431025294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:51:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:51:15.42462365 +0000 UTC m=+898.050293615" watchObservedRunningTime="2026-10-30 18:51:15.431025294 +0000 UTC m=+898.056695257"
Oct 30 18:51:15.630655 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:15.623102    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b473ede5-lndt8"]
Oct 30 18:51:15.631638 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:15.631099    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51fe5e9b-pskxl"
Oct 30 18:51:15.764598 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:15.763837    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-zvv2m"
Oct 30 18:51:16.013606 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:16.013446    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51fe5e9b-pskxl"]
Oct 30 18:51:16.015587 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:51:16.015562    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod412a1669_045f_4356_a0b4_773b3988ee04.slice/crio-8d628c9c17b60126423621e1491c3ed9cc8fafd00ba5eb363dda859aa9edb0d3 WatchSource:0}: Error finding container 8d628c9c17b60126423621e1491c3ed9cc8fafd00ba5eb363dda859aa9edb0d3: Status 404 returned error can't find the container with id 8d628c9c17b60126423621e1491c3ed9cc8fafd00ba5eb363dda859aa9edb0d3
Oct 30 18:51:16.337366 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:16.337336    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-gtdr6"
Oct 30 18:51:16.338507 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:16.338479    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-gtdr6"
Oct 30 18:51:16.339200 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:16.339178    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-gtdr6"
Oct 30 18:51:16.418416 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:16.418379    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b473ede5-lndt8" event={"ID":"e3d06734-31e4-4da4-9618-f8aadfad076a","Type":"ContainerStarted","Data":"94693d8f259404209b001c5a7b33d3e73316681d5af18545a241feff6acfc4dd"}
Oct 30 18:51:16.418586 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:16.418455    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b473ede5-lndt8" event={"ID":"e3d06734-31e4-4da4-9618-f8aadfad076a","Type":"ContainerStarted","Data":"160c85324e48089941603eb0690208c4231a21e5f3e68fe8ccfd2eee61c1dfc1"}
Oct 30 18:51:16.420182 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:16.420152    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51fe5e9b-pskxl" event={"ID":"412a1669-045f-4356-a0b4-773b3988ee04","Type":"ContainerStarted","Data":"8d628c9c17b60126423621e1491c3ed9cc8fafd00ba5eb363dda859aa9edb0d3"}
Oct 30 18:51:16.436002 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:16.435941    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b473ede5-lndt8" podStartSLOduration=2.435922296 podStartE2EDuration="2.435922296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:51:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:51:16.433863402 +0000 UTC m=+899.059533366" watchObservedRunningTime="2026-10-30 18:51:16.435922296 +0000 UTC m=+899.061592264"
Oct 30 18:51:16.763363 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:16.763328    4525 patch_prober.go:29] interesting pod/e2e-worker-w4-0458df-openshift-default-7c86d79f79-zvv2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 18:51:16.763552 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:16.763387    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-zvv2m" podUID="7c2697b3-2f09-4c2e-a6b6-abd10d93835c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 18:51:17.426788 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:51:17.426706    4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d49l" containerName="extractor"
Oct 30 18:51:17.430223 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:17.430200    4525 generic.go:356] "Generic (PLEG): container finished" podID="e3d06734-31e4-4da4-9618-f8aadfad076a" containerID="94693d8f259404209b001c5a7b33d3e73316681d5af18545a241feff6acfc4dd" exitCode=0
Oct 30 18:51:17.430549 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:17.430528    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b473ede5-lndt8" event={"ID":"e3d06734-31e4-4da4-9618-f8aadfad076a","Type":"ContainerDied","Data":"94693d8f259404209b001c5a7b33d3e73316681d5af18545a241feff6acfc4dd"}
Oct 30 18:51:17.432399 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:17.432367    4525 generic.go:356] "Generic (PLEG): container finished" podID="412a1669-045f-4356-a0b4-773b3988ee04" containerID="8eb32162ab54c2160093181a6f3dc189bbde406f1530c47fbc347f9b0477976a" exitCode=0
Oct 30 18:51:17.432517 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:17.432436    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51fe5e9b-pskxl" event={"ID":"412a1669-045f-4356-a0b4-773b3988ee04","Type":"ContainerDied","Data":"8eb32162ab54c2160093181a6f3dc189bbde406f1530c47fbc347f9b0477976a"}
Oct 30 18:51:17.761921 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:17.761889    4525 patch_prober.go:29] interesting pod/e2e-worker-w4-0458df-openshift-default-7c86d79f79-zvv2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 18:51:17.762095 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:17.761954    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-zvv2m" podUID="7c2697b3-2f09-4c2e-a6b6-abd10d93835c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 18:51:18.699756 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:18.697138    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b473ede5-lndt8"
Oct 30 18:51:18.735717 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:18.735651    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51fe5e9b-pskxl"
Oct 30 18:51:18.762147 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:18.762034    4525 patch_prober.go:29] interesting pod/e2e-worker-w4-0458df-openshift-default-7c86d79f79-zvv2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 18:51:18.762147 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:18.762091    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-zvv2m" podUID="7c2697b3-2f09-4c2e-a6b6-abd10d93835c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 18:51:18.764532 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:18.764056    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/412a1669-045f-4356-a0b4-773b3988ee04-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/412a1669-045f-4356-a0b4-773b3988ee04-maas-api-xks-ca\") pod \"412a1669-045f-4356-a0b4-773b3988ee04\" (UID: \"412a1669-045f-4356-a0b4-773b3988ee04\") "
Oct 30 18:51:18.764532 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:18.764147    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3d06734-31e4-4da4-9618-f8aadfad076a-kube-api-access-rj2w7\" (UniqueName: \"kubernetes.io/projected/e3d06734-31e4-4da4-9618-f8aadfad076a-kube-api-access-rj2w7\") pod \"e3d06734-31e4-4da4-9618-f8aadfad076a\" (UID: \"e3d06734-31e4-4da4-9618-f8aadfad076a\") "
Oct 30 18:51:18.764532 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:18.764182    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/412a1669-045f-4356-a0b4-773b3988ee04-kube-api-access-5ps9w\" (UniqueName: \"kubernetes.io/projected/412a1669-045f-4356-a0b4-773b3988ee04-kube-api-access-5ps9w\") pod \"412a1669-045f-4356-a0b4-773b3988ee04\" (UID: \"412a1669-045f-4356-a0b4-773b3988ee04\") "
Oct 30 18:51:18.764532 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:18.764263    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/412a1669-045f-4356-a0b4-773b3988ee04-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/412a1669-045f-4356-a0b4-773b3988ee04-maas-api-openshift-ca\") pod \"412a1669-045f-4356-a0b4-773b3988ee04\" (UID: \"412a1669-045f-4356-a0b4-773b3988ee04\") "
Oct 30 18:51:18.764532 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:18.764328    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3d06734-31e4-4da4-9618-f8aadfad076a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3d06734-31e4-4da4-9618-f8aadfad076a-maas-api-xks-ca\") pod \"e3d06734-31e4-4da4-9618-f8aadfad076a\" (UID: \"e3d06734-31e4-4da4-9618-f8aadfad076a\") "
Oct 30 18:51:18.764532 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:18.764394    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3d06734-31e4-4da4-9618-f8aadfad076a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3d06734-31e4-4da4-9618-f8aadfad076a-maas-api-openshift-ca\") pod \"e3d06734-31e4-4da4-9618-f8aadfad076a\" (UID: \"e3d06734-31e4-4da4-9618-f8aadfad076a\") "
Oct 30 18:51:18.765930 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:18.765102    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3d06734-31e4-4da4-9618-f8aadfad076a-maas-api-openshift-ca" pod "e3d06734-31e4-4da4-9618-f8aadfad076a" (UID: "e3d06734-31e4-4da4-9618-f8aadfad076a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:51:18.766055 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:18.765413    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/412a1669-045f-4356-a0b4-773b3988ee04-maas-api-openshift-ca" pod "412a1669-045f-4356-a0b4-773b3988ee04" (UID: "412a1669-045f-4356-a0b4-773b3988ee04"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:51:18.768054 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:18.768022    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/412a1669-045f-4356-a0b4-773b3988ee04-kube-api-access-5ps9w" pod "412a1669-045f-4356-a0b4-773b3988ee04" (UID: "412a1669-045f-4356-a0b4-773b3988ee04"). InnerVolumeSpecName "kube-api-access-5ps9w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:51:18.771938 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:18.771906    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/412a1669-045f-4356-a0b4-773b3988ee04-maas-api-xks-ca" pod "412a1669-045f-4356-a0b4-773b3988ee04" (UID: "412a1669-045f-4356-a0b4-773b3988ee04"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:51:18.772023 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:18.771941    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3d06734-31e4-4da4-9618-f8aadfad076a-kube-api-access-rj2w7" pod "e3d06734-31e4-4da4-9618-f8aadfad076a" (UID: "e3d06734-31e4-4da4-9618-f8aadfad076a"). InnerVolumeSpecName "kube-api-access-rj2w7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:51:18.773731 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:18.773694    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3d06734-31e4-4da4-9618-f8aadfad076a-maas-api-xks-ca" pod "e3d06734-31e4-4da4-9618-f8aadfad076a" (UID: "e3d06734-31e4-4da4-9618-f8aadfad076a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:51:18.866294 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:18.866260    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rj2w7\" (UniqueName: \"kubernetes.io/projected/e3d06734-31e4-4da4-9618-f8aadfad076a-kube-api-access-rj2w7\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:18.866294 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:18.866287    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ps9w\" (UniqueName: \"kubernetes.io/projected/412a1669-045f-4356-a0b4-773b3988ee04-kube-api-access-5ps9w\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:18.866294 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:18.866297    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/412a1669-045f-4356-a0b4-773b3988ee04-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:18.866532 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:18.866306    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3d06734-31e4-4da4-9618-f8aadfad076a-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:18.866532 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:18.866315    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3d06734-31e4-4da4-9618-f8aadfad076a-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:18.866532 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:18.866323    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/412a1669-045f-4356-a0b4-773b3988ee04-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:19.448110 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:19.448081    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b473ede5-lndt8"
Oct 30 18:51:19.448289 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:19.448107    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b473ede5-lndt8" event={"ID":"e3d06734-31e4-4da4-9618-f8aadfad076a","Type":"ContainerDied","Data":"160c85324e48089941603eb0690208c4231a21e5f3e68fe8ccfd2eee61c1dfc1"}
Oct 30 18:51:19.448289 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:19.448162    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="160c85324e48089941603eb0690208c4231a21e5f3e68fe8ccfd2eee61c1dfc1"
Oct 30 18:51:19.449737 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:19.449713    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51fe5e9b-pskxl"
Oct 30 18:51:19.449840 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:19.449715    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51fe5e9b-pskxl" event={"ID":"412a1669-045f-4356-a0b4-773b3988ee04","Type":"ContainerDied","Data":"8d628c9c17b60126423621e1491c3ed9cc8fafd00ba5eb363dda859aa9edb0d3"}
Oct 30 18:51:19.449840 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:19.449830    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d628c9c17b60126423621e1491c3ed9cc8fafd00ba5eb363dda859aa9edb0d3"
Oct 30 18:51:19.762775 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:19.762672    4525 patch_prober.go:29] interesting pod/e2e-worker-w4-0458df-openshift-default-7c86d79f79-zvv2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 18:51:19.762775 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:19.762743    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-zvv2m" podUID="7c2697b3-2f09-4c2e-a6b6-abd10d93835c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 18:51:20.762187 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:20.762156    4525 patch_prober.go:29] interesting pod/e2e-worker-w4-0458df-openshift-default-7c86d79f79-zvv2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 18:51:20.762356 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:20.762207    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-zvv2m" podUID="7c2697b3-2f09-4c2e-a6b6-abd10d93835c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 18:51:21.762195 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:21.761929    4525 patch_prober.go:29] interesting pod/e2e-worker-w4-0458df-openshift-default-7c86d79f79-zvv2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 18:51:21.762195 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:21.762026    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-zvv2m" podUID="7c2697b3-2f09-4c2e-a6b6-abd10d93835c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 18:51:23.811845 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:23.811801    4525 patch_prober.go:29] interesting pod/e2e-worker-w4-0458df-openshift-default-7c86d79f79-zvv2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 18:51:23.812458 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:23.812426    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-zvv2m" podUID="7c2697b3-2f09-4c2e-a6b6-abd10d93835c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:51:24.294835 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:24.294777    4525 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts" podUID="be6d4969-285b-4da0-883e-4847f2cbf3af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:51:24.559893 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:24.559781    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t" podUID="424fca14-6d8a-4663-86ec-0c91165f34ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:51:24.861240 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:24.861196    4525 patch_prober.go:29] interesting pod/e2e-worker-w4-0458df-openshift-default-7c86d79f79-zvv2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 18:51:24.861814 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:24.861286    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-zvv2m" podUID="7c2697b3-2f09-4c2e-a6b6-abd10d93835c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:51:25.183445 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:25.183362    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-zvv2m"
Oct 30 18:51:25.183798 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:25.183772    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-zvv2m"
Oct 30 18:51:25.203946 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:25.203918    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-zvv2m"
Oct 30 18:51:26.155314 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:26.155279    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32a67598-29l4p"]
Oct 30 18:51:26.155777 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:26.155764    4525 cpu_manager.go:414] "containerMap: removing container" podUID="e3d06734-31e4-4da4-9618-f8aadfad076a" containerName="revoke-keys"
Oct 30 18:51:26.155821 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:26.155779    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3d06734-31e4-4da4-9618-f8aadfad076a" containerName="revoke-keys"
Oct 30 18:51:26.155821 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:26.155799    4525 cpu_manager.go:414] "containerMap: removing container" podUID="412a1669-045f-4356-a0b4-773b3988ee04" containerName="revoke-keys"
Oct 30 18:51:26.155821 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:26.155805    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="412a1669-045f-4356-a0b4-773b3988ee04" containerName="revoke-keys"
Oct 30 18:51:26.155935 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:26.155896    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3d06734-31e4-4da4-9618-f8aadfad076a" containerName="revoke-keys"
Oct 30 18:51:26.155935 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:26.155906    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="412a1669-045f-4356-a0b4-773b3988ee04" containerName="revoke-keys"
Oct 30 18:51:26.160838 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:26.160823    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32a67598-29l4p"
Oct 30 18:51:26.164501 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:26.164478    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:51:26.164859 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:26.164838    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94hcn\""
Oct 30 18:51:26.167723 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:26.167702    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32a67598-29l4p"]
Oct 30 18:51:26.290036 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:26.289994    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24fb6a8c-100c-43d8-ae55-d1b6d6fcb437-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32a67598-29l4p\" (UID: \"24fb6a8c-100c-43d8-ae55-d1b6d6fcb437\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32a67598-29l4p"
Oct 30 18:51:26.290197 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:26.290042    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5c9x\" (UniqueName: \"kubernetes.io/projected/24fb6a8c-100c-43d8-ae55-d1b6d6fcb437-kube-api-access-h5c9x\") pod \"maas-api-revoke-sub-32a67598-29l4p\" (UID: \"24fb6a8c-100c-43d8-ae55-d1b6d6fcb437\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32a67598-29l4p"
Oct 30 18:51:26.290197 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:26.290154    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24fb6a8c-100c-43d8-ae55-d1b6d6fcb437-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32a67598-29l4p\" (UID: \"24fb6a8c-100c-43d8-ae55-d1b6d6fcb437\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32a67598-29l4p"
Oct 30 18:51:26.391409 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:26.391366    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24fb6a8c-100c-43d8-ae55-d1b6d6fcb437-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32a67598-29l4p\" (UID: \"24fb6a8c-100c-43d8-ae55-d1b6d6fcb437\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32a67598-29l4p"
Oct 30 18:51:26.391409 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:26.391408    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5c9x\" (UniqueName: \"kubernetes.io/projected/24fb6a8c-100c-43d8-ae55-d1b6d6fcb437-kube-api-access-h5c9x\") pod \"maas-api-revoke-sub-32a67598-29l4p\" (UID: \"24fb6a8c-100c-43d8-ae55-d1b6d6fcb437\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32a67598-29l4p"
Oct 30 18:51:26.391713 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:26.391444    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24fb6a8c-100c-43d8-ae55-d1b6d6fcb437-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32a67598-29l4p\" (UID: \"24fb6a8c-100c-43d8-ae55-d1b6d6fcb437\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32a67598-29l4p"
Oct 30 18:51:26.392132 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:26.392110    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24fb6a8c-100c-43d8-ae55-d1b6d6fcb437-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32a67598-29l4p\" (UID: \"24fb6a8c-100c-43d8-ae55-d1b6d6fcb437\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32a67598-29l4p"
Oct 30 18:51:26.394133 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:26.394106    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24fb6a8c-100c-43d8-ae55-d1b6d6fcb437-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32a67598-29l4p\" (UID: \"24fb6a8c-100c-43d8-ae55-d1b6d6fcb437\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32a67598-29l4p"
Oct 30 18:51:26.399018 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:26.398993    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5c9x\" (UniqueName: \"kubernetes.io/projected/24fb6a8c-100c-43d8-ae55-d1b6d6fcb437-kube-api-access-h5c9x\") pod \"maas-api-revoke-sub-32a67598-29l4p\" (UID: \"24fb6a8c-100c-43d8-ae55-d1b6d6fcb437\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32a67598-29l4p"
Oct 30 18:51:26.477642 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:26.477562    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32a67598-29l4p"
Oct 30 18:51:26.809737 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:26.809616    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32a67598-29l4p"]
Oct 30 18:51:27.555495 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:27.555465    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96c25bd9-7mwc6"]
Oct 30 18:51:27.559752 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:27.559725    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96c25bd9-7mwc6"]
Oct 30 18:51:27.559892 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:27.559836    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c25bd9-7mwc6"
Oct 30 18:51:27.699412 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:27.699373    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32a67598-29l4p" event={"ID":"24fb6a8c-100c-43d8-ae55-d1b6d6fcb437","Type":"ContainerStarted","Data":"5790f8ad72e52dd61d38c16fc2586ad064bbaf7094211564e869b69c19bd4ad5"}
Oct 30 18:51:27.699568 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:27.699449    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32a67598-29l4p" event={"ID":"24fb6a8c-100c-43d8-ae55-d1b6d6fcb437","Type":"ContainerStarted","Data":"c2917023e54242dae2084a46a540c4838a8404b1201d55371f6a80bdaf170b34"}
Oct 30 18:51:27.716540 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:27.716503    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97a1b489-5957-4e3c-93fd-866399427ee0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96c25bd9-7mwc6\" (UID: \"97a1b489-5957-4e3c-93fd-866399427ee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c25bd9-7mwc6"
Oct 30 18:51:27.716726 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:27.716567    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97a1b489-5957-4e3c-93fd-866399427ee0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96c25bd9-7mwc6\" (UID: \"97a1b489-5957-4e3c-93fd-866399427ee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c25bd9-7mwc6"
Oct 30 18:51:27.716726 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:27.716670    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vk5lj\" (UniqueName: \"kubernetes.io/projected/97a1b489-5957-4e3c-93fd-866399427ee0-kube-api-access-vk5lj\") pod \"maas-api-revoke-sub-96c25bd9-7mwc6\" (UID: \"97a1b489-5957-4e3c-93fd-866399427ee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c25bd9-7mwc6"
Oct 30 18:51:27.720278 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:27.720232    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32a67598-29l4p" podStartSLOduration=1.72022009 podStartE2EDuration="1.72022009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:51:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:51:27.71643194 +0000 UTC m=+910.342101905" watchObservedRunningTime="2026-10-30 18:51:27.72022009 +0000 UTC m=+910.345890054"
Oct 30 18:51:27.818350 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:27.818257    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk5lj\" (UniqueName: \"kubernetes.io/projected/97a1b489-5957-4e3c-93fd-866399427ee0-kube-api-access-vk5lj\") pod \"maas-api-revoke-sub-96c25bd9-7mwc6\" (UID: \"97a1b489-5957-4e3c-93fd-866399427ee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c25bd9-7mwc6"
Oct 30 18:51:27.818525 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:27.818451    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97a1b489-5957-4e3c-93fd-866399427ee0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96c25bd9-7mwc6\" (UID: \"97a1b489-5957-4e3c-93fd-866399427ee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c25bd9-7mwc6"
Oct 30 18:51:27.818525 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:27.818510    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97a1b489-5957-4e3c-93fd-866399427ee0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96c25bd9-7mwc6\" (UID: \"97a1b489-5957-4e3c-93fd-866399427ee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c25bd9-7mwc6"
Oct 30 18:51:27.819155 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:27.819129    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97a1b489-5957-4e3c-93fd-866399427ee0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96c25bd9-7mwc6\" (UID: \"97a1b489-5957-4e3c-93fd-866399427ee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c25bd9-7mwc6"
Oct 30 18:51:27.821036 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:27.821007    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97a1b489-5957-4e3c-93fd-866399427ee0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96c25bd9-7mwc6\" (UID: \"97a1b489-5957-4e3c-93fd-866399427ee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c25bd9-7mwc6"
Oct 30 18:51:27.826190 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:27.826167    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vk5lj\" (UniqueName: \"kubernetes.io/projected/97a1b489-5957-4e3c-93fd-866399427ee0-kube-api-access-vk5lj\") pod \"maas-api-revoke-sub-96c25bd9-7mwc6\" (UID: \"97a1b489-5957-4e3c-93fd-866399427ee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c25bd9-7mwc6"
Oct 30 18:51:27.878552 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:27.878519    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c25bd9-7mwc6"
Oct 30 18:51:28.167950 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:51:28.167920    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97a1b489_5957_4e3c_93fd_866399427ee0.slice/crio-f4cfde6171d0b94d7a4bdc09ad9a5dfbaf68b4fde54937b17ffa1269b6371759 WatchSource:0}: Error finding container f4cfde6171d0b94d7a4bdc09ad9a5dfbaf68b4fde54937b17ffa1269b6371759: Status 404 returned error can't find the container with id f4cfde6171d0b94d7a4bdc09ad9a5dfbaf68b4fde54937b17ffa1269b6371759
Oct 30 18:51:28.168234 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:28.168211    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96c25bd9-7mwc6"]
Oct 30 18:51:28.707865 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:28.707828    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c25bd9-7mwc6" event={"ID":"97a1b489-5957-4e3c-93fd-866399427ee0","Type":"ContainerStarted","Data":"c1836498eff8463cf8132187faf31f99c815d4c9019c23eeeaaa4f3d5195354c"}
Oct 30 18:51:28.708316 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:28.707885    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c25bd9-7mwc6" event={"ID":"97a1b489-5957-4e3c-93fd-866399427ee0","Type":"ContainerStarted","Data":"f4cfde6171d0b94d7a4bdc09ad9a5dfbaf68b4fde54937b17ffa1269b6371759"}
Oct 30 18:51:28.709596 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:28.709565    4525 generic.go:356] "Generic (PLEG): container finished" podID="24fb6a8c-100c-43d8-ae55-d1b6d6fcb437" containerID="5790f8ad72e52dd61d38c16fc2586ad064bbaf7094211564e869b69c19bd4ad5" exitCode=0
Oct 30 18:51:28.709771 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:28.709622    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32a67598-29l4p" event={"ID":"24fb6a8c-100c-43d8-ae55-d1b6d6fcb437","Type":"ContainerDied","Data":"5790f8ad72e52dd61d38c16fc2586ad064bbaf7094211564e869b69c19bd4ad5"}
Oct 30 18:51:28.730170 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:28.730113    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c25bd9-7mwc6" podStartSLOduration=1.73009537 podStartE2EDuration="1.73009537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:51:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:51:28.727407988 +0000 UTC m=+911.353077953" watchObservedRunningTime="2026-10-30 18:51:28.73009537 +0000 UTC m=+911.355765335"
Oct 30 18:51:29.721614 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:29.721555    4525 generic.go:356] "Generic (PLEG): container finished" podID="97a1b489-5957-4e3c-93fd-866399427ee0" containerID="c1836498eff8463cf8132187faf31f99c815d4c9019c23eeeaaa4f3d5195354c" exitCode=0
Oct 30 18:51:29.722475 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:29.722447    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c25bd9-7mwc6" event={"ID":"97a1b489-5957-4e3c-93fd-866399427ee0","Type":"ContainerDied","Data":"c1836498eff8463cf8132187faf31f99c815d4c9019c23eeeaaa4f3d5195354c"}
Oct 30 18:51:29.887776 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:29.887747    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32a67598-29l4p"
Oct 30 18:51:30.046380 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:30.046293    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24fb6a8c-100c-43d8-ae55-d1b6d6fcb437-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24fb6a8c-100c-43d8-ae55-d1b6d6fcb437-maas-api-xks-ca\") pod \"24fb6a8c-100c-43d8-ae55-d1b6d6fcb437\" (UID: \"24fb6a8c-100c-43d8-ae55-d1b6d6fcb437\") "
Oct 30 18:51:30.046538 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:30.046436    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24fb6a8c-100c-43d8-ae55-d1b6d6fcb437-kube-api-access-h5c9x\" (UniqueName: \"kubernetes.io/projected/24fb6a8c-100c-43d8-ae55-d1b6d6fcb437-kube-api-access-h5c9x\") pod \"24fb6a8c-100c-43d8-ae55-d1b6d6fcb437\" (UID: \"24fb6a8c-100c-43d8-ae55-d1b6d6fcb437\") "
Oct 30 18:51:30.046538 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:30.046482    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24fb6a8c-100c-43d8-ae55-d1b6d6fcb437-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24fb6a8c-100c-43d8-ae55-d1b6d6fcb437-maas-api-openshift-ca\") pod \"24fb6a8c-100c-43d8-ae55-d1b6d6fcb437\" (UID: \"24fb6a8c-100c-43d8-ae55-d1b6d6fcb437\") "
Oct 30 18:51:30.046867 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:30.046837    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24fb6a8c-100c-43d8-ae55-d1b6d6fcb437-maas-api-openshift-ca" pod "24fb6a8c-100c-43d8-ae55-d1b6d6fcb437" (UID: "24fb6a8c-100c-43d8-ae55-d1b6d6fcb437"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:51:30.047049 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:30.047029    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24fb6a8c-100c-43d8-ae55-d1b6d6fcb437-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:30.048744 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:30.048720    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24fb6a8c-100c-43d8-ae55-d1b6d6fcb437-maas-api-xks-ca" pod "24fb6a8c-100c-43d8-ae55-d1b6d6fcb437" (UID: "24fb6a8c-100c-43d8-ae55-d1b6d6fcb437"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:51:30.048849 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:30.048794    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24fb6a8c-100c-43d8-ae55-d1b6d6fcb437-kube-api-access-h5c9x" pod "24fb6a8c-100c-43d8-ae55-d1b6d6fcb437" (UID: "24fb6a8c-100c-43d8-ae55-d1b6d6fcb437"). InnerVolumeSpecName "kube-api-access-h5c9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:51:30.148202 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:30.148168    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24fb6a8c-100c-43d8-ae55-d1b6d6fcb437-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:30.148202 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:30.148196    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5c9x\" (UniqueName: \"kubernetes.io/projected/24fb6a8c-100c-43d8-ae55-d1b6d6fcb437-kube-api-access-h5c9x\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:30.304978 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:30.304891    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f2aca4d-b6k2n"]
Oct 30 18:51:30.305742 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:30.305724    4525 cpu_manager.go:414] "containerMap: removing container" podUID="24fb6a8c-100c-43d8-ae55-d1b6d6fcb437" containerName="revoke-keys"
Oct 30 18:51:30.305797 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:30.305745    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24fb6a8c-100c-43d8-ae55-d1b6d6fcb437" containerName="revoke-keys"
Oct 30 18:51:30.305891 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:30.305880    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="24fb6a8c-100c-43d8-ae55-d1b6d6fcb437" containerName="revoke-keys"
Oct 30 18:51:30.309966 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:30.309950    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f2aca4d-b6k2n"
Oct 30 18:51:30.317152 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:30.317122    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f2aca4d-b6k2n"]
Oct 30 18:51:30.451187 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:30.451154    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13b3b9e4-817c-48c5-8803-c80d0cccb410-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f2aca4d-b6k2n\" (UID: \"13b3b9e4-817c-48c5-8803-c80d0cccb410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f2aca4d-b6k2n"
Oct 30 18:51:30.451383 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:30.451206    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2vbd\" (UniqueName: \"kubernetes.io/projected/13b3b9e4-817c-48c5-8803-c80d0cccb410-kube-api-access-d2vbd\") pod \"maas-api-revoke-sub-4f2aca4d-b6k2n\" (UID: \"13b3b9e4-817c-48c5-8803-c80d0cccb410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f2aca4d-b6k2n"
Oct 30 18:51:30.451383 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:30.451362    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13b3b9e4-817c-48c5-8803-c80d0cccb410-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f2aca4d-b6k2n\" (UID: \"13b3b9e4-817c-48c5-8803-c80d0cccb410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f2aca4d-b6k2n"
Oct 30 18:51:30.552622 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:30.552589    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13b3b9e4-817c-48c5-8803-c80d0cccb410-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f2aca4d-b6k2n\" (UID: \"13b3b9e4-817c-48c5-8803-c80d0cccb410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f2aca4d-b6k2n"
Oct 30 18:51:30.552822 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:30.552652    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2vbd\" (UniqueName: \"kubernetes.io/projected/13b3b9e4-817c-48c5-8803-c80d0cccb410-kube-api-access-d2vbd\") pod \"maas-api-revoke-sub-4f2aca4d-b6k2n\" (UID: \"13b3b9e4-817c-48c5-8803-c80d0cccb410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f2aca4d-b6k2n"
Oct 30 18:51:30.552867 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:30.552818    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13b3b9e4-817c-48c5-8803-c80d0cccb410-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f2aca4d-b6k2n\" (UID: \"13b3b9e4-817c-48c5-8803-c80d0cccb410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f2aca4d-b6k2n"
Oct 30 18:51:30.553261 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:30.553239    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13b3b9e4-817c-48c5-8803-c80d0cccb410-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f2aca4d-b6k2n\" (UID: \"13b3b9e4-817c-48c5-8803-c80d0cccb410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f2aca4d-b6k2n"
Oct 30 18:51:30.555369 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:30.555311    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13b3b9e4-817c-48c5-8803-c80d0cccb410-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f2aca4d-b6k2n\" (UID: \"13b3b9e4-817c-48c5-8803-c80d0cccb410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f2aca4d-b6k2n"
Oct 30 18:51:30.560767 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:30.560744    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2vbd\" (UniqueName: \"kubernetes.io/projected/13b3b9e4-817c-48c5-8803-c80d0cccb410-kube-api-access-d2vbd\") pod \"maas-api-revoke-sub-4f2aca4d-b6k2n\" (UID: \"13b3b9e4-817c-48c5-8803-c80d0cccb410\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f2aca4d-b6k2n"
Oct 30 18:51:30.629566 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:30.629534    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f2aca4d-b6k2n"
Oct 30 18:51:30.732387 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:30.732302    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32a67598-29l4p" event={"ID":"24fb6a8c-100c-43d8-ae55-d1b6d6fcb437","Type":"ContainerDied","Data":"c2917023e54242dae2084a46a540c4838a8404b1201d55371f6a80bdaf170b34"}
Oct 30 18:51:30.732387 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:30.732365    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2917023e54242dae2084a46a540c4838a8404b1201d55371f6a80bdaf170b34"
Oct 30 18:51:30.733458 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:30.733436    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32a67598-29l4p"
Oct 30 18:51:30.908011 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:30.907986    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c25bd9-7mwc6"
Oct 30 18:51:30.909946 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:51:30.909921    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13b3b9e4_817c_48c5_8803_c80d0cccb410.slice/crio-ef1b90ee92e05e191862680bf43b84f6f6aff3ad613d1a1157eabbaa3273f625 WatchSource:0}: Error finding container ef1b90ee92e05e191862680bf43b84f6f6aff3ad613d1a1157eabbaa3273f625: Status 404 returned error can't find the container with id ef1b90ee92e05e191862680bf43b84f6f6aff3ad613d1a1157eabbaa3273f625
Oct 30 18:51:30.910091 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:30.910064    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f2aca4d-b6k2n"]
Oct 30 18:51:31.058771 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:31.058739    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97a1b489-5957-4e3c-93fd-866399427ee0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97a1b489-5957-4e3c-93fd-866399427ee0-maas-api-xks-ca\") pod \"97a1b489-5957-4e3c-93fd-866399427ee0\" (UID: \"97a1b489-5957-4e3c-93fd-866399427ee0\") "
Oct 30 18:51:31.059030 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:31.059004    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97a1b489-5957-4e3c-93fd-866399427ee0-kube-api-access-vk5lj\" (UniqueName: \"kubernetes.io/projected/97a1b489-5957-4e3c-93fd-866399427ee0-kube-api-access-vk5lj\") pod \"97a1b489-5957-4e3c-93fd-866399427ee0\" (UID: \"97a1b489-5957-4e3c-93fd-866399427ee0\") "
Oct 30 18:51:31.059194 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:31.059060    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97a1b489-5957-4e3c-93fd-866399427ee0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97a1b489-5957-4e3c-93fd-866399427ee0-maas-api-openshift-ca\") pod \"97a1b489-5957-4e3c-93fd-866399427ee0\" (UID: \"97a1b489-5957-4e3c-93fd-866399427ee0\") "
Oct 30 18:51:31.059662 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:31.059637    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97a1b489-5957-4e3c-93fd-866399427ee0-maas-api-openshift-ca" pod "97a1b489-5957-4e3c-93fd-866399427ee0" (UID: "97a1b489-5957-4e3c-93fd-866399427ee0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:51:31.061110 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:31.061076    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97a1b489-5957-4e3c-93fd-866399427ee0-maas-api-xks-ca" pod "97a1b489-5957-4e3c-93fd-866399427ee0" (UID: "97a1b489-5957-4e3c-93fd-866399427ee0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:51:31.061312 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:31.061287    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97a1b489-5957-4e3c-93fd-866399427ee0-kube-api-access-vk5lj" pod "97a1b489-5957-4e3c-93fd-866399427ee0" (UID: "97a1b489-5957-4e3c-93fd-866399427ee0"). InnerVolumeSpecName "kube-api-access-vk5lj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:51:31.160299 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:31.160217    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vk5lj\" (UniqueName: \"kubernetes.io/projected/97a1b489-5957-4e3c-93fd-866399427ee0-kube-api-access-vk5lj\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:31.160299 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:31.160247    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97a1b489-5957-4e3c-93fd-866399427ee0-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:31.160299 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:31.160257    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97a1b489-5957-4e3c-93fd-866399427ee0-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:31.740705 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:31.740649    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c25bd9-7mwc6"
Oct 30 18:51:31.740705 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:31.740649    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c25bd9-7mwc6" event={"ID":"97a1b489-5957-4e3c-93fd-866399427ee0","Type":"ContainerDied","Data":"f4cfde6171d0b94d7a4bdc09ad9a5dfbaf68b4fde54937b17ffa1269b6371759"}
Oct 30 18:51:31.741254 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:31.740738    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4cfde6171d0b94d7a4bdc09ad9a5dfbaf68b4fde54937b17ffa1269b6371759"
Oct 30 18:51:31.742219 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:31.742197    4525 generic.go:356] "Generic (PLEG): container finished" podID="13b3b9e4-817c-48c5-8803-c80d0cccb410" containerID="05a1bb9035bdbe67bbd29f0414f8aa603d1bb96d7cd7f6c7f3ff17ba68224865" exitCode=0
Oct 30 18:51:31.742369 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:31.742270    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f2aca4d-b6k2n" event={"ID":"13b3b9e4-817c-48c5-8803-c80d0cccb410","Type":"ContainerDied","Data":"05a1bb9035bdbe67bbd29f0414f8aa603d1bb96d7cd7f6c7f3ff17ba68224865"}
Oct 30 18:51:31.742369 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:31.742333    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f2aca4d-b6k2n" event={"ID":"13b3b9e4-817c-48c5-8803-c80d0cccb410","Type":"ContainerStarted","Data":"ef1b90ee92e05e191862680bf43b84f6f6aff3ad613d1a1157eabbaa3273f625"}
Oct 30 18:51:32.902967 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:32.902942    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f2aca4d-b6k2n"
Oct 30 18:51:32.982017 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:32.981990    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13b3b9e4-817c-48c5-8803-c80d0cccb410-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13b3b9e4-817c-48c5-8803-c80d0cccb410-maas-api-xks-ca\") pod \"13b3b9e4-817c-48c5-8803-c80d0cccb410\" (UID: \"13b3b9e4-817c-48c5-8803-c80d0cccb410\") "
Oct 30 18:51:32.982209 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:32.982057    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13b3b9e4-817c-48c5-8803-c80d0cccb410-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13b3b9e4-817c-48c5-8803-c80d0cccb410-maas-api-openshift-ca\") pod \"13b3b9e4-817c-48c5-8803-c80d0cccb410\" (UID: \"13b3b9e4-817c-48c5-8803-c80d0cccb410\") "
Oct 30 18:51:32.982209 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:32.982103    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13b3b9e4-817c-48c5-8803-c80d0cccb410-kube-api-access-d2vbd\" (UniqueName: \"kubernetes.io/projected/13b3b9e4-817c-48c5-8803-c80d0cccb410-kube-api-access-d2vbd\") pod \"13b3b9e4-817c-48c5-8803-c80d0cccb410\" (UID: \"13b3b9e4-817c-48c5-8803-c80d0cccb410\") "
Oct 30 18:51:32.982481 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:32.982456    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13b3b9e4-817c-48c5-8803-c80d0cccb410-maas-api-openshift-ca" pod "13b3b9e4-817c-48c5-8803-c80d0cccb410" (UID: "13b3b9e4-817c-48c5-8803-c80d0cccb410"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:51:32.982653 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:32.982638    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13b3b9e4-817c-48c5-8803-c80d0cccb410-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:32.984340 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:32.984310    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13b3b9e4-817c-48c5-8803-c80d0cccb410-maas-api-xks-ca" pod "13b3b9e4-817c-48c5-8803-c80d0cccb410" (UID: "13b3b9e4-817c-48c5-8803-c80d0cccb410"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:51:32.984440 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:32.984342    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13b3b9e4-817c-48c5-8803-c80d0cccb410-kube-api-access-d2vbd" pod "13b3b9e4-817c-48c5-8803-c80d0cccb410" (UID: "13b3b9e4-817c-48c5-8803-c80d0cccb410"). InnerVolumeSpecName "kube-api-access-d2vbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:51:33.083660 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:33.083586    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13b3b9e4-817c-48c5-8803-c80d0cccb410-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:33.083660 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:33.083623    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2vbd\" (UniqueName: \"kubernetes.io/projected/13b3b9e4-817c-48c5-8803-c80d0cccb410-kube-api-access-d2vbd\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:33.761692 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:33.760111    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f2aca4d-b6k2n" event={"ID":"13b3b9e4-817c-48c5-8803-c80d0cccb410","Type":"ContainerDied","Data":"ef1b90ee92e05e191862680bf43b84f6f6aff3ad613d1a1157eabbaa3273f625"}
Oct 30 18:51:33.761692 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:33.760173    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef1b90ee92e05e191862680bf43b84f6f6aff3ad613d1a1157eabbaa3273f625"
Oct 30 18:51:33.761692 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:33.760265    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f2aca4d-b6k2n"
Oct 30 18:51:36.385420 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:36.385380    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2721a29-vtccj"]
Oct 30 18:51:36.385932 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:36.385916    4525 cpu_manager.go:414] "containerMap: removing container" podUID="97a1b489-5957-4e3c-93fd-866399427ee0" containerName="revoke-keys"
Oct 30 18:51:36.385972 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:36.385933    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97a1b489-5957-4e3c-93fd-866399427ee0" containerName="revoke-keys"
Oct 30 18:51:36.385972 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:36.385957    4525 cpu_manager.go:414] "containerMap: removing container" podUID="13b3b9e4-817c-48c5-8803-c80d0cccb410" containerName="revoke-keys"
Oct 30 18:51:36.385972 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:36.385963    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13b3b9e4-817c-48c5-8803-c80d0cccb410" containerName="revoke-keys"
Oct 30 18:51:36.386057 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:36.386051    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="97a1b489-5957-4e3c-93fd-866399427ee0" containerName="revoke-keys"
Oct 30 18:51:36.386105 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:36.386062    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="13b3b9e4-817c-48c5-8803-c80d0cccb410" containerName="revoke-keys"
Oct 30 18:51:36.389503 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:36.389488    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2721a29-vtccj"
Oct 30 18:51:36.394875 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:36.393512    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94hcn\""
Oct 30 18:51:36.394875 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:36.393574    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:51:36.397870 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:36.397839    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2721a29-vtccj"]
Oct 30 18:51:36.532205 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:36.532166    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mz6n9\" (UniqueName: \"kubernetes.io/projected/af853a58-3659-4146-85bc-b81f55b5e39a-kube-api-access-mz6n9\") pod \"maas-api-revoke-sub-e2721a29-vtccj\" (UID: \"af853a58-3659-4146-85bc-b81f55b5e39a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2721a29-vtccj"
Oct 30 18:51:36.532360 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:36.532233    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af853a58-3659-4146-85bc-b81f55b5e39a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2721a29-vtccj\" (UID: \"af853a58-3659-4146-85bc-b81f55b5e39a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2721a29-vtccj"
Oct 30 18:51:36.532422 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:36.532390    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af853a58-3659-4146-85bc-b81f55b5e39a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2721a29-vtccj\" (UID: \"af853a58-3659-4146-85bc-b81f55b5e39a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2721a29-vtccj"
Oct 30 18:51:36.633504 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:36.633466    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af853a58-3659-4146-85bc-b81f55b5e39a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2721a29-vtccj\" (UID: \"af853a58-3659-4146-85bc-b81f55b5e39a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2721a29-vtccj"
Oct 30 18:51:36.633711 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:36.633579    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mz6n9\" (UniqueName: \"kubernetes.io/projected/af853a58-3659-4146-85bc-b81f55b5e39a-kube-api-access-mz6n9\") pod \"maas-api-revoke-sub-e2721a29-vtccj\" (UID: \"af853a58-3659-4146-85bc-b81f55b5e39a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2721a29-vtccj"
Oct 30 18:51:36.633711 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:36.633645    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af853a58-3659-4146-85bc-b81f55b5e39a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2721a29-vtccj\" (UID: \"af853a58-3659-4146-85bc-b81f55b5e39a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2721a29-vtccj"
Oct 30 18:51:36.634385 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:36.634350    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af853a58-3659-4146-85bc-b81f55b5e39a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2721a29-vtccj\" (UID: \"af853a58-3659-4146-85bc-b81f55b5e39a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2721a29-vtccj"
Oct 30 18:51:36.636622 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:36.636557    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af853a58-3659-4146-85bc-b81f55b5e39a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2721a29-vtccj\" (UID: \"af853a58-3659-4146-85bc-b81f55b5e39a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2721a29-vtccj"
Oct 30 18:51:36.641491 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:36.641471    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mz6n9\" (UniqueName: \"kubernetes.io/projected/af853a58-3659-4146-85bc-b81f55b5e39a-kube-api-access-mz6n9\") pod \"maas-api-revoke-sub-e2721a29-vtccj\" (UID: \"af853a58-3659-4146-85bc-b81f55b5e39a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2721a29-vtccj"
Oct 30 18:51:36.709868 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:36.709838    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2721a29-vtccj"
Oct 30 18:51:36.991651 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:51:36.991621    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf853a58_3659_4146_85bc_b81f55b5e39a.slice/crio-2debd0fbde923ce069042b52af1a5bcb64a10a632045243ec68699a8baeeb758 WatchSource:0}: Error finding container 2debd0fbde923ce069042b52af1a5bcb64a10a632045243ec68699a8baeeb758: Status 404 returned error can't find the container with id 2debd0fbde923ce069042b52af1a5bcb64a10a632045243ec68699a8baeeb758
Oct 30 18:51:36.992141 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:36.992078    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2721a29-vtccj"]
Oct 30 18:51:37.823346 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:37.823303    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2721a29-vtccj" event={"ID":"af853a58-3659-4146-85bc-b81f55b5e39a","Type":"ContainerStarted","Data":"4fcdcafd6023518035a6f2fbbb6c76d543f529bed3c425ea730d0af9d5ff9e18"}
Oct 30 18:51:37.823901 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:37.823376    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2721a29-vtccj" event={"ID":"af853a58-3659-4146-85bc-b81f55b5e39a","Type":"ContainerStarted","Data":"2debd0fbde923ce069042b52af1a5bcb64a10a632045243ec68699a8baeeb758"}
Oct 30 18:51:37.847341 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:37.847286    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2721a29-vtccj" podStartSLOduration=1.847270644 podStartE2EDuration="1.847270644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:51:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:51:37.84400398 +0000 UTC m=+920.469673944" watchObservedRunningTime="2026-10-30 18:51:37.847270644 +0000 UTC m=+920.472940608"
Oct 30 18:51:38.523818 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:38.523779    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-bc6ff8476-tfpc7"]
Oct 30 18:51:38.844120 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:38.841958    4525 generic.go:356] "Generic (PLEG): container finished" podID="af853a58-3659-4146-85bc-b81f55b5e39a" containerID="4fcdcafd6023518035a6f2fbbb6c76d543f529bed3c425ea730d0af9d5ff9e18" exitCode=0
Oct 30 18:51:38.844120 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:38.842214    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2721a29-vtccj" event={"ID":"af853a58-3659-4146-85bc-b81f55b5e39a","Type":"ContainerDied","Data":"4fcdcafd6023518035a6f2fbbb6c76d543f529bed3c425ea730d0af9d5ff9e18"}
Oct 30 18:51:40.635662 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:40.634778    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw" podUID="ac6852fd-01be-4d69-8c9b-0c591e50b889" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:51:40.802662 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:40.780180    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj" podUID="84c99c49-1d60-4176-bda4-cad29c7d801c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 18:51:40.802662 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:40.782671    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr" podUID="82bbbf85-ed93-48d2-9f17-49ac1f31c0a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:51:41.002192 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:41.000836    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts" podUID="be6d4969-285b-4da0-883e-4847f2cbf3af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:51:41.142706 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:41.136176    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2721a29-vtccj"
Oct 30 18:51:41.194704 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:41.194269    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af853a58-3659-4146-85bc-b81f55b5e39a-kube-api-access-mz6n9\" (UniqueName: \"kubernetes.io/projected/af853a58-3659-4146-85bc-b81f55b5e39a-kube-api-access-mz6n9\") pod \"af853a58-3659-4146-85bc-b81f55b5e39a\" (UID: \"af853a58-3659-4146-85bc-b81f55b5e39a\") "
Oct 30 18:51:41.194704 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:41.194343    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af853a58-3659-4146-85bc-b81f55b5e39a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af853a58-3659-4146-85bc-b81f55b5e39a-maas-api-xks-ca\") pod \"af853a58-3659-4146-85bc-b81f55b5e39a\" (UID: \"af853a58-3659-4146-85bc-b81f55b5e39a\") "
Oct 30 18:51:41.194704 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:41.194494    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af853a58-3659-4146-85bc-b81f55b5e39a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af853a58-3659-4146-85bc-b81f55b5e39a-maas-api-openshift-ca\") pod \"af853a58-3659-4146-85bc-b81f55b5e39a\" (UID: \"af853a58-3659-4146-85bc-b81f55b5e39a\") "
Oct 30 18:51:41.216708 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:41.214934    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af853a58-3659-4146-85bc-b81f55b5e39a-maas-api-openshift-ca" pod "af853a58-3659-4146-85bc-b81f55b5e39a" (UID: "af853a58-3659-4146-85bc-b81f55b5e39a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:51:41.263708 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:41.262966    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af853a58-3659-4146-85bc-b81f55b5e39a-maas-api-xks-ca" pod "af853a58-3659-4146-85bc-b81f55b5e39a" (UID: "af853a58-3659-4146-85bc-b81f55b5e39a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:51:41.263708 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:41.263056    4525 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv" podUID="595230e3-55f3-43eb-9a7b-6d89250af9b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:51:41.343705 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:41.343649    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af853a58-3659-4146-85bc-b81f55b5e39a-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:41.343915 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:41.343712    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af853a58-3659-4146-85bc-b81f55b5e39a-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:41.353963 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:41.353907    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af853a58-3659-4146-85bc-b81f55b5e39a-kube-api-access-mz6n9" pod "af853a58-3659-4146-85bc-b81f55b5e39a" (UID: "af853a58-3659-4146-85bc-b81f55b5e39a"). InnerVolumeSpecName "kube-api-access-mz6n9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:51:41.450734 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:41.450573    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mz6n9\" (UniqueName: \"kubernetes.io/projected/af853a58-3659-4146-85bc-b81f55b5e39a-kube-api-access-mz6n9\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:41.553531 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:41.551766    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t" podUID="424fca14-6d8a-4663-86ec-0c91165f34ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:51:41.648708 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:41.638023    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw" podUID="ac6852fd-01be-4d69-8c9b-0c591e50b889" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:51:41.648708 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:41.639380    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw"
Oct 30 18:51:41.712496 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:41.711113    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb" podUID="12ebc8b9-c794-47b5-b87b-a63c4bd993e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:51:41.735061 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:41.735012    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv" podUID="595230e3-55f3-43eb-9a7b-6d89250af9b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:51:41.795754 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:41.795702    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr" podUID="82bbbf85-ed93-48d2-9f17-49ac1f31c0a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:51:41.796378 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:41.796345    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj" podUID="84c99c49-1d60-4176-bda4-cad29c7d801c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 18:51:41.796867 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:41.796848    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr"
Oct 30 18:51:41.797045 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:41.797031    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj"
Oct 30 18:51:42.016629 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:42.016529    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts" podUID="be6d4969-285b-4da0-883e-4847f2cbf3af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:51:42.029713 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:42.021358    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts"
Oct 30 18:51:42.601080 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:42.600805    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t" podUID="424fca14-6d8a-4663-86ec-0c91165f34ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:51:42.617965 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:42.617931    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t"
Oct 30 18:51:42.878039 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:42.857159    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw" podUID="ac6852fd-01be-4d69-8c9b-0c591e50b889" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:51:42.926772 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:42.919080    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr" podUID="82bbbf85-ed93-48d2-9f17-49ac1f31c0a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 30 18:51:42.926772 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:42.919448    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv" podUID="595230e3-55f3-43eb-9a7b-6d89250af9b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 18:51:42.926772 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:42.920073    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv"
Oct 30 18:51:42.947735 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:42.929000    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t"
Oct 30 18:51:42.947735 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:42.929457    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj" podUID="84c99c49-1d60-4176-bda4-cad29c7d801c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 18:51:42.985648 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:42.981920    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2721a29-vtccj"
Oct 30 18:51:43.004796 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:43.004749    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2721a29-vtccj" event={"ID":"af853a58-3659-4146-85bc-b81f55b5e39a","Type":"ContainerDied","Data":"2debd0fbde923ce069042b52af1a5bcb64a10a632045243ec68699a8baeeb758"}
Oct 30 18:51:43.005060 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:43.005036    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2debd0fbde923ce069042b52af1a5bcb64a10a632045243ec68699a8baeeb758"
Oct 30 18:51:43.021839 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:42.895300    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb" podUID="12ebc8b9-c794-47b5-b87b-a63c4bd993e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 18:51:43.022806 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:43.022781    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb"
Oct 30 18:51:43.024555 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:43.024527    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts" podUID="be6d4969-285b-4da0-883e-4847f2cbf3af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:51:43.256801 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:43.240070    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv"
Oct 30 18:51:43.429841 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:43.429810    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw"
Oct 30 18:51:43.430171 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:43.430152    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb"
Oct 30 18:51:43.483079 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:43.483043    4525 patch_prober.go:29] interesting pod/e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 18:51:43.508761 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:43.483349    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp" podUID="99aac9ad-b8dc-46cf-ac3f-69a814014bd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:51:43.524716 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:43.524600    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-bc6ff8476-tfpc7" podUID="1da1c465-0d66-484c-a6de-393667a61a86" containerName="payload-processing" containerID="cri-o://f7c6bcb80dc8f9bd0242a227b294b8e9306d02698646af27293dde8b2e86b656" gracePeriod=25
Oct 30 18:51:43.685455 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:43.675390    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts"
Oct 30 18:51:43.685455 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:43.675445    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr"
Oct 30 18:51:43.685455 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:43.675465    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-7njf5"]
Oct 30 18:51:43.685455 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:43.676194    4525 cpu_manager.go:414] "containerMap: removing container" podUID="af853a58-3659-4146-85bc-b81f55b5e39a" containerName="revoke-keys"
Oct 30 18:51:43.685455 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:43.676210    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af853a58-3659-4146-85bc-b81f55b5e39a" containerName="revoke-keys"
Oct 30 18:51:43.685455 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:43.676357    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="af853a58-3659-4146-85bc-b81f55b5e39a" containerName="revoke-keys"
Oct 30 18:51:43.698316 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:43.697004    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-7njf5"]
Oct 30 18:51:43.698316 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:43.697080    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj"
Oct 30 18:51:43.698316 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:43.697845    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-7njf5"
Oct 30 18:51:43.732420 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:43.729368    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d90f627-6806-4883-823a-1bb50c0afcf6-istio-envoy\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-7njf5\" (UID: \"0d90f627-6806-4883-823a-1bb50c0afcf6\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-7njf5"
Oct 30 18:51:43.732420 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:43.729453    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d90f627-6806-4883-823a-1bb50c0afcf6-credential-socket\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-7njf5\" (UID: \"0d90f627-6806-4883-823a-1bb50c0afcf6\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-7njf5"
Oct 30 18:51:43.732420 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:43.729490    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d90f627-6806-4883-823a-1bb50c0afcf6-istio-data\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-7njf5\" (UID: \"0d90f627-6806-4883-823a-1bb50c0afcf6\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-7njf5"
Oct 30 18:51:43.732420 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:43.729518    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d90f627-6806-4883-823a-1bb50c0afcf6-istio-podinfo\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-7njf5\" (UID: \"0d90f627-6806-4883-823a-1bb50c0afcf6\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-7njf5"
Oct 30 18:51:43.732420 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:43.729602    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d90f627-6806-4883-823a-1bb50c0afcf6-workload-certs\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-7njf5\" (UID: \"0d90f627-6806-4883-823a-1bb50c0afcf6\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-7njf5"
Oct 30 18:51:43.732420 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:43.729665    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d90f627-6806-4883-823a-1bb50c0afcf6-istio-token\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-7njf5\" (UID: \"0d90f627-6806-4883-823a-1bb50c0afcf6\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-7njf5"
Oct 30 18:51:43.732420 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:43.729736    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d90f627-6806-4883-823a-1bb50c0afcf6-istiod-ca-cert\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-7njf5\" (UID: \"0d90f627-6806-4883-823a-1bb50c0afcf6\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-7njf5"
Oct 30 18:51:43.732420 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:43.729795    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d90f627-6806-4883-823a-1bb50c0afcf6-workload-socket\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-7njf5\" (UID: \"0d90f627-6806-4883-823a-1bb50c0afcf6\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-7njf5"
Oct 30 18:51:43.732420 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:43.729854    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9n26x\" (UniqueName: \"kubernetes.io/projected/0d90f627-6806-4883-823a-1bb50c0afcf6-kube-api-access-9n26x\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-7njf5\" (UID: \"0d90f627-6806-4883-823a-1bb50c0afcf6\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-7njf5"
Oct 30 18:51:43.848064 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:43.831434    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d90f627-6806-4883-823a-1bb50c0afcf6-istio-envoy\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-7njf5\" (UID: \"0d90f627-6806-4883-823a-1bb50c0afcf6\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-7njf5"
Oct 30 18:51:43.848064 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:43.831507    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d90f627-6806-4883-823a-1bb50c0afcf6-credential-socket\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-7njf5\" (UID: \"0d90f627-6806-4883-823a-1bb50c0afcf6\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-7njf5"
Oct 30 18:51:43.848064 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:43.831546    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d90f627-6806-4883-823a-1bb50c0afcf6-istio-data\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-7njf5\" (UID: \"0d90f627-6806-4883-823a-1bb50c0afcf6\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-7njf5"
Oct 30 18:51:43.848064 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:43.831571    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d90f627-6806-4883-823a-1bb50c0afcf6-istio-podinfo\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-7njf5\" (UID: \"0d90f627-6806-4883-823a-1bb50c0afcf6\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-7njf5"
Oct 30 18:51:43.848064 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:43.831762    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d90f627-6806-4883-823a-1bb50c0afcf6-workload-certs\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-7njf5\" (UID: \"0d90f627-6806-4883-823a-1bb50c0afcf6\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-7njf5"
Oct 30 18:51:43.848064 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:43.831823    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d90f627-6806-4883-823a-1bb50c0afcf6-istio-token\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-7njf5\" (UID: \"0d90f627-6806-4883-823a-1bb50c0afcf6\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-7njf5"
Oct 30 18:51:43.848064 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:43.831881    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d90f627-6806-4883-823a-1bb50c0afcf6-istiod-ca-cert\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-7njf5\" (UID: \"0d90f627-6806-4883-823a-1bb50c0afcf6\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-7njf5"
Oct 30 18:51:43.848064 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:43.831941    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d90f627-6806-4883-823a-1bb50c0afcf6-workload-socket\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-7njf5\" (UID: \"0d90f627-6806-4883-823a-1bb50c0afcf6\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-7njf5"
Oct 30 18:51:43.848064 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:43.831992    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n26x\" (UniqueName: \"kubernetes.io/projected/0d90f627-6806-4883-823a-1bb50c0afcf6-kube-api-access-9n26x\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-7njf5\" (UID: \"0d90f627-6806-4883-823a-1bb50c0afcf6\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-7njf5"
Oct 30 18:51:43.848064 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:43.833633    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d90f627-6806-4883-823a-1bb50c0afcf6-workload-certs\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-7njf5\" (UID: \"0d90f627-6806-4883-823a-1bb50c0afcf6\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-7njf5"
Oct 30 18:51:43.848064 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:43.834720    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d90f627-6806-4883-823a-1bb50c0afcf6-istiod-ca-cert\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-7njf5\" (UID: \"0d90f627-6806-4883-823a-1bb50c0afcf6\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-7njf5"
Oct 30 18:51:43.848064 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:43.835231    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d90f627-6806-4883-823a-1bb50c0afcf6-workload-socket\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-7njf5\" (UID: \"0d90f627-6806-4883-823a-1bb50c0afcf6\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-7njf5"
Oct 30 18:51:43.848064 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:43.835532    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d90f627-6806-4883-823a-1bb50c0afcf6-istio-data\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-7njf5\" (UID: \"0d90f627-6806-4883-823a-1bb50c0afcf6\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-7njf5"
Oct 30 18:51:43.848064 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:43.839380    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d90f627-6806-4883-823a-1bb50c0afcf6-credential-socket\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-7njf5\" (UID: \"0d90f627-6806-4883-823a-1bb50c0afcf6\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-7njf5"
Oct 30 18:51:43.855803 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:43.851981    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d90f627-6806-4883-823a-1bb50c0afcf6-istio-envoy\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-7njf5\" (UID: \"0d90f627-6806-4883-823a-1bb50c0afcf6\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-7njf5"
Oct 30 18:51:43.858596 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:43.858541    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d90f627-6806-4883-823a-1bb50c0afcf6-istio-podinfo\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-7njf5\" (UID: \"0d90f627-6806-4883-823a-1bb50c0afcf6\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-7njf5"
Oct 30 18:51:43.859526 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:43.859474    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9n26x\" (UniqueName: \"kubernetes.io/projected/0d90f627-6806-4883-823a-1bb50c0afcf6-kube-api-access-9n26x\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-7njf5\" (UID: \"0d90f627-6806-4883-823a-1bb50c0afcf6\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-7njf5"
Oct 30 18:51:43.860481 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:43.860434    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d90f627-6806-4883-823a-1bb50c0afcf6-istio-token\") pod \"e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-7njf5\" (UID: \"0d90f627-6806-4883-823a-1bb50c0afcf6\") " pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-7njf5"
Oct 30 18:51:44.002564 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:44.002528    4525 generic.go:356] "Generic (PLEG): container finished" podID="1da1c465-0d66-484c-a6de-393667a61a86" containerID="f7c6bcb80dc8f9bd0242a227b294b8e9306d02698646af27293dde8b2e86b656" exitCode=0
Oct 30 18:51:44.004455 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:44.003261    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-bc6ff8476-tfpc7" event={"ID":"1da1c465-0d66-484c-a6de-393667a61a86","Type":"ContainerDied","Data":"f7c6bcb80dc8f9bd0242a227b294b8e9306d02698646af27293dde8b2e86b656"}
Oct 30 18:51:44.111581 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:44.111341    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-7njf5"
Oct 30 18:51:44.120722 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:44.120386    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-bc6ff8476-tfpc7"
Oct 30 18:51:44.138508 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:44.138450    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1da1c465-0d66-484c-a6de-393667a61a86-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1da1c465-0d66-484c-a6de-393667a61a86-provider-credentials\") pod \"1da1c465-0d66-484c-a6de-393667a61a86\" (UID: \"1da1c465-0d66-484c-a6de-393667a61a86\") "
Oct 30 18:51:44.138704 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:44.138511    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1da1c465-0d66-484c-a6de-393667a61a86-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1da1c465-0d66-484c-a6de-393667a61a86-plugins-config-volume\") pod \"1da1c465-0d66-484c-a6de-393667a61a86\" (UID: \"1da1c465-0d66-484c-a6de-393667a61a86\") "
Oct 30 18:51:44.138704 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:44.138625    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1da1c465-0d66-484c-a6de-393667a61a86-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1da1c465-0d66-484c-a6de-393667a61a86-routing-overlay-volume\") pod \"1da1c465-0d66-484c-a6de-393667a61a86\" (UID: \"1da1c465-0d66-484c-a6de-393667a61a86\") "
Oct 30 18:51:44.138943 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:44.138713    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1da1c465-0d66-484c-a6de-393667a61a86-kube-api-access-2svdg\" (UniqueName: \"kubernetes.io/projected/1da1c465-0d66-484c-a6de-393667a61a86-kube-api-access-2svdg\") pod \"1da1c465-0d66-484c-a6de-393667a61a86\" (UID: \"1da1c465-0d66-484c-a6de-393667a61a86\") "
Oct 30 18:51:44.139413 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:44.139366    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1da1c465-0d66-484c-a6de-393667a61a86-plugins-config-volume" pod "1da1c465-0d66-484c-a6de-393667a61a86" (UID: "1da1c465-0d66-484c-a6de-393667a61a86"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:51:44.154741 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:44.141754    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1da1c465-0d66-484c-a6de-393667a61a86-routing-overlay-volume" pod "1da1c465-0d66-484c-a6de-393667a61a86" (UID: "1da1c465-0d66-484c-a6de-393667a61a86"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:51:44.154741 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:44.146948    4525 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1da1c465-0d66-484c-a6de-393667a61a86-plugins-config-volume\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:44.154741 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:44.146973    4525 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1da1c465-0d66-484c-a6de-393667a61a86-routing-overlay-volume\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:44.154741 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:44.148631    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1da1c465-0d66-484c-a6de-393667a61a86-provider-credentials" pod "1da1c465-0d66-484c-a6de-393667a61a86" (UID: "1da1c465-0d66-484c-a6de-393667a61a86"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:51:44.187477 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:44.187432    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1da1c465-0d66-484c-a6de-393667a61a86-kube-api-access-2svdg" pod "1da1c465-0d66-484c-a6de-393667a61a86" (UID: "1da1c465-0d66-484c-a6de-393667a61a86"). InnerVolumeSpecName "kube-api-access-2svdg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:51:44.258882 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:44.258820    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2svdg\" (UniqueName: \"kubernetes.io/projected/1da1c465-0d66-484c-a6de-393667a61a86-kube-api-access-2svdg\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:44.258882 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:44.258855    4525 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1da1c465-0d66-484c-a6de-393667a61a86-provider-credentials\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:44.360659 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:44.360623    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e57547ca-tcbxq"]
Oct 30 18:51:44.361517 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:44.361496    4525 cpu_manager.go:414] "containerMap: removing container" podUID="1da1c465-0d66-484c-a6de-393667a61a86" containerName="payload-processing"
Oct 30 18:51:44.361584 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:44.361521    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1da1c465-0d66-484c-a6de-393667a61a86" containerName="payload-processing"
Oct 30 18:51:44.361766 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:44.361753    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="1da1c465-0d66-484c-a6de-393667a61a86" containerName="payload-processing"
Oct 30 18:51:44.366648 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:44.366629    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e57547ca-tcbxq"
Oct 30 18:51:44.370225 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:44.370187    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:51:44.370996 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:44.370358    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94hcn\""
Oct 30 18:51:44.375149 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:44.375123    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e57547ca-tcbxq"]
Oct 30 18:51:44.461465 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:44.461437    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caf20f32-8f96-49cf-91a0-e4e651f6f48a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e57547ca-tcbxq\" (UID: \"caf20f32-8f96-49cf-91a0-e4e651f6f48a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e57547ca-tcbxq"
Oct 30 18:51:44.461635 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:44.461501    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pwsd\" (UniqueName: \"kubernetes.io/projected/caf20f32-8f96-49cf-91a0-e4e651f6f48a-kube-api-access-9pwsd\") pod \"maas-api-revoke-sub-e57547ca-tcbxq\" (UID: \"caf20f32-8f96-49cf-91a0-e4e651f6f48a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e57547ca-tcbxq"
Oct 30 18:51:44.461717 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:44.461647    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caf20f32-8f96-49cf-91a0-e4e651f6f48a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e57547ca-tcbxq\" (UID: \"caf20f32-8f96-49cf-91a0-e4e651f6f48a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e57547ca-tcbxq"
Oct 30 18:51:44.473543 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:44.473517    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-7njf5"]
Oct 30 18:51:44.473888 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:51:44.473866    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d90f627_6806_4883_823a_1bb50c0afcf6.slice/crio-17321660de501aba10aec54d2d70ca83bebd1214243df70b2af8b44839c435bb WatchSource:0}: Error finding container 17321660de501aba10aec54d2d70ca83bebd1214243df70b2af8b44839c435bb: Status 404 returned error can't find the container with id 17321660de501aba10aec54d2d70ca83bebd1214243df70b2af8b44839c435bb
Oct 30 18:51:44.476410 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:44.476383    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:51:44.476499 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:44.476444    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:51:44.476499 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:44.476475    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:51:44.563173 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:44.563146    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caf20f32-8f96-49cf-91a0-e4e651f6f48a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e57547ca-tcbxq\" (UID: \"caf20f32-8f96-49cf-91a0-e4e651f6f48a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e57547ca-tcbxq"
Oct 30 18:51:44.563311 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:44.563220    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pwsd\" (UniqueName: \"kubernetes.io/projected/caf20f32-8f96-49cf-91a0-e4e651f6f48a-kube-api-access-9pwsd\") pod \"maas-api-revoke-sub-e57547ca-tcbxq\" (UID: \"caf20f32-8f96-49cf-91a0-e4e651f6f48a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e57547ca-tcbxq"
Oct 30 18:51:44.563311 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:44.563277    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caf20f32-8f96-49cf-91a0-e4e651f6f48a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e57547ca-tcbxq\" (UID: \"caf20f32-8f96-49cf-91a0-e4e651f6f48a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e57547ca-tcbxq"
Oct 30 18:51:44.563743 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:44.563722    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caf20f32-8f96-49cf-91a0-e4e651f6f48a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e57547ca-tcbxq\" (UID: \"caf20f32-8f96-49cf-91a0-e4e651f6f48a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e57547ca-tcbxq"
Oct 30 18:51:44.565845 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:44.565816    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caf20f32-8f96-49cf-91a0-e4e651f6f48a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e57547ca-tcbxq\" (UID: \"caf20f32-8f96-49cf-91a0-e4e651f6f48a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e57547ca-tcbxq"
Oct 30 18:51:44.571225 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:44.571177    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pwsd\" (UniqueName: \"kubernetes.io/projected/caf20f32-8f96-49cf-91a0-e4e651f6f48a-kube-api-access-9pwsd\") pod \"maas-api-revoke-sub-e57547ca-tcbxq\" (UID: \"caf20f32-8f96-49cf-91a0-e4e651f6f48a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e57547ca-tcbxq"
Oct 30 18:51:44.684965 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:44.684894    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e57547ca-tcbxq"
Oct 30 18:51:44.975349 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:51:44.975313    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaf20f32_8f96_49cf_91a0_e4e651f6f48a.slice/crio-f59c6a3ff653842cc2fe12866d48e77b29a98d1a9a638a933dd2dd22dc42b506 WatchSource:0}: Error finding container f59c6a3ff653842cc2fe12866d48e77b29a98d1a9a638a933dd2dd22dc42b506: Status 404 returned error can't find the container with id f59c6a3ff653842cc2fe12866d48e77b29a98d1a9a638a933dd2dd22dc42b506
Oct 30 18:51:44.975633 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:44.975613    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e57547ca-tcbxq"]
Oct 30 18:51:45.013140 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:45.013108    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-bc6ff8476-tfpc7"
Oct 30 18:51:45.013519 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:45.013137    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-bc6ff8476-tfpc7" event={"ID":"1da1c465-0d66-484c-a6de-393667a61a86","Type":"ContainerDied","Data":"6108819b5c440dc0591bdb2ab3b1fa0ee6d4284615f880984b20c70f635b925b"}
Oct 30 18:51:45.013519 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:45.013201    4525 scope.go:122] "RemoveContainer" containerID="f7c6bcb80dc8f9bd0242a227b294b8e9306d02698646af27293dde8b2e86b656"
Oct 30 18:51:45.014554 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:45.014535    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e57547ca-tcbxq" event={"ID":"caf20f32-8f96-49cf-91a0-e4e651f6f48a","Type":"ContainerStarted","Data":"f59c6a3ff653842cc2fe12866d48e77b29a98d1a9a638a933dd2dd22dc42b506"}
Oct 30 18:51:45.017113 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:45.016572    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-7njf5" event={"ID":"0d90f627-6806-4883-823a-1bb50c0afcf6","Type":"ContainerStarted","Data":"3e0aac395c4b21cc200a5b5506f7ba373b8df4734ba1676f18f6df7daf201d49"}
Oct 30 18:51:45.017113 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:45.016617    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-7njf5" event={"ID":"0d90f627-6806-4883-823a-1bb50c0afcf6","Type":"ContainerStarted","Data":"17321660de501aba10aec54d2d70ca83bebd1214243df70b2af8b44839c435bb"}
Oct 30 18:51:45.045508 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:45.045459    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-7njf5" podStartSLOduration=3.045448114 podStartE2EDuration="3.045448114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:51:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:51:45.042064114 +0000 UTC m=+927.667734074" watchObservedRunningTime="2026-10-30 18:51:45.045448114 +0000 UTC m=+927.671118074"
Oct 30 18:51:45.061896 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:45.061862    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-bc6ff8476-tfpc7"]
Oct 30 18:51:45.065570 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:45.065551    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-bc6ff8476-tfpc7"]
Oct 30 18:51:45.112362 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:45.112338    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-7njf5"
Oct 30 18:51:45.113627 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:45.113598    4525 patch_prober.go:29] interesting pod/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-7njf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 30 18:51:45.113737 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:45.113660    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-7njf5" podUID="0d90f627-6806-4883-823a-1bb50c0afcf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 30 18:51:45.544325 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:45.544292    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1da1c465-0d66-484c-a6de-393667a61a86" path="/var/lib/kubelet/pods/1da1c465-0d66-484c-a6de-393667a61a86/volumes"
Oct 30 18:51:46.025611 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:46.025579    4525 generic.go:356] "Generic (PLEG): container finished" podID="caf20f32-8f96-49cf-91a0-e4e651f6f48a" containerID="937da3f5011bdb4c24f0ddbfe6538d366643f24333e4ca37bced02eb87a0ff74" exitCode=0
Oct 30 18:51:46.026079 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:46.025654    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e57547ca-tcbxq" event={"ID":"caf20f32-8f96-49cf-91a0-e4e651f6f48a","Type":"ContainerDied","Data":"937da3f5011bdb4c24f0ddbfe6538d366643f24333e4ca37bced02eb87a0ff74"}
Oct 30 18:51:46.112783 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:46.112751    4525 patch_prober.go:29] interesting pod/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-7njf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 30 18:51:46.112959 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:46.112807    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-7njf5" podUID="0d90f627-6806-4883-823a-1bb50c0afcf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 30 18:51:47.112728 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:47.112690    4525 patch_prober.go:29] interesting pod/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-7njf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 30 18:51:47.113133 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:47.112753    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-7njf5" podUID="0d90f627-6806-4883-823a-1bb50c0afcf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 30 18:51:47.230349 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:47.230323    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e57547ca-tcbxq"
Oct 30 18:51:47.324048 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:47.323960    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/caf20f32-8f96-49cf-91a0-e4e651f6f48a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caf20f32-8f96-49cf-91a0-e4e651f6f48a-maas-api-xks-ca\") pod \"caf20f32-8f96-49cf-91a0-e4e651f6f48a\" (UID: \"caf20f32-8f96-49cf-91a0-e4e651f6f48a\") "
Oct 30 18:51:47.324048 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:47.324012    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/caf20f32-8f96-49cf-91a0-e4e651f6f48a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caf20f32-8f96-49cf-91a0-e4e651f6f48a-maas-api-openshift-ca\") pod \"caf20f32-8f96-49cf-91a0-e4e651f6f48a\" (UID: \"caf20f32-8f96-49cf-91a0-e4e651f6f48a\") "
Oct 30 18:51:47.324280 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:47.324126    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/caf20f32-8f96-49cf-91a0-e4e651f6f48a-kube-api-access-9pwsd\" (UniqueName: \"kubernetes.io/projected/caf20f32-8f96-49cf-91a0-e4e651f6f48a-kube-api-access-9pwsd\") pod \"caf20f32-8f96-49cf-91a0-e4e651f6f48a\" (UID: \"caf20f32-8f96-49cf-91a0-e4e651f6f48a\") "
Oct 30 18:51:47.324407 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:47.324381    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/caf20f32-8f96-49cf-91a0-e4e651f6f48a-maas-api-openshift-ca" pod "caf20f32-8f96-49cf-91a0-e4e651f6f48a" (UID: "caf20f32-8f96-49cf-91a0-e4e651f6f48a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:51:47.324718 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:47.324697    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caf20f32-8f96-49cf-91a0-e4e651f6f48a-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:47.326465 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:47.326438    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/caf20f32-8f96-49cf-91a0-e4e651f6f48a-maas-api-xks-ca" pod "caf20f32-8f96-49cf-91a0-e4e651f6f48a" (UID: "caf20f32-8f96-49cf-91a0-e4e651f6f48a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:51:47.326564 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:47.326441    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caf20f32-8f96-49cf-91a0-e4e651f6f48a-kube-api-access-9pwsd" pod "caf20f32-8f96-49cf-91a0-e4e651f6f48a" (UID: "caf20f32-8f96-49cf-91a0-e4e651f6f48a"). InnerVolumeSpecName "kube-api-access-9pwsd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:51:47.426128 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:51:47.426069    4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d49l" containerName="extractor"
Oct 30 18:51:47.426287 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:47.426159    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caf20f32-8f96-49cf-91a0-e4e651f6f48a-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:47.426287 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:47.426186    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pwsd\" (UniqueName: \"kubernetes.io/projected/caf20f32-8f96-49cf-91a0-e4e651f6f48a-kube-api-access-9pwsd\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:48.041671 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:48.041643    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e57547ca-tcbxq"
Oct 30 18:51:48.041883 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:48.041632    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e57547ca-tcbxq" event={"ID":"caf20f32-8f96-49cf-91a0-e4e651f6f48a","Type":"ContainerDied","Data":"f59c6a3ff653842cc2fe12866d48e77b29a98d1a9a638a933dd2dd22dc42b506"}
Oct 30 18:51:48.041883 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:48.041774    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f59c6a3ff653842cc2fe12866d48e77b29a98d1a9a638a933dd2dd22dc42b506"
Oct 30 18:51:48.112220 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:48.112179    4525 patch_prober.go:29] interesting pod/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-7njf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 30 18:51:48.112419 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:48.112240    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-7njf5" podUID="0d90f627-6806-4883-823a-1bb50c0afcf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 30 18:51:49.115707 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:49.113080    4525 patch_prober.go:29] interesting pod/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-7njf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 30 18:51:49.115707 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:49.113147    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-7njf5" podUID="0d90f627-6806-4883-823a-1bb50c0afcf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 30 18:51:49.138115 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:49.138068    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50aa509a-nx7xh"]
Oct 30 18:51:49.138641 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:49.138623    4525 cpu_manager.go:414] "containerMap: removing container" podUID="caf20f32-8f96-49cf-91a0-e4e651f6f48a" containerName="revoke-keys"
Oct 30 18:51:49.138768 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:49.138644    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="caf20f32-8f96-49cf-91a0-e4e651f6f48a" containerName="revoke-keys"
Oct 30 18:51:49.138863 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:49.138848    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="caf20f32-8f96-49cf-91a0-e4e651f6f48a" containerName="revoke-keys"
Oct 30 18:51:49.142883 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:49.142865    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50aa509a-nx7xh"
Oct 30 18:51:49.150153 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:49.150135    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94hcn\""
Oct 30 18:51:49.150254 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:49.150174    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:51:49.155778 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:49.155758    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50aa509a-nx7xh"]
Oct 30 18:51:49.251653 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:49.251614    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/103a5ab0-3191-4bc8-9d7f-9bae95dd1fd2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50aa509a-nx7xh\" (UID: \"103a5ab0-3191-4bc8-9d7f-9bae95dd1fd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50aa509a-nx7xh"
Oct 30 18:51:49.251808 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:49.251766    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/103a5ab0-3191-4bc8-9d7f-9bae95dd1fd2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50aa509a-nx7xh\" (UID: \"103a5ab0-3191-4bc8-9d7f-9bae95dd1fd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50aa509a-nx7xh"
Oct 30 18:51:49.251808 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:49.251804    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5fnb\" (UniqueName: \"kubernetes.io/projected/103a5ab0-3191-4bc8-9d7f-9bae95dd1fd2-kube-api-access-c5fnb\") pod \"maas-api-revoke-sub-50aa509a-nx7xh\" (UID: \"103a5ab0-3191-4bc8-9d7f-9bae95dd1fd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50aa509a-nx7xh"
Oct 30 18:51:49.352959 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:49.352925    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/103a5ab0-3191-4bc8-9d7f-9bae95dd1fd2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50aa509a-nx7xh\" (UID: \"103a5ab0-3191-4bc8-9d7f-9bae95dd1fd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50aa509a-nx7xh"
Oct 30 18:51:49.353128 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:49.353016    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/103a5ab0-3191-4bc8-9d7f-9bae95dd1fd2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50aa509a-nx7xh\" (UID: \"103a5ab0-3191-4bc8-9d7f-9bae95dd1fd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50aa509a-nx7xh"
Oct 30 18:51:49.353128 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:49.353057    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5fnb\" (UniqueName: \"kubernetes.io/projected/103a5ab0-3191-4bc8-9d7f-9bae95dd1fd2-kube-api-access-c5fnb\") pod \"maas-api-revoke-sub-50aa509a-nx7xh\" (UID: \"103a5ab0-3191-4bc8-9d7f-9bae95dd1fd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50aa509a-nx7xh"
Oct 30 18:51:49.353578 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:49.353547    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/103a5ab0-3191-4bc8-9d7f-9bae95dd1fd2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50aa509a-nx7xh\" (UID: \"103a5ab0-3191-4bc8-9d7f-9bae95dd1fd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50aa509a-nx7xh"
Oct 30 18:51:49.355511 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:49.355487    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/103a5ab0-3191-4bc8-9d7f-9bae95dd1fd2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50aa509a-nx7xh\" (UID: \"103a5ab0-3191-4bc8-9d7f-9bae95dd1fd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50aa509a-nx7xh"
Oct 30 18:51:49.360601 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:49.360582    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5fnb\" (UniqueName: \"kubernetes.io/projected/103a5ab0-3191-4bc8-9d7f-9bae95dd1fd2-kube-api-access-c5fnb\") pod \"maas-api-revoke-sub-50aa509a-nx7xh\" (UID: \"103a5ab0-3191-4bc8-9d7f-9bae95dd1fd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50aa509a-nx7xh"
Oct 30 18:51:49.457851 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:49.457773    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50aa509a-nx7xh"
Oct 30 18:51:49.768368 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:49.768334    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50aa509a-nx7xh"]
Oct 30 18:51:50.063065 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:50.062635    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50aa509a-nx7xh" event={"ID":"103a5ab0-3191-4bc8-9d7f-9bae95dd1fd2","Type":"ContainerStarted","Data":"401b68f9ec6454e8064c7b15bccd65461246729f287cedb95b17b3d56f2b65fa"}
Oct 30 18:51:50.110013 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:50.109982    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31dbf6f3-xhmf2"]
Oct 30 18:51:50.112086 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:50.112056    4525 patch_prober.go:29] interesting pod/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-7njf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 30 18:51:50.112315 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:50.112292    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-7njf5" podUID="0d90f627-6806-4883-823a-1bb50c0afcf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 30 18:51:50.115151 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:50.115128    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31dbf6f3-xhmf2"
Oct 30 18:51:50.129719 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:50.129042    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31dbf6f3-xhmf2"]
Oct 30 18:51:50.270589 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:50.270555    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9tls\" (UniqueName: \"kubernetes.io/projected/f42ffdee-b882-41e9-ae44-c8a3ba102853-kube-api-access-q9tls\") pod \"maas-api-revoke-sub-31dbf6f3-xhmf2\" (UID: \"f42ffdee-b882-41e9-ae44-c8a3ba102853\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31dbf6f3-xhmf2"
Oct 30 18:51:50.270760 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:50.270619    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f42ffdee-b882-41e9-ae44-c8a3ba102853-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31dbf6f3-xhmf2\" (UID: \"f42ffdee-b882-41e9-ae44-c8a3ba102853\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31dbf6f3-xhmf2"
Oct 30 18:51:50.270760 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:50.270735    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f42ffdee-b882-41e9-ae44-c8a3ba102853-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31dbf6f3-xhmf2\" (UID: \"f42ffdee-b882-41e9-ae44-c8a3ba102853\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31dbf6f3-xhmf2"
Oct 30 18:51:50.371971 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:50.371866    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9tls\" (UniqueName: \"kubernetes.io/projected/f42ffdee-b882-41e9-ae44-c8a3ba102853-kube-api-access-q9tls\") pod \"maas-api-revoke-sub-31dbf6f3-xhmf2\" (UID: \"f42ffdee-b882-41e9-ae44-c8a3ba102853\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31dbf6f3-xhmf2"
Oct 30 18:51:50.371971 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:50.371957    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f42ffdee-b882-41e9-ae44-c8a3ba102853-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31dbf6f3-xhmf2\" (UID: \"f42ffdee-b882-41e9-ae44-c8a3ba102853\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31dbf6f3-xhmf2"
Oct 30 18:51:50.372237 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:50.372015    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f42ffdee-b882-41e9-ae44-c8a3ba102853-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31dbf6f3-xhmf2\" (UID: \"f42ffdee-b882-41e9-ae44-c8a3ba102853\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31dbf6f3-xhmf2"
Oct 30 18:51:50.372670 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:50.372648    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f42ffdee-b882-41e9-ae44-c8a3ba102853-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31dbf6f3-xhmf2\" (UID: \"f42ffdee-b882-41e9-ae44-c8a3ba102853\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31dbf6f3-xhmf2"
Oct 30 18:51:50.374548 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:50.374523    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f42ffdee-b882-41e9-ae44-c8a3ba102853-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31dbf6f3-xhmf2\" (UID: \"f42ffdee-b882-41e9-ae44-c8a3ba102853\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31dbf6f3-xhmf2"
Oct 30 18:51:50.403637 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:50.403607    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9tls\" (UniqueName: \"kubernetes.io/projected/f42ffdee-b882-41e9-ae44-c8a3ba102853-kube-api-access-q9tls\") pod \"maas-api-revoke-sub-31dbf6f3-xhmf2\" (UID: \"f42ffdee-b882-41e9-ae44-c8a3ba102853\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31dbf6f3-xhmf2"
Oct 30 18:51:50.487611 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:50.487579    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31dbf6f3-xhmf2"
Oct 30 18:51:50.777149 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:51:50.777110    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf42ffdee_b882_41e9_ae44_c8a3ba102853.slice/crio-9d8dd70279483d9c569f94c71017eb513c48d0f16bdeb8361a5002854b52fc64 WatchSource:0}: Error finding container 9d8dd70279483d9c569f94c71017eb513c48d0f16bdeb8361a5002854b52fc64: Status 404 returned error can't find the container with id 9d8dd70279483d9c569f94c71017eb513c48d0f16bdeb8361a5002854b52fc64
Oct 30 18:51:50.778096 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:50.778062    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31dbf6f3-xhmf2"]
Oct 30 18:51:51.071146 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:51.071052    4525 generic.go:356] "Generic (PLEG): container finished" podID="103a5ab0-3191-4bc8-9d7f-9bae95dd1fd2" containerID="b829be9404f0f5a9c47703cd5a6783f89ae17af44b029a4b2fb3fc5b8c8d68b7" exitCode=0
Oct 30 18:51:51.071296 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:51.071144    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50aa509a-nx7xh" event={"ID":"103a5ab0-3191-4bc8-9d7f-9bae95dd1fd2","Type":"ContainerDied","Data":"b829be9404f0f5a9c47703cd5a6783f89ae17af44b029a4b2fb3fc5b8c8d68b7"}
Oct 30 18:51:51.072874 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:51.072848    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31dbf6f3-xhmf2" event={"ID":"f42ffdee-b882-41e9-ae44-c8a3ba102853","Type":"ContainerStarted","Data":"9d8dd70279483d9c569f94c71017eb513c48d0f16bdeb8361a5002854b52fc64"}
Oct 30 18:51:51.113209 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:51.113177    4525 patch_prober.go:29] interesting pod/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-7njf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 30 18:51:51.113418 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:51.113242    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-7njf5" podUID="0d90f627-6806-4883-823a-1bb50c0afcf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 30 18:51:52.085244 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:52.085154    4525 generic.go:356] "Generic (PLEG): container finished" podID="f42ffdee-b882-41e9-ae44-c8a3ba102853" containerID="5334ff9564b6f59d2da80847c259f6e2f0a2470603789f84363617710c95a1c1" exitCode=0
Oct 30 18:51:52.085619 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:52.085248    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31dbf6f3-xhmf2" event={"ID":"f42ffdee-b882-41e9-ae44-c8a3ba102853","Type":"ContainerDied","Data":"5334ff9564b6f59d2da80847c259f6e2f0a2470603789f84363617710c95a1c1"}
Oct 30 18:51:52.138251 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:52.138220    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-7njf5"
Oct 30 18:51:52.139077 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:52.139055    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-7njf5"
Oct 30 18:51:52.140110 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:52.140092    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-7njf5"
Oct 30 18:51:52.256711 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:52.256664    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50aa509a-nx7xh"
Oct 30 18:51:52.397115 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:52.397086    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/103a5ab0-3191-4bc8-9d7f-9bae95dd1fd2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/103a5ab0-3191-4bc8-9d7f-9bae95dd1fd2-maas-api-xks-ca\") pod \"103a5ab0-3191-4bc8-9d7f-9bae95dd1fd2\" (UID: \"103a5ab0-3191-4bc8-9d7f-9bae95dd1fd2\") "
Oct 30 18:51:52.397259 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:52.397174    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/103a5ab0-3191-4bc8-9d7f-9bae95dd1fd2-kube-api-access-c5fnb\" (UniqueName: \"kubernetes.io/projected/103a5ab0-3191-4bc8-9d7f-9bae95dd1fd2-kube-api-access-c5fnb\") pod \"103a5ab0-3191-4bc8-9d7f-9bae95dd1fd2\" (UID: \"103a5ab0-3191-4bc8-9d7f-9bae95dd1fd2\") "
Oct 30 18:51:52.397259 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:52.397204    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/103a5ab0-3191-4bc8-9d7f-9bae95dd1fd2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/103a5ab0-3191-4bc8-9d7f-9bae95dd1fd2-maas-api-openshift-ca\") pod \"103a5ab0-3191-4bc8-9d7f-9bae95dd1fd2\" (UID: \"103a5ab0-3191-4bc8-9d7f-9bae95dd1fd2\") "
Oct 30 18:51:52.397635 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:52.397614    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/103a5ab0-3191-4bc8-9d7f-9bae95dd1fd2-maas-api-openshift-ca" pod "103a5ab0-3191-4bc8-9d7f-9bae95dd1fd2" (UID: "103a5ab0-3191-4bc8-9d7f-9bae95dd1fd2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:51:52.399381 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:52.399352    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/103a5ab0-3191-4bc8-9d7f-9bae95dd1fd2-kube-api-access-c5fnb" pod "103a5ab0-3191-4bc8-9d7f-9bae95dd1fd2" (UID: "103a5ab0-3191-4bc8-9d7f-9bae95dd1fd2"). InnerVolumeSpecName "kube-api-access-c5fnb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:51:52.399381 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:52.399381    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/103a5ab0-3191-4bc8-9d7f-9bae95dd1fd2-maas-api-xks-ca" pod "103a5ab0-3191-4bc8-9d7f-9bae95dd1fd2" (UID: "103a5ab0-3191-4bc8-9d7f-9bae95dd1fd2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:51:52.498941 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:52.498910    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/103a5ab0-3191-4bc8-9d7f-9bae95dd1fd2-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:52.498941 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:52.498937    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5fnb\" (UniqueName: \"kubernetes.io/projected/103a5ab0-3191-4bc8-9d7f-9bae95dd1fd2-kube-api-access-c5fnb\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:52.498941 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:52.498947    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/103a5ab0-3191-4bc8-9d7f-9bae95dd1fd2-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:53.093505 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:53.093475    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50aa509a-nx7xh"
Oct 30 18:51:53.094012 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:53.093519    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50aa509a-nx7xh" event={"ID":"103a5ab0-3191-4bc8-9d7f-9bae95dd1fd2","Type":"ContainerDied","Data":"401b68f9ec6454e8064c7b15bccd65461246729f287cedb95b17b3d56f2b65fa"}
Oct 30 18:51:53.094012 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:53.093568    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="401b68f9ec6454e8064c7b15bccd65461246729f287cedb95b17b3d56f2b65fa"
Oct 30 18:51:53.271788 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:53.271760    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31dbf6f3-xhmf2"
Oct 30 18:51:53.414408 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:53.414374    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f42ffdee-b882-41e9-ae44-c8a3ba102853-kube-api-access-q9tls\" (UniqueName: \"kubernetes.io/projected/f42ffdee-b882-41e9-ae44-c8a3ba102853-kube-api-access-q9tls\") pod \"f42ffdee-b882-41e9-ae44-c8a3ba102853\" (UID: \"f42ffdee-b882-41e9-ae44-c8a3ba102853\") "
Oct 30 18:51:53.414586 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:53.414439    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f42ffdee-b882-41e9-ae44-c8a3ba102853-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f42ffdee-b882-41e9-ae44-c8a3ba102853-maas-api-openshift-ca\") pod \"f42ffdee-b882-41e9-ae44-c8a3ba102853\" (UID: \"f42ffdee-b882-41e9-ae44-c8a3ba102853\") "
Oct 30 18:51:53.414638 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:53.414596    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f42ffdee-b882-41e9-ae44-c8a3ba102853-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f42ffdee-b882-41e9-ae44-c8a3ba102853-maas-api-xks-ca\") pod \"f42ffdee-b882-41e9-ae44-c8a3ba102853\" (UID: \"f42ffdee-b882-41e9-ae44-c8a3ba102853\") "
Oct 30 18:51:53.414847 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:53.414817    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f42ffdee-b882-41e9-ae44-c8a3ba102853-maas-api-openshift-ca" pod "f42ffdee-b882-41e9-ae44-c8a3ba102853" (UID: "f42ffdee-b882-41e9-ae44-c8a3ba102853"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:51:53.414973 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:53.414956    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f42ffdee-b882-41e9-ae44-c8a3ba102853-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:53.416771 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:53.416750    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f42ffdee-b882-41e9-ae44-c8a3ba102853-kube-api-access-q9tls" pod "f42ffdee-b882-41e9-ae44-c8a3ba102853" (UID: "f42ffdee-b882-41e9-ae44-c8a3ba102853"). InnerVolumeSpecName "kube-api-access-q9tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:51:53.416853 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:53.416818    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f42ffdee-b882-41e9-ae44-c8a3ba102853-maas-api-xks-ca" pod "f42ffdee-b882-41e9-ae44-c8a3ba102853" (UID: "f42ffdee-b882-41e9-ae44-c8a3ba102853"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:51:53.515982 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:53.515947    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f42ffdee-b882-41e9-ae44-c8a3ba102853-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:53.515982 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:53.515978    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9tls\" (UniqueName: \"kubernetes.io/projected/f42ffdee-b882-41e9-ae44-c8a3ba102853-kube-api-access-q9tls\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:54.102416 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:54.102383    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31dbf6f3-xhmf2"
Oct 30 18:51:54.102944 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:54.102413    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31dbf6f3-xhmf2" event={"ID":"f42ffdee-b882-41e9-ae44-c8a3ba102853","Type":"ContainerDied","Data":"9d8dd70279483d9c569f94c71017eb513c48d0f16bdeb8361a5002854b52fc64"}
Oct 30 18:51:54.102944 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:54.102461    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d8dd70279483d9c569f94c71017eb513c48d0f16bdeb8361a5002854b52fc64"
Oct 30 18:51:57.712707 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:57.710762    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj" podUID="84c99c49-1d60-4176-bda4-cad29c7d801c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 18:51:57.823880 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:57.823826    4525 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr" podUID="82bbbf85-ed93-48d2-9f17-49ac1f31c0a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:51:57.923838 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:57.923773    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr" podUID="82bbbf85-ed93-48d2-9f17-49ac1f31c0a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:51:57.938777 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:57.937755    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t" podUID="424fca14-6d8a-4663-86ec-0c91165f34ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 18:51:58.112827 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:58.112775    4525 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj" podUID="84c99c49-1d60-4176-bda4-cad29c7d801c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:51:58.139706 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:58.138774    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv" podUID="595230e3-55f3-43eb-9a7b-6d89250af9b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:51:58.427768 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:58.426781    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw" podUID="ac6852fd-01be-4d69-8c9b-0c591e50b889" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:51:58.455862 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:58.455812    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb" podUID="12ebc8b9-c794-47b5-b87b-a63c4bd993e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 18:51:58.456545 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:58.456132    4525 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw" podUID="ac6852fd-01be-4d69-8c9b-0c591e50b889" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:51:58.640798 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:58.640753    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts" podUID="be6d4969-285b-4da0-883e-4847f2cbf3af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:51:58.712761 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:58.711661    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj" podUID="84c99c49-1d60-4176-bda4-cad29c7d801c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:51:58.712761 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:58.712221    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj"
Oct 30 18:51:58.935235 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:58.935192    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr" podUID="82bbbf85-ed93-48d2-9f17-49ac1f31c0a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 30 18:51:58.942710 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:58.936171    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr"
Oct 30 18:51:58.952696 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:58.943777    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t" podUID="424fca14-6d8a-4663-86ec-0c91165f34ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:51:58.965763 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:58.961162    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t"
Oct 30 18:51:59.187710 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:59.159014    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv" podUID="595230e3-55f3-43eb-9a7b-6d89250af9b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:51:59.208711 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:59.188715    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv"
Oct 30 18:51:59.510711 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:59.448590    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-h4qrf"]
Oct 30 18:51:59.510711 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:59.449373    4525 cpu_manager.go:414] "containerMap: removing container" podUID="f42ffdee-b882-41e9-ae44-c8a3ba102853" containerName="revoke-keys"
Oct 30 18:51:59.510711 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:59.449398    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f42ffdee-b882-41e9-ae44-c8a3ba102853" containerName="revoke-keys"
Oct 30 18:51:59.510711 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:59.449434    4525 cpu_manager.go:414] "containerMap: removing container" podUID="103a5ab0-3191-4bc8-9d7f-9bae95dd1fd2" containerName="revoke-keys"
Oct 30 18:51:59.510711 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:59.449443    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="103a5ab0-3191-4bc8-9d7f-9bae95dd1fd2" containerName="revoke-keys"
Oct 30 18:51:59.510711 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:59.449596    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="f42ffdee-b882-41e9-ae44-c8a3ba102853" containerName="revoke-keys"
Oct 30 18:51:59.510711 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:59.449614    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="103a5ab0-3191-4bc8-9d7f-9bae95dd1fd2" containerName="revoke-keys"
Oct 30 18:51:59.510711 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:59.458154    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb" podUID="12ebc8b9-c794-47b5-b87b-a63c4bd993e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 18:51:59.510711 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:59.485307    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw" podUID="ac6852fd-01be-4d69-8c9b-0c591e50b889" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 18:51:59.521972 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:59.521943    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb"
Oct 30 18:51:59.530120 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:59.530096    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw"
Oct 30 18:51:59.538713 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:59.525103    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-h4qrf"
Oct 30 18:51:59.684040 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:59.684001    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts" podUID="be6d4969-285b-4da0-883e-4847f2cbf3af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:51:59.731704 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:59.717405    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj" podUID="84c99c49-1d60-4176-bda4-cad29c7d801c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:51:59.743553 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:59.743519    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd500131-13db-47b6-a1a7-b377e950a539-credential-socket\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-h4qrf\" (UID: \"cd500131-13db-47b6-a1a7-b377e950a539\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-h4qrf"
Oct 30 18:51:59.817953 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:59.770364    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts"
Oct 30 18:51:59.841294 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:59.841254    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-h4qrf"]
Oct 30 18:51:59.923102 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:59.923064    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd500131-13db-47b6-a1a7-b377e950a539-workload-certs\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-h4qrf\" (UID: \"cd500131-13db-47b6-a1a7-b377e950a539\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-h4qrf"
Oct 30 18:51:59.923854 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:59.923825    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd500131-13db-47b6-a1a7-b377e950a539-istio-envoy\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-h4qrf\" (UID: \"cd500131-13db-47b6-a1a7-b377e950a539\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-h4qrf"
Oct 30 18:51:59.927779 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:59.927744    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd500131-13db-47b6-a1a7-b377e950a539-istio-data\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-h4qrf\" (UID: \"cd500131-13db-47b6-a1a7-b377e950a539\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-h4qrf"
Oct 30 18:51:59.931888 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:59.931865    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-md8vr\" (UniqueName: \"kubernetes.io/projected/cd500131-13db-47b6-a1a7-b377e950a539-kube-api-access-md8vr\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-h4qrf\" (UID: \"cd500131-13db-47b6-a1a7-b377e950a539\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-h4qrf"
Oct 30 18:51:59.994837 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:59.935745    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr" podUID="82bbbf85-ed93-48d2-9f17-49ac1f31c0a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 30 18:51:59.995517 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:51:59.961816    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t" podUID="424fca14-6d8a-4663-86ec-0c91165f34ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:52:00.022755 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:00.022719    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd500131-13db-47b6-a1a7-b377e950a539-istio-podinfo\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-h4qrf\" (UID: \"cd500131-13db-47b6-a1a7-b377e950a539\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-h4qrf"
Oct 30 18:52:00.023065 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:00.023042    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd500131-13db-47b6-a1a7-b377e950a539-workload-socket\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-h4qrf\" (UID: \"cd500131-13db-47b6-a1a7-b377e950a539\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-h4qrf"
Oct 30 18:52:00.023290 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:00.023272    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd500131-13db-47b6-a1a7-b377e950a539-istio-token\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-h4qrf\" (UID: \"cd500131-13db-47b6-a1a7-b377e950a539\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-h4qrf"
Oct 30 18:52:00.033927 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:00.033873    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd500131-13db-47b6-a1a7-b377e950a539-istiod-ca-cert\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-h4qrf\" (UID: \"cd500131-13db-47b6-a1a7-b377e950a539\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-h4qrf"
Oct 30 18:52:00.184378 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:00.184211    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd500131-13db-47b6-a1a7-b377e950a539-credential-socket\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-h4qrf\" (UID: \"cd500131-13db-47b6-a1a7-b377e950a539\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-h4qrf"
Oct 30 18:52:00.274701 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:00.272909    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd500131-13db-47b6-a1a7-b377e950a539-workload-certs\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-h4qrf\" (UID: \"cd500131-13db-47b6-a1a7-b377e950a539\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-h4qrf"
Oct 30 18:52:00.274701 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:00.272965    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd500131-13db-47b6-a1a7-b377e950a539-istio-envoy\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-h4qrf\" (UID: \"cd500131-13db-47b6-a1a7-b377e950a539\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-h4qrf"
Oct 30 18:52:00.274701 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:00.272997    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd500131-13db-47b6-a1a7-b377e950a539-istio-data\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-h4qrf\" (UID: \"cd500131-13db-47b6-a1a7-b377e950a539\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-h4qrf"
Oct 30 18:52:00.274701 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:00.273039    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-md8vr\" (UniqueName: \"kubernetes.io/projected/cd500131-13db-47b6-a1a7-b377e950a539-kube-api-access-md8vr\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-h4qrf\" (UID: \"cd500131-13db-47b6-a1a7-b377e950a539\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-h4qrf"
Oct 30 18:52:00.274701 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:00.273143    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd500131-13db-47b6-a1a7-b377e950a539-istio-podinfo\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-h4qrf\" (UID: \"cd500131-13db-47b6-a1a7-b377e950a539\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-h4qrf"
Oct 30 18:52:00.274701 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:00.273181    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd500131-13db-47b6-a1a7-b377e950a539-workload-socket\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-h4qrf\" (UID: \"cd500131-13db-47b6-a1a7-b377e950a539\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-h4qrf"
Oct 30 18:52:00.274701 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:00.273228    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd500131-13db-47b6-a1a7-b377e950a539-istio-token\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-h4qrf\" (UID: \"cd500131-13db-47b6-a1a7-b377e950a539\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-h4qrf"
Oct 30 18:52:00.274701 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:00.273294    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd500131-13db-47b6-a1a7-b377e950a539-istiod-ca-cert\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-h4qrf\" (UID: \"cd500131-13db-47b6-a1a7-b377e950a539\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-h4qrf"
Oct 30 18:52:00.275367 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:00.198941    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv" podUID="595230e3-55f3-43eb-9a7b-6d89250af9b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:52:00.276756 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:00.276729    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd500131-13db-47b6-a1a7-b377e950a539-workload-certs\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-h4qrf\" (UID: \"cd500131-13db-47b6-a1a7-b377e950a539\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-h4qrf"
Oct 30 18:52:00.276953 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:00.201730    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd500131-13db-47b6-a1a7-b377e950a539-credential-socket\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-h4qrf\" (UID: \"cd500131-13db-47b6-a1a7-b377e950a539\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-h4qrf"
Oct 30 18:52:00.292556 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:00.292524    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd500131-13db-47b6-a1a7-b377e950a539-istio-data\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-h4qrf\" (UID: \"cd500131-13db-47b6-a1a7-b377e950a539\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-h4qrf"
Oct 30 18:52:00.328731 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:00.316745    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd500131-13db-47b6-a1a7-b377e950a539-istiod-ca-cert\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-h4qrf\" (UID: \"cd500131-13db-47b6-a1a7-b377e950a539\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-h4qrf"
Oct 30 18:52:00.350415 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:00.348654    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts"
Oct 30 18:52:00.370242 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:00.370206    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd500131-13db-47b6-a1a7-b377e950a539-workload-socket\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-h4qrf\" (UID: \"cd500131-13db-47b6-a1a7-b377e950a539\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-h4qrf"
Oct 30 18:52:00.373573 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:00.373534    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-md8vr\" (UniqueName: \"kubernetes.io/projected/cd500131-13db-47b6-a1a7-b377e950a539-kube-api-access-md8vr\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-h4qrf\" (UID: \"cd500131-13db-47b6-a1a7-b377e950a539\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-h4qrf"
Oct 30 18:52:00.374667 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:00.374052    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd500131-13db-47b6-a1a7-b377e950a539-istio-envoy\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-h4qrf\" (UID: \"cd500131-13db-47b6-a1a7-b377e950a539\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-h4qrf"
Oct 30 18:52:00.374843 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:00.374135    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr"
Oct 30 18:52:00.375004 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:00.374974    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t"
Oct 30 18:52:00.375160 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:00.375146    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj"
Oct 30 18:52:00.441778 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:00.436993    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb"
Oct 30 18:52:00.459710 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:00.452909    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd500131-13db-47b6-a1a7-b377e950a539-istio-podinfo\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-h4qrf\" (UID: \"cd500131-13db-47b6-a1a7-b377e950a539\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-h4qrf"
Oct 30 18:52:00.460420 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:00.460393    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd500131-13db-47b6-a1a7-b377e950a539-istio-token\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-h4qrf\" (UID: \"cd500131-13db-47b6-a1a7-b377e950a539\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-h4qrf"
Oct 30 18:52:00.467735 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:00.466197    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw"
Oct 30 18:52:00.474832 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:00.474811    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv"
Oct 30 18:52:00.704388 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:00.704301    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-h4qrf"
Oct 30 18:52:01.132844 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:52:01.132816    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd500131_13db_47b6_a1a7_b377e950a539.slice/crio-2caf11bff9ead7b3d364b6494ca142d61f4cfbfa680532295cf07296a2cf601f WatchSource:0}: Error finding container 2caf11bff9ead7b3d364b6494ca142d61f4cfbfa680532295cf07296a2cf601f: Status 404 returned error can't find the container with id 2caf11bff9ead7b3d364b6494ca142d61f4cfbfa680532295cf07296a2cf601f
Oct 30 18:52:01.133394 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:01.133331    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-h4qrf"]
Oct 30 18:52:01.135214 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:01.135186    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:52:01.135286 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:01.135244    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:52:01.135286 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:01.135273    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:52:01.478949 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:01.478854    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-h4qrf" event={"ID":"cd500131-13db-47b6-a1a7-b377e950a539","Type":"ContainerStarted","Data":"43ebfce1803a281dc319f4fb9c1ed5af0ec0a5c21c5a2daf6f78e1adc22e8a74"}
Oct 30 18:52:01.478949 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:01.478908    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-h4qrf" event={"ID":"cd500131-13db-47b6-a1a7-b377e950a539","Type":"ContainerStarted","Data":"2caf11bff9ead7b3d364b6494ca142d61f4cfbfa680532295cf07296a2cf601f"}
Oct 30 18:52:01.501389 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:01.501337    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-h4qrf" podStartSLOduration=2.501323619 podStartE2EDuration="2.501323619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:51:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:01.497395617 +0000 UTC m=+944.123065580" watchObservedRunningTime="2026-10-30 18:52:01.501323619 +0000 UTC m=+944.126993582"
Oct 30 18:52:01.704971 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:01.704941    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-h4qrf"
Oct 30 18:52:01.786234 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:01.786151    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac06a57d-kxzjp"]
Oct 30 18:52:01.790743 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:01.790708    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac06a57d-kxzjp"
Oct 30 18:52:01.795074 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:01.794399    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:52:01.795295 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:01.795272    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94hcn\""
Oct 30 18:52:01.803140 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:01.802698    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac06a57d-kxzjp"]
Oct 30 18:52:01.902475 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:01.902446    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hd4f\" (UniqueName: \"kubernetes.io/projected/bc4bf99a-34e8-47f1-9380-9a0c9676423f-kube-api-access-6hd4f\") pod \"maas-api-revoke-sub-ac06a57d-kxzjp\" (UID: \"bc4bf99a-34e8-47f1-9380-9a0c9676423f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac06a57d-kxzjp"
Oct 30 18:52:01.902723 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:01.902671    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc4bf99a-34e8-47f1-9380-9a0c9676423f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac06a57d-kxzjp\" (UID: \"bc4bf99a-34e8-47f1-9380-9a0c9676423f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac06a57d-kxzjp"
Oct 30 18:52:01.902830 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:01.902810    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc4bf99a-34e8-47f1-9380-9a0c9676423f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac06a57d-kxzjp\" (UID: \"bc4bf99a-34e8-47f1-9380-9a0c9676423f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac06a57d-kxzjp"
Oct 30 18:52:02.003863 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:02.003815    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc4bf99a-34e8-47f1-9380-9a0c9676423f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac06a57d-kxzjp\" (UID: \"bc4bf99a-34e8-47f1-9380-9a0c9676423f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac06a57d-kxzjp"
Oct 30 18:52:02.003863 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:02.003877    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc4bf99a-34e8-47f1-9380-9a0c9676423f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac06a57d-kxzjp\" (UID: \"bc4bf99a-34e8-47f1-9380-9a0c9676423f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac06a57d-kxzjp"
Oct 30 18:52:02.004149 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:02.004003    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hd4f\" (UniqueName: \"kubernetes.io/projected/bc4bf99a-34e8-47f1-9380-9a0c9676423f-kube-api-access-6hd4f\") pod \"maas-api-revoke-sub-ac06a57d-kxzjp\" (UID: \"bc4bf99a-34e8-47f1-9380-9a0c9676423f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac06a57d-kxzjp"
Oct 30 18:52:02.004717 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:02.004658    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc4bf99a-34e8-47f1-9380-9a0c9676423f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac06a57d-kxzjp\" (UID: \"bc4bf99a-34e8-47f1-9380-9a0c9676423f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac06a57d-kxzjp"
Oct 30 18:52:02.006533 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:02.006509    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc4bf99a-34e8-47f1-9380-9a0c9676423f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac06a57d-kxzjp\" (UID: \"bc4bf99a-34e8-47f1-9380-9a0c9676423f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac06a57d-kxzjp"
Oct 30 18:52:02.013791 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:02.013767    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hd4f\" (UniqueName: \"kubernetes.io/projected/bc4bf99a-34e8-47f1-9380-9a0c9676423f-kube-api-access-6hd4f\") pod \"maas-api-revoke-sub-ac06a57d-kxzjp\" (UID: \"bc4bf99a-34e8-47f1-9380-9a0c9676423f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac06a57d-kxzjp"
Oct 30 18:52:02.111928 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:02.111897    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac06a57d-kxzjp"
Oct 30 18:52:02.421159 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:02.421128    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac06a57d-kxzjp"]
Oct 30 18:52:02.508887 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:02.508841    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac06a57d-kxzjp" event={"ID":"bc4bf99a-34e8-47f1-9380-9a0c9676423f","Type":"ContainerStarted","Data":"9c0c8ee20b1b2b3be85d9caabed7be0887cb54a3cb0a9384642cfc778af29fc3"}
Oct 30 18:52:02.710981 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:02.710893    4525 patch_prober.go:29] interesting pod/e2e-worker-w4-0458df-openshift-default-7c86d79f79-h4qrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 30 18:52:02.711533 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:02.711246    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-h4qrf" podUID="cd500131-13db-47b6-a1a7-b377e950a539" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 30 18:52:03.009313 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:03.009222    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f61c55b0-w4hbz"]
Oct 30 18:52:03.015610 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:03.015577    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f61c55b0-w4hbz"
Oct 30 18:52:03.016097 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:03.016071    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f61c55b0-w4hbz"]
Oct 30 18:52:03.024214 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:03.024184    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/558bb0e9-54ce-4848-92ca-1e65c3c05957-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f61c55b0-w4hbz\" (UID: \"558bb0e9-54ce-4848-92ca-1e65c3c05957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f61c55b0-w4hbz"
Oct 30 18:52:03.024363 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:03.024350    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnmw7\" (UniqueName: \"kubernetes.io/projected/558bb0e9-54ce-4848-92ca-1e65c3c05957-kube-api-access-lnmw7\") pod \"maas-api-revoke-sub-f61c55b0-w4hbz\" (UID: \"558bb0e9-54ce-4848-92ca-1e65c3c05957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f61c55b0-w4hbz"
Oct 30 18:52:03.024453 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:03.024377    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/558bb0e9-54ce-4848-92ca-1e65c3c05957-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f61c55b0-w4hbz\" (UID: \"558bb0e9-54ce-4848-92ca-1e65c3c05957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f61c55b0-w4hbz"
Oct 30 18:52:03.126346 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:03.125623    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/558bb0e9-54ce-4848-92ca-1e65c3c05957-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f61c55b0-w4hbz\" (UID: \"558bb0e9-54ce-4848-92ca-1e65c3c05957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f61c55b0-w4hbz"
Oct 30 18:52:03.126346 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:03.125788    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnmw7\" (UniqueName: \"kubernetes.io/projected/558bb0e9-54ce-4848-92ca-1e65c3c05957-kube-api-access-lnmw7\") pod \"maas-api-revoke-sub-f61c55b0-w4hbz\" (UID: \"558bb0e9-54ce-4848-92ca-1e65c3c05957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f61c55b0-w4hbz"
Oct 30 18:52:03.126346 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:03.125825    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/558bb0e9-54ce-4848-92ca-1e65c3c05957-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f61c55b0-w4hbz\" (UID: \"558bb0e9-54ce-4848-92ca-1e65c3c05957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f61c55b0-w4hbz"
Oct 30 18:52:03.127402 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:03.127234    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/558bb0e9-54ce-4848-92ca-1e65c3c05957-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f61c55b0-w4hbz\" (UID: \"558bb0e9-54ce-4848-92ca-1e65c3c05957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f61c55b0-w4hbz"
Oct 30 18:52:03.131529 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:03.129698    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/558bb0e9-54ce-4848-92ca-1e65c3c05957-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f61c55b0-w4hbz\" (UID: \"558bb0e9-54ce-4848-92ca-1e65c3c05957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f61c55b0-w4hbz"
Oct 30 18:52:03.135914 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:03.135889    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnmw7\" (UniqueName: \"kubernetes.io/projected/558bb0e9-54ce-4848-92ca-1e65c3c05957-kube-api-access-lnmw7\") pod \"maas-api-revoke-sub-f61c55b0-w4hbz\" (UID: \"558bb0e9-54ce-4848-92ca-1e65c3c05957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f61c55b0-w4hbz"
Oct 30 18:52:03.333838 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:03.333761    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f61c55b0-w4hbz"
Oct 30 18:52:03.528709 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:03.526188    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac06a57d-kxzjp" event={"ID":"bc4bf99a-34e8-47f1-9380-9a0c9676423f","Type":"ContainerStarted","Data":"87a7f5ae89f7261636d240758e6ce01b456dd637bea2ffb3bab8e614e20c5301"}
Oct 30 18:52:03.642315 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:03.642281    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f61c55b0-w4hbz"]
Oct 30 18:52:03.706168 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:03.706142    4525 patch_prober.go:29] interesting pod/e2e-worker-w4-0458df-openshift-default-7c86d79f79-h4qrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 30 18:52:03.706292 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:03.706191    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-h4qrf" podUID="cd500131-13db-47b6-a1a7-b377e950a539" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 30 18:52:04.030655 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:04.030626    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e29de711-qpch8"]
Oct 30 18:52:04.035964 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:04.035945    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e29de711-qpch8"
Oct 30 18:52:04.044643 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:04.044615    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e29de711-qpch8"]
Oct 30 18:52:04.048059 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:04.048032    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/070b5c5e-06a4-4ecd-a12e-37ddf440f097-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e29de711-qpch8\" (UID: \"070b5c5e-06a4-4ecd-a12e-37ddf440f097\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e29de711-qpch8"
Oct 30 18:52:04.048176 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:04.048122    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/070b5c5e-06a4-4ecd-a12e-37ddf440f097-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e29de711-qpch8\" (UID: \"070b5c5e-06a4-4ecd-a12e-37ddf440f097\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e29de711-qpch8"
Oct 30 18:52:04.048258 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:04.048181    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv8kd\" (UniqueName: \"kubernetes.io/projected/070b5c5e-06a4-4ecd-a12e-37ddf440f097-kube-api-access-gv8kd\") pod \"maas-api-revoke-sub-e29de711-qpch8\" (UID: \"070b5c5e-06a4-4ecd-a12e-37ddf440f097\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e29de711-qpch8"
Oct 30 18:52:04.148660 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:04.148619    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/070b5c5e-06a4-4ecd-a12e-37ddf440f097-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e29de711-qpch8\" (UID: \"070b5c5e-06a4-4ecd-a12e-37ddf440f097\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e29de711-qpch8"
Oct 30 18:52:04.148856 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:04.148746    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv8kd\" (UniqueName: \"kubernetes.io/projected/070b5c5e-06a4-4ecd-a12e-37ddf440f097-kube-api-access-gv8kd\") pod \"maas-api-revoke-sub-e29de711-qpch8\" (UID: \"070b5c5e-06a4-4ecd-a12e-37ddf440f097\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e29de711-qpch8"
Oct 30 18:52:04.148921 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:04.148860    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/070b5c5e-06a4-4ecd-a12e-37ddf440f097-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e29de711-qpch8\" (UID: \"070b5c5e-06a4-4ecd-a12e-37ddf440f097\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e29de711-qpch8"
Oct 30 18:52:04.149463 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:04.149438    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/070b5c5e-06a4-4ecd-a12e-37ddf440f097-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e29de711-qpch8\" (UID: \"070b5c5e-06a4-4ecd-a12e-37ddf440f097\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e29de711-qpch8"
Oct 30 18:52:04.152586 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:04.152562    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/070b5c5e-06a4-4ecd-a12e-37ddf440f097-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e29de711-qpch8\" (UID: \"070b5c5e-06a4-4ecd-a12e-37ddf440f097\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e29de711-qpch8"
Oct 30 18:52:04.156341 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:04.156318    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv8kd\" (UniqueName: \"kubernetes.io/projected/070b5c5e-06a4-4ecd-a12e-37ddf440f097-kube-api-access-gv8kd\") pod \"maas-api-revoke-sub-e29de711-qpch8\" (UID: \"070b5c5e-06a4-4ecd-a12e-37ddf440f097\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e29de711-qpch8"
Oct 30 18:52:04.355592 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:04.355561    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e29de711-qpch8"
Oct 30 18:52:04.552129 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:04.552090    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f61c55b0-w4hbz" event={"ID":"558bb0e9-54ce-4848-92ca-1e65c3c05957","Type":"ContainerStarted","Data":"7e6df2e9d8225b4274a90bbb6f50bcedc6c8514fd6426e951516ab0129717cde"}
Oct 30 18:52:04.552618 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:04.552589    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f61c55b0-w4hbz" event={"ID":"558bb0e9-54ce-4848-92ca-1e65c3c05957","Type":"ContainerStarted","Data":"b80daf6ebbe1fbb06a78d4f4e80448dbacd04332d923f8ed8184dff01ab03b31"}
Oct 30 18:52:04.556330 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:04.556211    4525 generic.go:356] "Generic (PLEG): container finished" podID="bc4bf99a-34e8-47f1-9380-9a0c9676423f" containerID="87a7f5ae89f7261636d240758e6ce01b456dd637bea2ffb3bab8e614e20c5301" exitCode=0
Oct 30 18:52:04.556330 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:04.556281    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac06a57d-kxzjp" event={"ID":"bc4bf99a-34e8-47f1-9380-9a0c9676423f","Type":"ContainerDied","Data":"87a7f5ae89f7261636d240758e6ce01b456dd637bea2ffb3bab8e614e20c5301"}
Oct 30 18:52:04.622767 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:04.621664    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f61c55b0-w4hbz" podStartSLOduration=2.62164196 podStartE2EDuration="2.62164196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:52:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:04.574023918 +0000 UTC m=+947.199693883" watchObservedRunningTime="2026-10-30 18:52:04.62164196 +0000 UTC m=+947.247311924"
Oct 30 18:52:04.713242 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:04.712834    4525 patch_prober.go:29] interesting pod/e2e-worker-w4-0458df-openshift-default-7c86d79f79-h4qrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 30 18:52:04.713242 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:04.712894    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-h4qrf" podUID="cd500131-13db-47b6-a1a7-b377e950a539" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 30 18:52:04.726726 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:04.721033    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e29de711-qpch8"]
Oct 30 18:52:04.813710 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:04.811457    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac06a57d-kxzjp"
Oct 30 18:52:04.961418 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:04.961386    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc4bf99a-34e8-47f1-9380-9a0c9676423f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc4bf99a-34e8-47f1-9380-9a0c9676423f-maas-api-openshift-ca\") pod \"bc4bf99a-34e8-47f1-9380-9a0c9676423f\" (UID: \"bc4bf99a-34e8-47f1-9380-9a0c9676423f\") "
Oct 30 18:52:04.961731 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:04.961717    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc4bf99a-34e8-47f1-9380-9a0c9676423f-kube-api-access-6hd4f\" (UniqueName: \"kubernetes.io/projected/bc4bf99a-34e8-47f1-9380-9a0c9676423f-kube-api-access-6hd4f\") pod \"bc4bf99a-34e8-47f1-9380-9a0c9676423f\" (UID: \"bc4bf99a-34e8-47f1-9380-9a0c9676423f\") "
Oct 30 18:52:04.961837 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:04.961825    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc4bf99a-34e8-47f1-9380-9a0c9676423f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc4bf99a-34e8-47f1-9380-9a0c9676423f-maas-api-xks-ca\") pod \"bc4bf99a-34e8-47f1-9380-9a0c9676423f\" (UID: \"bc4bf99a-34e8-47f1-9380-9a0c9676423f\") "
Oct 30 18:52:04.962130 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:04.961741    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc4bf99a-34e8-47f1-9380-9a0c9676423f-maas-api-openshift-ca" pod "bc4bf99a-34e8-47f1-9380-9a0c9676423f" (UID: "bc4bf99a-34e8-47f1-9380-9a0c9676423f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:52:04.962345 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:04.962331    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc4bf99a-34e8-47f1-9380-9a0c9676423f-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:04.963890 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:04.963868    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc4bf99a-34e8-47f1-9380-9a0c9676423f-kube-api-access-6hd4f" pod "bc4bf99a-34e8-47f1-9380-9a0c9676423f" (UID: "bc4bf99a-34e8-47f1-9380-9a0c9676423f"). InnerVolumeSpecName "kube-api-access-6hd4f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:52:04.963956 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:04.963908    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc4bf99a-34e8-47f1-9380-9a0c9676423f-maas-api-xks-ca" pod "bc4bf99a-34e8-47f1-9380-9a0c9676423f" (UID: "bc4bf99a-34e8-47f1-9380-9a0c9676423f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:52:05.063436 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:05.063402    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hd4f\" (UniqueName: \"kubernetes.io/projected/bc4bf99a-34e8-47f1-9380-9a0c9676423f-kube-api-access-6hd4f\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:05.063615 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:05.063604    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc4bf99a-34e8-47f1-9380-9a0c9676423f-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:05.564177 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:05.564109    4525 generic.go:356] "Generic (PLEG): container finished" podID="558bb0e9-54ce-4848-92ca-1e65c3c05957" containerID="7e6df2e9d8225b4274a90bbb6f50bcedc6c8514fd6426e951516ab0129717cde" exitCode=0
Oct 30 18:52:05.565092 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:05.564911    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f61c55b0-w4hbz" event={"ID":"558bb0e9-54ce-4848-92ca-1e65c3c05957","Type":"ContainerDied","Data":"7e6df2e9d8225b4274a90bbb6f50bcedc6c8514fd6426e951516ab0129717cde"}
Oct 30 18:52:05.566912 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:05.566355    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac06a57d-kxzjp" event={"ID":"bc4bf99a-34e8-47f1-9380-9a0c9676423f","Type":"ContainerDied","Data":"9c0c8ee20b1b2b3be85d9caabed7be0887cb54a3cb0a9384642cfc778af29fc3"}
Oct 30 18:52:05.566912 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:05.566392    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c0c8ee20b1b2b3be85d9caabed7be0887cb54a3cb0a9384642cfc778af29fc3"
Oct 30 18:52:05.566912 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:05.566433    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac06a57d-kxzjp"
Oct 30 18:52:05.568637 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:05.568563    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e29de711-qpch8" event={"ID":"070b5c5e-06a4-4ecd-a12e-37ddf440f097","Type":"ContainerStarted","Data":"d691dec3ecf13a08158a3f4f5de331d98b8b011daa40adb9fb2832333133cbd9"}
Oct 30 18:52:05.568637 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:05.568608    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e29de711-qpch8" event={"ID":"070b5c5e-06a4-4ecd-a12e-37ddf440f097","Type":"ContainerStarted","Data":"f185e78777916b13409f843f5c72e67f319852bbe5f383e1977c3544bad117cb"}
Oct 30 18:52:05.621077 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:05.621028    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e29de711-qpch8" podStartSLOduration=1.621015383 podStartE2EDuration="1.621015383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:52:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:05.618864951 +0000 UTC m=+948.244534922" watchObservedRunningTime="2026-10-30 18:52:05.621015383 +0000 UTC m=+948.246685346"
Oct 30 18:52:05.705474 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:05.705391    4525 patch_prober.go:29] interesting pod/e2e-worker-w4-0458df-openshift-default-7c86d79f79-h4qrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 30 18:52:05.705474 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:05.705443    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-h4qrf" podUID="cd500131-13db-47b6-a1a7-b377e950a539" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 30 18:52:06.602046 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:06.602014    4525 generic.go:356] "Generic (PLEG): container finished" podID="070b5c5e-06a4-4ecd-a12e-37ddf440f097" containerID="d691dec3ecf13a08158a3f4f5de331d98b8b011daa40adb9fb2832333133cbd9" exitCode=0
Oct 30 18:52:06.602461 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:06.602095    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e29de711-qpch8" event={"ID":"070b5c5e-06a4-4ecd-a12e-37ddf440f097","Type":"ContainerDied","Data":"d691dec3ecf13a08158a3f4f5de331d98b8b011daa40adb9fb2832333133cbd9"}
Oct 30 18:52:06.706508 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:06.706052    4525 patch_prober.go:29] interesting pod/e2e-worker-w4-0458df-openshift-default-7c86d79f79-h4qrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 30 18:52:06.706508 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:06.706124    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-h4qrf" podUID="cd500131-13db-47b6-a1a7-b377e950a539" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 30 18:52:06.810128 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:06.806772    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f61c55b0-w4hbz"
Oct 30 18:52:06.810128 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:06.808636    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/558bb0e9-54ce-4848-92ca-1e65c3c05957-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/558bb0e9-54ce-4848-92ca-1e65c3c05957-maas-api-xks-ca\") pod \"558bb0e9-54ce-4848-92ca-1e65c3c05957\" (UID: \"558bb0e9-54ce-4848-92ca-1e65c3c05957\") "
Oct 30 18:52:06.810128 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:06.808773    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/558bb0e9-54ce-4848-92ca-1e65c3c05957-kube-api-access-lnmw7\" (UniqueName: \"kubernetes.io/projected/558bb0e9-54ce-4848-92ca-1e65c3c05957-kube-api-access-lnmw7\") pod \"558bb0e9-54ce-4848-92ca-1e65c3c05957\" (UID: \"558bb0e9-54ce-4848-92ca-1e65c3c05957\") "
Oct 30 18:52:06.810128 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:06.808870    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/558bb0e9-54ce-4848-92ca-1e65c3c05957-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/558bb0e9-54ce-4848-92ca-1e65c3c05957-maas-api-openshift-ca\") pod \"558bb0e9-54ce-4848-92ca-1e65c3c05957\" (UID: \"558bb0e9-54ce-4848-92ca-1e65c3c05957\") "
Oct 30 18:52:06.810128 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:06.809800    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/558bb0e9-54ce-4848-92ca-1e65c3c05957-maas-api-openshift-ca" pod "558bb0e9-54ce-4848-92ca-1e65c3c05957" (UID: "558bb0e9-54ce-4848-92ca-1e65c3c05957"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:52:06.816077 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:06.816039    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/558bb0e9-54ce-4848-92ca-1e65c3c05957-maas-api-xks-ca" pod "558bb0e9-54ce-4848-92ca-1e65c3c05957" (UID: "558bb0e9-54ce-4848-92ca-1e65c3c05957"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:52:06.816267 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:06.816227    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/558bb0e9-54ce-4848-92ca-1e65c3c05957-kube-api-access-lnmw7" pod "558bb0e9-54ce-4848-92ca-1e65c3c05957" (UID: "558bb0e9-54ce-4848-92ca-1e65c3c05957"). InnerVolumeSpecName "kube-api-access-lnmw7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:52:06.910934 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:06.910898    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnmw7\" (UniqueName: \"kubernetes.io/projected/558bb0e9-54ce-4848-92ca-1e65c3c05957-kube-api-access-lnmw7\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:06.910934 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:06.910930    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/558bb0e9-54ce-4848-92ca-1e65c3c05957-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:06.910934 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:06.910940    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/558bb0e9-54ce-4848-92ca-1e65c3c05957-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:07.612543 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:07.612516    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f61c55b0-w4hbz"
Oct 30 18:52:07.613053 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:07.612546    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f61c55b0-w4hbz" event={"ID":"558bb0e9-54ce-4848-92ca-1e65c3c05957","Type":"ContainerDied","Data":"b80daf6ebbe1fbb06a78d4f4e80448dbacd04332d923f8ed8184dff01ab03b31"}
Oct 30 18:52:07.613053 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:07.612604    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b80daf6ebbe1fbb06a78d4f4e80448dbacd04332d923f8ed8184dff01ab03b31"
Oct 30 18:52:07.705499 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:07.705465    4525 patch_prober.go:29] interesting pod/e2e-worker-w4-0458df-openshift-default-7c86d79f79-h4qrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 30 18:52:07.705657 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:07.705560    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-h4qrf" podUID="cd500131-13db-47b6-a1a7-b377e950a539" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 30 18:52:07.768915 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:07.768893    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e29de711-qpch8"
Oct 30 18:52:07.823694 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:07.823192    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65"]
Oct 30 18:52:07.825123 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:07.825095    4525 cpu_manager.go:414] "containerMap: removing container" podUID="558bb0e9-54ce-4848-92ca-1e65c3c05957" containerName="revoke-keys"
Oct 30 18:52:07.825293 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:07.825275    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="558bb0e9-54ce-4848-92ca-1e65c3c05957" containerName="revoke-keys"
Oct 30 18:52:07.825400 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:07.825388    4525 cpu_manager.go:414] "containerMap: removing container" podUID="bc4bf99a-34e8-47f1-9380-9a0c9676423f" containerName="revoke-keys"
Oct 30 18:52:07.825554 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:07.825539    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc4bf99a-34e8-47f1-9380-9a0c9676423f" containerName="revoke-keys"
Oct 30 18:52:07.825664 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:07.825650    4525 cpu_manager.go:414] "containerMap: removing container" podUID="070b5c5e-06a4-4ecd-a12e-37ddf440f097" containerName="revoke-keys"
Oct 30 18:52:07.825780 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:07.825767    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="070b5c5e-06a4-4ecd-a12e-37ddf440f097" containerName="revoke-keys"
Oct 30 18:52:07.826221 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:07.826201    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="558bb0e9-54ce-4848-92ca-1e65c3c05957" containerName="revoke-keys"
Oct 30 18:52:07.826345 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:07.826333    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc4bf99a-34e8-47f1-9380-9a0c9676423f" containerName="revoke-keys"
Oct 30 18:52:07.826442 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:07.826431    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="070b5c5e-06a4-4ecd-a12e-37ddf440f097" containerName="revoke-keys"
Oct 30 18:52:07.826618 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:07.826464    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/070b5c5e-06a4-4ecd-a12e-37ddf440f097-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/070b5c5e-06a4-4ecd-a12e-37ddf440f097-maas-api-xks-ca\") pod \"070b5c5e-06a4-4ecd-a12e-37ddf440f097\" (UID: \"070b5c5e-06a4-4ecd-a12e-37ddf440f097\") "
Oct 30 18:52:07.833734 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:07.828439    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/070b5c5e-06a4-4ecd-a12e-37ddf440f097-kube-api-access-gv8kd\" (UniqueName: \"kubernetes.io/projected/070b5c5e-06a4-4ecd-a12e-37ddf440f097-kube-api-access-gv8kd\") pod \"070b5c5e-06a4-4ecd-a12e-37ddf440f097\" (UID: \"070b5c5e-06a4-4ecd-a12e-37ddf440f097\") "
Oct 30 18:52:07.833734 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:07.828542    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/070b5c5e-06a4-4ecd-a12e-37ddf440f097-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/070b5c5e-06a4-4ecd-a12e-37ddf440f097-maas-api-openshift-ca\") pod \"070b5c5e-06a4-4ecd-a12e-37ddf440f097\" (UID: \"070b5c5e-06a4-4ecd-a12e-37ddf440f097\") "
Oct 30 18:52:07.833734 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:07.829021    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/070b5c5e-06a4-4ecd-a12e-37ddf440f097-maas-api-openshift-ca" pod "070b5c5e-06a4-4ecd-a12e-37ddf440f097" (UID: "070b5c5e-06a4-4ecd-a12e-37ddf440f097"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:52:07.833734 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:07.829523    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/070b5c5e-06a4-4ecd-a12e-37ddf440f097-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:07.836515 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:07.836487    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/070b5c5e-06a4-4ecd-a12e-37ddf440f097-maas-api-xks-ca" pod "070b5c5e-06a4-4ecd-a12e-37ddf440f097" (UID: "070b5c5e-06a4-4ecd-a12e-37ddf440f097"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:52:07.839335 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:07.839314    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65"
Oct 30 18:52:07.841957 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:07.841928    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/070b5c5e-06a4-4ecd-a12e-37ddf440f097-kube-api-access-gv8kd" pod "070b5c5e-06a4-4ecd-a12e-37ddf440f097" (UID: "070b5c5e-06a4-4ecd-a12e-37ddf440f097"). InnerVolumeSpecName "kube-api-access-gv8kd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:52:07.842225 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:07.842203    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65"]
Oct 30 18:52:07.930243 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:07.930206    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/391c7032-c25c-47c3-925f-35ea421e07c5-istio-data\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65\" (UID: \"391c7032-c25c-47c3-925f-35ea421e07c5\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65"
Oct 30 18:52:07.930437 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:07.930266    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/391c7032-c25c-47c3-925f-35ea421e07c5-workload-socket\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65\" (UID: \"391c7032-c25c-47c3-925f-35ea421e07c5\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65"
Oct 30 18:52:07.930437 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:07.930395    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/391c7032-c25c-47c3-925f-35ea421e07c5-workload-certs\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65\" (UID: \"391c7032-c25c-47c3-925f-35ea421e07c5\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65"
Oct 30 18:52:07.930437 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:07.930431    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/391c7032-c25c-47c3-925f-35ea421e07c5-credential-socket\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65\" (UID: \"391c7032-c25c-47c3-925f-35ea421e07c5\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65"
Oct 30 18:52:07.930610 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:07.930454    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/391c7032-c25c-47c3-925f-35ea421e07c5-istio-envoy\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65\" (UID: \"391c7032-c25c-47c3-925f-35ea421e07c5\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65"
Oct 30 18:52:07.930610 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:07.930502    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/391c7032-c25c-47c3-925f-35ea421e07c5-istiod-ca-cert\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65\" (UID: \"391c7032-c25c-47c3-925f-35ea421e07c5\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65"
Oct 30 18:52:07.930751 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:07.930656    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/391c7032-c25c-47c3-925f-35ea421e07c5-istio-token\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65\" (UID: \"391c7032-c25c-47c3-925f-35ea421e07c5\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65"
Oct 30 18:52:07.930811 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:07.930745    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t8n4\" (UniqueName: \"kubernetes.io/projected/391c7032-c25c-47c3-925f-35ea421e07c5-kube-api-access-4t8n4\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65\" (UID: \"391c7032-c25c-47c3-925f-35ea421e07c5\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65"
Oct 30 18:52:07.930866 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:07.930824    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/391c7032-c25c-47c3-925f-35ea421e07c5-istio-podinfo\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65\" (UID: \"391c7032-c25c-47c3-925f-35ea421e07c5\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65"
Oct 30 18:52:07.930970 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:07.930953    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/070b5c5e-06a4-4ecd-a12e-37ddf440f097-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:07.931020 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:07.930976    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gv8kd\" (UniqueName: \"kubernetes.io/projected/070b5c5e-06a4-4ecd-a12e-37ddf440f097-kube-api-access-gv8kd\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:08.032030 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:08.031994    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/391c7032-c25c-47c3-925f-35ea421e07c5-istio-token\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65\" (UID: \"391c7032-c25c-47c3-925f-35ea421e07c5\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65"
Oct 30 18:52:08.032030 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:08.032034    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t8n4\" (UniqueName: \"kubernetes.io/projected/391c7032-c25c-47c3-925f-35ea421e07c5-kube-api-access-4t8n4\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65\" (UID: \"391c7032-c25c-47c3-925f-35ea421e07c5\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65"
Oct 30 18:52:08.032260 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:08.032091    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/391c7032-c25c-47c3-925f-35ea421e07c5-istio-podinfo\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65\" (UID: \"391c7032-c25c-47c3-925f-35ea421e07c5\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65"
Oct 30 18:52:08.032260 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:08.032133    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/391c7032-c25c-47c3-925f-35ea421e07c5-istio-data\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65\" (UID: \"391c7032-c25c-47c3-925f-35ea421e07c5\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65"
Oct 30 18:52:08.032260 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:08.032190    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/391c7032-c25c-47c3-925f-35ea421e07c5-workload-socket\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65\" (UID: \"391c7032-c25c-47c3-925f-35ea421e07c5\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65"
Oct 30 18:52:08.032448 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:08.032279    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/391c7032-c25c-47c3-925f-35ea421e07c5-workload-certs\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65\" (UID: \"391c7032-c25c-47c3-925f-35ea421e07c5\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65"
Oct 30 18:52:08.032448 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:08.032316    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/391c7032-c25c-47c3-925f-35ea421e07c5-credential-socket\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65\" (UID: \"391c7032-c25c-47c3-925f-35ea421e07c5\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65"
Oct 30 18:52:08.032448 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:08.032342    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/391c7032-c25c-47c3-925f-35ea421e07c5-istio-envoy\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65\" (UID: \"391c7032-c25c-47c3-925f-35ea421e07c5\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65"
Oct 30 18:52:08.032448 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:08.032368    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/391c7032-c25c-47c3-925f-35ea421e07c5-istiod-ca-cert\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65\" (UID: \"391c7032-c25c-47c3-925f-35ea421e07c5\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65"
Oct 30 18:52:08.032696 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:08.032631    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/391c7032-c25c-47c3-925f-35ea421e07c5-istio-data\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65\" (UID: \"391c7032-c25c-47c3-925f-35ea421e07c5\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65"
Oct 30 18:52:08.032759 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:08.032710    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/391c7032-c25c-47c3-925f-35ea421e07c5-workload-certs\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65\" (UID: \"391c7032-c25c-47c3-925f-35ea421e07c5\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65"
Oct 30 18:52:08.033117 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:08.033090    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/391c7032-c25c-47c3-925f-35ea421e07c5-credential-socket\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65\" (UID: \"391c7032-c25c-47c3-925f-35ea421e07c5\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65"
Oct 30 18:52:08.033217 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:08.033095    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/391c7032-c25c-47c3-925f-35ea421e07c5-workload-socket\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65\" (UID: \"391c7032-c25c-47c3-925f-35ea421e07c5\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65"
Oct 30 18:52:08.033217 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:08.033163    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/391c7032-c25c-47c3-925f-35ea421e07c5-istiod-ca-cert\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65\" (UID: \"391c7032-c25c-47c3-925f-35ea421e07c5\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65"
Oct 30 18:52:08.035142 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:08.035120    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/391c7032-c25c-47c3-925f-35ea421e07c5-istio-envoy\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65\" (UID: \"391c7032-c25c-47c3-925f-35ea421e07c5\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65"
Oct 30 18:52:08.035260 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:08.035184    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/391c7032-c25c-47c3-925f-35ea421e07c5-istio-podinfo\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65\" (UID: \"391c7032-c25c-47c3-925f-35ea421e07c5\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65"
Oct 30 18:52:08.040094 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:08.040067    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/391c7032-c25c-47c3-925f-35ea421e07c5-istio-token\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65\" (UID: \"391c7032-c25c-47c3-925f-35ea421e07c5\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65"
Oct 30 18:52:08.041070 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:08.041043    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t8n4\" (UniqueName: \"kubernetes.io/projected/391c7032-c25c-47c3-925f-35ea421e07c5-kube-api-access-4t8n4\") pod \"e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65\" (UID: \"391c7032-c25c-47c3-925f-35ea421e07c5\") " pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65"
Oct 30 18:52:08.169958 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:08.169926    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65"
Oct 30 18:52:08.464262 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:08.464235    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65"]
Oct 30 18:52:08.464855 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:52:08.464828    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod391c7032_c25c_47c3_925f_35ea421e07c5.slice/crio-5bec36c891864617167a987a03ef7ea50fb05bff58edcc80e112232b04495f1e WatchSource:0}: Error finding container 5bec36c891864617167a987a03ef7ea50fb05bff58edcc80e112232b04495f1e: Status 404 returned error can't find the container with id 5bec36c891864617167a987a03ef7ea50fb05bff58edcc80e112232b04495f1e
Oct 30 18:52:08.467640 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:08.467598    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:52:08.467790 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:08.467710    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:52:08.467790 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:08.467764    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:52:08.632298 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:08.625409    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e29de711-qpch8"
Oct 30 18:52:08.632298 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:08.626066    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e29de711-qpch8" event={"ID":"070b5c5e-06a4-4ecd-a12e-37ddf440f097","Type":"ContainerDied","Data":"f185e78777916b13409f843f5c72e67f319852bbe5f383e1977c3544bad117cb"}
Oct 30 18:52:08.632298 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:08.626125    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f185e78777916b13409f843f5c72e67f319852bbe5f383e1977c3544bad117cb"
Oct 30 18:52:08.632298 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:08.628838    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65" event={"ID":"391c7032-c25c-47c3-925f-35ea421e07c5","Type":"ContainerStarted","Data":"b89bf1225edd646a777d562500d7c8f895cb29ea21cc764e998b9afa96b14f03"}
Oct 30 18:52:08.632298 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:08.628898    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65" event={"ID":"391c7032-c25c-47c3-925f-35ea421e07c5","Type":"ContainerStarted","Data":"5bec36c891864617167a987a03ef7ea50fb05bff58edcc80e112232b04495f1e"}
Oct 30 18:52:08.659373 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:08.657844    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65" podStartSLOduration=1.6578276870000002 podStartE2EDuration="1.657827687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:52:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:08.654535439 +0000 UTC m=+951.280205404" watchObservedRunningTime="2026-10-30 18:52:08.657827687 +0000 UTC m=+951.283497652"
Oct 30 18:52:08.941087 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:08.938861    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-h4qrf"
Oct 30 18:52:08.941483 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:08.941460    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-h4qrf"
Oct 30 18:52:08.943614 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:08.942693    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-h4qrf"
Oct 30 18:52:09.181524 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:09.175755    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65"
Oct 30 18:52:09.192768 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:09.191843    4525 patch_prober.go:29] interesting pod/e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 30 18:52:09.192768 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:09.191903    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65" podUID="391c7032-c25c-47c3-925f-35ea421e07c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 30 18:52:10.171090 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:10.171060    4525 patch_prober.go:29] interesting pod/e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 30 18:52:10.171585 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:10.171110    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65" podUID="391c7032-c25c-47c3-925f-35ea421e07c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 30 18:52:11.170348 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:11.170309    4525 patch_prober.go:29] interesting pod/e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 30 18:52:11.170544 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:11.170370    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65" podUID="391c7032-c25c-47c3-925f-35ea421e07c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 30 18:52:12.192735 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:12.192357    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d4bed94-hrbfm"]
Oct 30 18:52:12.235709 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:12.212063    4525 patch_prober.go:29] interesting pod/e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 30 18:52:12.307154 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:12.307013    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d4bed94-hrbfm"
Oct 30 18:52:12.313911 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:12.313870    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:52:12.335239 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:12.316584    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94hcn\""
Oct 30 18:52:12.335514 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:12.307094    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65" podUID="391c7032-c25c-47c3-925f-35ea421e07c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 30 18:52:12.355712 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:12.340541    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac945b70-9980-4d7c-8b6a-01b3fb349c44-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d4bed94-hrbfm\" (UID: \"ac945b70-9980-4d7c-8b6a-01b3fb349c44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d4bed94-hrbfm"
Oct 30 18:52:12.365705 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:12.356108    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac945b70-9980-4d7c-8b6a-01b3fb349c44-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d4bed94-hrbfm\" (UID: \"ac945b70-9980-4d7c-8b6a-01b3fb349c44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d4bed94-hrbfm"
Oct 30 18:52:12.366039 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:12.366008    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pljrq\" (UniqueName: \"kubernetes.io/projected/ac945b70-9980-4d7c-8b6a-01b3fb349c44-kube-api-access-pljrq\") pod \"maas-api-revoke-sub-9d4bed94-hrbfm\" (UID: \"ac945b70-9980-4d7c-8b6a-01b3fb349c44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d4bed94-hrbfm"
Oct 30 18:52:12.380958 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:12.380918    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d4bed94-hrbfm"]
Oct 30 18:52:12.517770 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:12.515760    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac945b70-9980-4d7c-8b6a-01b3fb349c44-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d4bed94-hrbfm\" (UID: \"ac945b70-9980-4d7c-8b6a-01b3fb349c44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d4bed94-hrbfm"
Oct 30 18:52:12.517770 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:12.515932    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac945b70-9980-4d7c-8b6a-01b3fb349c44-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d4bed94-hrbfm\" (UID: \"ac945b70-9980-4d7c-8b6a-01b3fb349c44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d4bed94-hrbfm"
Oct 30 18:52:12.517770 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:12.515992    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pljrq\" (UniqueName: \"kubernetes.io/projected/ac945b70-9980-4d7c-8b6a-01b3fb349c44-kube-api-access-pljrq\") pod \"maas-api-revoke-sub-9d4bed94-hrbfm\" (UID: \"ac945b70-9980-4d7c-8b6a-01b3fb349c44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d4bed94-hrbfm"
Oct 30 18:52:12.517770 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:12.517670    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac945b70-9980-4d7c-8b6a-01b3fb349c44-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d4bed94-hrbfm\" (UID: \"ac945b70-9980-4d7c-8b6a-01b3fb349c44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d4bed94-hrbfm"
Oct 30 18:52:12.605649 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:12.605617    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pljrq\" (UniqueName: \"kubernetes.io/projected/ac945b70-9980-4d7c-8b6a-01b3fb349c44-kube-api-access-pljrq\") pod \"maas-api-revoke-sub-9d4bed94-hrbfm\" (UID: \"ac945b70-9980-4d7c-8b6a-01b3fb349c44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d4bed94-hrbfm"
Oct 30 18:52:12.619223 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:12.619189    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac945b70-9980-4d7c-8b6a-01b3fb349c44-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d4bed94-hrbfm\" (UID: \"ac945b70-9980-4d7c-8b6a-01b3fb349c44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d4bed94-hrbfm"
Oct 30 18:52:12.645707 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:12.639926    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d4bed94-hrbfm"
Oct 30 18:52:13.177709 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:13.172124    4525 patch_prober.go:29] interesting pod/e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 30 18:52:13.177709 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:13.172192    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65" podUID="391c7032-c25c-47c3-925f-35ea421e07c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 30 18:52:13.428769 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:13.400926    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj" podUID="84c99c49-1d60-4176-bda4-cad29c7d801c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:52:13.483993 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:13.476784    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb" podUID="12ebc8b9-c794-47b5-b87b-a63c4bd993e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 18:52:13.520202 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:13.498799    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv" podUID="595230e3-55f3-43eb-9a7b-6d89250af9b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:52:13.520202 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:13.519782    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw" podUID="ac6852fd-01be-4d69-8c9b-0c591e50b889" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:52:13.960710 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:13.940583    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d4bed94-hrbfm"]
Oct 30 18:52:14.011545 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:14.011505    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t" podUID="424fca14-6d8a-4663-86ec-0c91165f34ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:52:14.035722 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:14.013865    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr" podUID="82bbbf85-ed93-48d2-9f17-49ac1f31c0a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 30 18:52:14.177316 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:14.177179    4525 patch_prober.go:29] interesting pod/e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 30 18:52:14.177316 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:14.177238    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65" podUID="391c7032-c25c-47c3-925f-35ea421e07c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 30 18:52:14.362706 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:14.359332    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts" podUID="be6d4969-285b-4da0-883e-4847f2cbf3af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:52:14.518120 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:14.518044    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv" podUID="595230e3-55f3-43eb-9a7b-6d89250af9b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 18:52:14.519100 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:14.519068    4525 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts" podUID="be6d4969-285b-4da0-883e-4847f2cbf3af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": context deadline exceeded"
Oct 30 18:52:14.519464 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:14.519444    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv"
Oct 30 18:52:14.527329 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:14.527291    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj" podUID="84c99c49-1d60-4176-bda4-cad29c7d801c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 18:52:14.576844 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:14.576811    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj"
Oct 30 18:52:14.894390 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:14.893145    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw" podUID="ac6852fd-01be-4d69-8c9b-0c591e50b889" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:52:14.921707 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:14.895323    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw"
Oct 30 18:52:15.262118 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:15.258308    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr" podUID="82bbbf85-ed93-48d2-9f17-49ac1f31c0a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:52:15.288454 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:15.284140    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t" podUID="424fca14-6d8a-4663-86ec-0c91165f34ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 18:52:15.451716 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:15.402142    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts" podUID="be6d4969-285b-4da0-883e-4847f2cbf3af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:52:15.478708 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:15.403611    4525 patch_prober.go:29] interesting pod/e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 30 18:52:15.479033 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:15.478995    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65" podUID="391c7032-c25c-47c3-925f-35ea421e07c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 30 18:52:15.479204 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:15.436959    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t"
Oct 30 18:52:15.529064 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:15.528966    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv" podUID="595230e3-55f3-43eb-9a7b-6d89250af9b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:52:15.555795 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:15.543911    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj" podUID="84c99c49-1d60-4176-bda4-cad29c7d801c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:52:15.676905 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:15.676857    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb" podUID="12ebc8b9-c794-47b5-b87b-a63c4bd993e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:52:15.770242 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:15.757037    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr"
Oct 30 18:52:16.045702 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:16.040800    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts"
Oct 30 18:52:16.079717 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:16.072051    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw" podUID="ac6852fd-01be-4d69-8c9b-0c591e50b889" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 18:52:16.136154 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:16.136054    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d4bed94-hrbfm" event={"ID":"ac945b70-9980-4d7c-8b6a-01b3fb349c44","Type":"ContainerStarted","Data":"e8c9ecae69b77e1ffd8f592c6d0790349d9d9744dddeaac61d7abfeef8f9b572"}
Oct 30 18:52:16.174812 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:16.174778    4525 patch_prober.go:29] interesting pod/e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 30 18:52:16.175103 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:16.175047    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65" podUID="391c7032-c25c-47c3-925f-35ea421e07c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 30 18:52:16.277810 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:16.277783    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t"
Oct 30 18:52:16.799657 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:16.799617    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-242299b0-f4z9m"]
Oct 30 18:52:16.806704 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:16.804955    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-242299b0-f4z9m"
Oct 30 18:52:16.827987 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:16.827947    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts" podUID="be6d4969-285b-4da0-883e-4847f2cbf3af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:52:16.854707 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:16.830787    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-242299b0-f4z9m"]
Oct 30 18:52:16.854707 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:16.830886    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj"
Oct 30 18:52:16.854707 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:16.830924    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv"
Oct 30 18:52:16.854707 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:16.847002    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw"
Oct 30 18:52:16.854707 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:16.848364    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr"
Oct 30 18:52:16.892989 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:16.891950    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts"
Oct 30 18:52:16.964803 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:16.964762    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ead1810-d291-4c15-ad2a-e310e653d399-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-242299b0-f4z9m\" (UID: \"4ead1810-d291-4c15-ad2a-e310e653d399\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-242299b0-f4z9m"
Oct 30 18:52:16.964999 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:16.964828    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4stgt\" (UniqueName: \"kubernetes.io/projected/4ead1810-d291-4c15-ad2a-e310e653d399-kube-api-access-4stgt\") pod \"maas-api-revoke-sub-242299b0-f4z9m\" (UID: \"4ead1810-d291-4c15-ad2a-e310e653d399\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-242299b0-f4z9m"
Oct 30 18:52:16.964999 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:16.964958    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ead1810-d291-4c15-ad2a-e310e653d399-maas-api-xks-ca\") pod \"maas-api-revoke-sub-242299b0-f4z9m\" (UID: \"4ead1810-d291-4c15-ad2a-e310e653d399\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-242299b0-f4z9m"
Oct 30 18:52:17.067045 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:17.066954    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ead1810-d291-4c15-ad2a-e310e653d399-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-242299b0-f4z9m\" (UID: \"4ead1810-d291-4c15-ad2a-e310e653d399\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-242299b0-f4z9m"
Oct 30 18:52:17.067045 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:17.067018    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4stgt\" (UniqueName: \"kubernetes.io/projected/4ead1810-d291-4c15-ad2a-e310e653d399-kube-api-access-4stgt\") pod \"maas-api-revoke-sub-242299b0-f4z9m\" (UID: \"4ead1810-d291-4c15-ad2a-e310e653d399\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-242299b0-f4z9m"
Oct 30 18:52:17.067345 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:17.067095    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ead1810-d291-4c15-ad2a-e310e653d399-maas-api-xks-ca\") pod \"maas-api-revoke-sub-242299b0-f4z9m\" (UID: \"4ead1810-d291-4c15-ad2a-e310e653d399\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-242299b0-f4z9m"
Oct 30 18:52:17.067719 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:17.067674    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ead1810-d291-4c15-ad2a-e310e653d399-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-242299b0-f4z9m\" (UID: \"4ead1810-d291-4c15-ad2a-e310e653d399\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-242299b0-f4z9m"
Oct 30 18:52:17.069777 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:17.069753    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ead1810-d291-4c15-ad2a-e310e653d399-maas-api-xks-ca\") pod \"maas-api-revoke-sub-242299b0-f4z9m\" (UID: \"4ead1810-d291-4c15-ad2a-e310e653d399\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-242299b0-f4z9m"
Oct 30 18:52:17.077380 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:17.077359    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4stgt\" (UniqueName: \"kubernetes.io/projected/4ead1810-d291-4c15-ad2a-e310e653d399-kube-api-access-4stgt\") pod \"maas-api-revoke-sub-242299b0-f4z9m\" (UID: \"4ead1810-d291-4c15-ad2a-e310e653d399\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-242299b0-f4z9m"
Oct 30 18:52:17.147511 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:17.147482    4525 generic.go:356] "Generic (PLEG): container finished" podID="ac945b70-9980-4d7c-8b6a-01b3fb349c44" containerID="d48ef5b500331c1faf8272f0c340885b96c30959e16a97c9ec2f9a5471306cd8" exitCode=0
Oct 30 18:52:17.147652 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:17.147575    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d4bed94-hrbfm" event={"ID":"ac945b70-9980-4d7c-8b6a-01b3fb349c44","Type":"ContainerDied","Data":"d48ef5b500331c1faf8272f0c340885b96c30959e16a97c9ec2f9a5471306cd8"}
Oct 30 18:52:17.170328 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:17.170305    4525 patch_prober.go:29] interesting pod/e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 30 18:52:17.170460 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:17.170358    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65" podUID="391c7032-c25c-47c3-925f-35ea421e07c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 30 18:52:17.203271 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:17.203235    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-242299b0-f4z9m"
Oct 30 18:52:17.399066 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:52:17.399028    4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d49l" containerName="extractor"
Oct 30 18:52:17.492590 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:52:17.492562    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ead1810_d291_4c15_ad2a_e310e653d399.slice/crio-b5238d7b3a8115f4ddcfb041eb637d416b3e8f9794e456fe9874295f37838248 WatchSource:0}: Error finding container b5238d7b3a8115f4ddcfb041eb637d416b3e8f9794e456fe9874295f37838248: Status 404 returned error can't find the container with id b5238d7b3a8115f4ddcfb041eb637d416b3e8f9794e456fe9874295f37838248
Oct 30 18:52:17.493066 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:17.493044    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-242299b0-f4z9m"]
Oct 30 18:52:18.158235 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:18.158191    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-242299b0-f4z9m" event={"ID":"4ead1810-d291-4c15-ad2a-e310e653d399","Type":"ContainerStarted","Data":"9a3ee946b74b86bd3ae429663137c93bb300aa73afb57d25310876739db101e2"}
Oct 30 18:52:18.158718 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:18.158258    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-242299b0-f4z9m" event={"ID":"4ead1810-d291-4c15-ad2a-e310e653d399","Type":"ContainerStarted","Data":"b5238d7b3a8115f4ddcfb041eb637d416b3e8f9794e456fe9874295f37838248"}
Oct 30 18:52:18.191727 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:18.179645    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-242299b0-f4z9m" podStartSLOduration=3.179571029 podStartE2EDuration="3.179571029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:52:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:18.174889677 +0000 UTC m=+960.800559640" watchObservedRunningTime="2026-10-30 18:52:18.179571029 +0000 UTC m=+960.805241002"
Oct 30 18:52:18.325106 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:18.325077    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d4bed94-hrbfm"
Oct 30 18:52:18.394016 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:18.393985    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac945b70-9980-4d7c-8b6a-01b3fb349c44-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac945b70-9980-4d7c-8b6a-01b3fb349c44-maas-api-xks-ca\") pod \"ac945b70-9980-4d7c-8b6a-01b3fb349c44\" (UID: \"ac945b70-9980-4d7c-8b6a-01b3fb349c44\") "
Oct 30 18:52:18.394191 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:18.394076    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac945b70-9980-4d7c-8b6a-01b3fb349c44-kube-api-access-pljrq\" (UniqueName: \"kubernetes.io/projected/ac945b70-9980-4d7c-8b6a-01b3fb349c44-kube-api-access-pljrq\") pod \"ac945b70-9980-4d7c-8b6a-01b3fb349c44\" (UID: \"ac945b70-9980-4d7c-8b6a-01b3fb349c44\") "
Oct 30 18:52:18.394266 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:18.394214    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac945b70-9980-4d7c-8b6a-01b3fb349c44-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac945b70-9980-4d7c-8b6a-01b3fb349c44-maas-api-openshift-ca\") pod \"ac945b70-9980-4d7c-8b6a-01b3fb349c44\" (UID: \"ac945b70-9980-4d7c-8b6a-01b3fb349c44\") "
Oct 30 18:52:18.395145 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:18.395110    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac945b70-9980-4d7c-8b6a-01b3fb349c44-maas-api-openshift-ca" pod "ac945b70-9980-4d7c-8b6a-01b3fb349c44" (UID: "ac945b70-9980-4d7c-8b6a-01b3fb349c44"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:52:18.396605 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:18.396576    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac945b70-9980-4d7c-8b6a-01b3fb349c44-kube-api-access-pljrq" pod "ac945b70-9980-4d7c-8b6a-01b3fb349c44" (UID: "ac945b70-9980-4d7c-8b6a-01b3fb349c44"). InnerVolumeSpecName "kube-api-access-pljrq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:52:18.396899 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:18.396878    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac945b70-9980-4d7c-8b6a-01b3fb349c44-maas-api-xks-ca" pod "ac945b70-9980-4d7c-8b6a-01b3fb349c44" (UID: "ac945b70-9980-4d7c-8b6a-01b3fb349c44"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:52:18.495836 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:18.495770    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac945b70-9980-4d7c-8b6a-01b3fb349c44-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:18.495836 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:18.495799    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pljrq\" (UniqueName: \"kubernetes.io/projected/ac945b70-9980-4d7c-8b6a-01b3fb349c44-kube-api-access-pljrq\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:18.495836 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:18.495811    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac945b70-9980-4d7c-8b6a-01b3fb349c44-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:19.167489 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:19.167453    4525 generic.go:356] "Generic (PLEG): container finished" podID="4ead1810-d291-4c15-ad2a-e310e653d399" containerID="9a3ee946b74b86bd3ae429663137c93bb300aa73afb57d25310876739db101e2" exitCode=0
Oct 30 18:52:19.168015 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:19.167538    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-242299b0-f4z9m" event={"ID":"4ead1810-d291-4c15-ad2a-e310e653d399","Type":"ContainerDied","Data":"9a3ee946b74b86bd3ae429663137c93bb300aa73afb57d25310876739db101e2"}
Oct 30 18:52:19.169084 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:19.169062    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d4bed94-hrbfm" event={"ID":"ac945b70-9980-4d7c-8b6a-01b3fb349c44","Type":"ContainerDied","Data":"e8c9ecae69b77e1ffd8f592c6d0790349d9d9744dddeaac61d7abfeef8f9b572"}
Oct 30 18:52:19.169084 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:19.169084    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d4bed94-hrbfm"
Oct 30 18:52:19.169241 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:19.169098    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e8c9ecae69b77e1ffd8f592c6d0790349d9d9744dddeaac61d7abfeef8f9b572"
Oct 30 18:52:19.211880 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:19.211838    4525 patch_prober.go:29] interesting pod/e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 18:52:19.212038 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:19.211919    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65" podUID="391c7032-c25c-47c3-925f-35ea421e07c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:52:19.425913 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:19.425825    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65"
Oct 30 18:52:19.426316 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:19.426292    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65"
Oct 30 18:52:19.432062 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:19.432042    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65"
Oct 30 18:52:20.343901 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:20.343866    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-242299b0-f4z9m"
Oct 30 18:52:20.420381 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:20.420322    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ead1810-d291-4c15-ad2a-e310e653d399-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ead1810-d291-4c15-ad2a-e310e653d399-maas-api-openshift-ca\") pod \"4ead1810-d291-4c15-ad2a-e310e653d399\" (UID: \"4ead1810-d291-4c15-ad2a-e310e653d399\") "
Oct 30 18:52:20.420588 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:20.420408    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ead1810-d291-4c15-ad2a-e310e653d399-kube-api-access-4stgt\" (UniqueName: \"kubernetes.io/projected/4ead1810-d291-4c15-ad2a-e310e653d399-kube-api-access-4stgt\") pod \"4ead1810-d291-4c15-ad2a-e310e653d399\" (UID: \"4ead1810-d291-4c15-ad2a-e310e653d399\") "
Oct 30 18:52:20.420705 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:20.420659    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ead1810-d291-4c15-ad2a-e310e653d399-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ead1810-d291-4c15-ad2a-e310e653d399-maas-api-xks-ca\") pod \"4ead1810-d291-4c15-ad2a-e310e653d399\" (UID: \"4ead1810-d291-4c15-ad2a-e310e653d399\") "
Oct 30 18:52:20.420808 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:20.420669    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ead1810-d291-4c15-ad2a-e310e653d399-maas-api-openshift-ca" pod "4ead1810-d291-4c15-ad2a-e310e653d399" (UID: "4ead1810-d291-4c15-ad2a-e310e653d399"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:52:20.421144 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:20.421124    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ead1810-d291-4c15-ad2a-e310e653d399-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:20.422855 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:20.422829    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ead1810-d291-4c15-ad2a-e310e653d399-maas-api-xks-ca" pod "4ead1810-d291-4c15-ad2a-e310e653d399" (UID: "4ead1810-d291-4c15-ad2a-e310e653d399"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:52:20.423069 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:20.423054    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ead1810-d291-4c15-ad2a-e310e653d399-kube-api-access-4stgt" pod "4ead1810-d291-4c15-ad2a-e310e653d399" (UID: "4ead1810-d291-4c15-ad2a-e310e653d399"). InnerVolumeSpecName "kube-api-access-4stgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:52:20.522546 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:20.522514    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ead1810-d291-4c15-ad2a-e310e653d399-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:20.522546 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:20.522544    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4stgt\" (UniqueName: \"kubernetes.io/projected/4ead1810-d291-4c15-ad2a-e310e653d399-kube-api-access-4stgt\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:21.186907 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:21.186879    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-242299b0-f4z9m"
Oct 30 18:52:21.187101 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:21.186915    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-242299b0-f4z9m" event={"ID":"4ead1810-d291-4c15-ad2a-e310e653d399","Type":"ContainerDied","Data":"b5238d7b3a8115f4ddcfb041eb637d416b3e8f9794e456fe9874295f37838248"}
Oct 30 18:52:21.187101 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:21.186966    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b5238d7b3a8115f4ddcfb041eb637d416b3e8f9794e456fe9874295f37838248"
Oct 30 18:52:24.026963 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:24.026927    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6926d01d-cqhns"]
Oct 30 18:52:24.027513 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:24.027459    4525 cpu_manager.go:414] "containerMap: removing container" podUID="ac945b70-9980-4d7c-8b6a-01b3fb349c44" containerName="revoke-keys"
Oct 30 18:52:24.027513 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:24.027471    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac945b70-9980-4d7c-8b6a-01b3fb349c44" containerName="revoke-keys"
Oct 30 18:52:24.027513 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:24.027487    4525 cpu_manager.go:414] "containerMap: removing container" podUID="4ead1810-d291-4c15-ad2a-e310e653d399" containerName="revoke-keys"
Oct 30 18:52:24.027513 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:24.027492    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ead1810-d291-4c15-ad2a-e310e653d399" containerName="revoke-keys"
Oct 30 18:52:24.027779 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:24.027578    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac945b70-9980-4d7c-8b6a-01b3fb349c44" containerName="revoke-keys"
Oct 30 18:52:24.027779 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:24.027588    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ead1810-d291-4c15-ad2a-e310e653d399" containerName="revoke-keys"
Oct 30 18:52:24.032429 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:24.032402    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6926d01d-cqhns"
Oct 30 18:52:24.036637 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:24.036611    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:52:24.037016 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:24.036997    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94hcn\""
Oct 30 18:52:24.041065 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:24.041037    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6926d01d-cqhns"]
Oct 30 18:52:24.168010 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:24.167968    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/262b6401-e723-497a-9637-5d80e4d722d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6926d01d-cqhns\" (UID: \"262b6401-e723-497a-9637-5d80e4d722d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6926d01d-cqhns"
Oct 30 18:52:24.168207 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:24.168025    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kssf\" (UniqueName: \"kubernetes.io/projected/262b6401-e723-497a-9637-5d80e4d722d9-kube-api-access-7kssf\") pod \"maas-api-revoke-sub-6926d01d-cqhns\" (UID: \"262b6401-e723-497a-9637-5d80e4d722d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6926d01d-cqhns"
Oct 30 18:52:24.168281 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:24.168215    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/262b6401-e723-497a-9637-5d80e4d722d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6926d01d-cqhns\" (UID: \"262b6401-e723-497a-9637-5d80e4d722d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6926d01d-cqhns"
Oct 30 18:52:24.272795 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:24.272757    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/262b6401-e723-497a-9637-5d80e4d722d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6926d01d-cqhns\" (UID: \"262b6401-e723-497a-9637-5d80e4d722d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6926d01d-cqhns"
Oct 30 18:52:24.273057 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:24.273039    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kssf\" (UniqueName: \"kubernetes.io/projected/262b6401-e723-497a-9637-5d80e4d722d9-kube-api-access-7kssf\") pod \"maas-api-revoke-sub-6926d01d-cqhns\" (UID: \"262b6401-e723-497a-9637-5d80e4d722d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6926d01d-cqhns"
Oct 30 18:52:24.273265 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:24.273250    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/262b6401-e723-497a-9637-5d80e4d722d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6926d01d-cqhns\" (UID: \"262b6401-e723-497a-9637-5d80e4d722d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6926d01d-cqhns"
Oct 30 18:52:24.274888 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:24.274090    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/262b6401-e723-497a-9637-5d80e4d722d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6926d01d-cqhns\" (UID: \"262b6401-e723-497a-9637-5d80e4d722d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6926d01d-cqhns"
Oct 30 18:52:24.297917 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:24.297831    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kssf\" (UniqueName: \"kubernetes.io/projected/262b6401-e723-497a-9637-5d80e4d722d9-kube-api-access-7kssf\") pod \"maas-api-revoke-sub-6926d01d-cqhns\" (UID: \"262b6401-e723-497a-9637-5d80e4d722d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6926d01d-cqhns"
Oct 30 18:52:24.298493 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:24.298423    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/262b6401-e723-497a-9637-5d80e4d722d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6926d01d-cqhns\" (UID: \"262b6401-e723-497a-9637-5d80e4d722d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6926d01d-cqhns"
Oct 30 18:52:24.359042 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:24.359001    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6926d01d-cqhns"
Oct 30 18:52:24.651315 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:52:24.650877    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod262b6401_e723_497a_9637_5d80e4d722d9.slice/crio-de1f942c215849d2dc1dfca1c067f3da9c5ff8125e03215710b7ef7e7458134a WatchSource:0}: Error finding container de1f942c215849d2dc1dfca1c067f3da9c5ff8125e03215710b7ef7e7458134a: Status 404 returned error can't find the container with id de1f942c215849d2dc1dfca1c067f3da9c5ff8125e03215710b7ef7e7458134a
Oct 30 18:52:24.652673 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:24.652639    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6926d01d-cqhns"]
Oct 30 18:52:25.225373 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:25.225279    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6926d01d-cqhns" event={"ID":"262b6401-e723-497a-9637-5d80e4d722d9","Type":"ContainerStarted","Data":"c1a5039fa72dd937ad5cdb1b2bd7c27991e5e5255a1cbf9ef0d5c73fa75e527f"}
Oct 30 18:52:25.226531 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:25.226488    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6926d01d-cqhns" event={"ID":"262b6401-e723-497a-9637-5d80e4d722d9","Type":"ContainerStarted","Data":"de1f942c215849d2dc1dfca1c067f3da9c5ff8125e03215710b7ef7e7458134a"}
Oct 30 18:52:25.244316 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:25.244248    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6926d01d-cqhns" podStartSLOduration=1.244226989 podStartE2EDuration="1.244226989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:52:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:25.241003118 +0000 UTC m=+967.866673082" watchObservedRunningTime="2026-10-30 18:52:25.244226989 +0000 UTC m=+967.869896954"
Oct 30 18:52:26.276772 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:26.275918    4525 generic.go:356] "Generic (PLEG): container finished" podID="262b6401-e723-497a-9637-5d80e4d722d9" containerID="c1a5039fa72dd937ad5cdb1b2bd7c27991e5e5255a1cbf9ef0d5c73fa75e527f" exitCode=0
Oct 30 18:52:26.276772 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:26.276197    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6926d01d-cqhns" event={"ID":"262b6401-e723-497a-9637-5d80e4d722d9","Type":"ContainerDied","Data":"c1a5039fa72dd937ad5cdb1b2bd7c27991e5e5255a1cbf9ef0d5c73fa75e527f"}
Oct 30 18:52:27.446282 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:27.446257    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6926d01d-cqhns"
Oct 30 18:52:27.560045 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:27.559974    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/262b6401-e723-497a-9637-5d80e4d722d9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/262b6401-e723-497a-9637-5d80e4d722d9-maas-api-xks-ca\") pod \"262b6401-e723-497a-9637-5d80e4d722d9\" (UID: \"262b6401-e723-497a-9637-5d80e4d722d9\") "
Oct 30 18:52:27.560045 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:27.560011    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/262b6401-e723-497a-9637-5d80e4d722d9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/262b6401-e723-497a-9637-5d80e4d722d9-maas-api-openshift-ca\") pod \"262b6401-e723-497a-9637-5d80e4d722d9\" (UID: \"262b6401-e723-497a-9637-5d80e4d722d9\") "
Oct 30 18:52:27.560256 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:27.560076    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/262b6401-e723-497a-9637-5d80e4d722d9-kube-api-access-7kssf\" (UniqueName: \"kubernetes.io/projected/262b6401-e723-497a-9637-5d80e4d722d9-kube-api-access-7kssf\") pod \"262b6401-e723-497a-9637-5d80e4d722d9\" (UID: \"262b6401-e723-497a-9637-5d80e4d722d9\") "
Oct 30 18:52:27.560372 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:27.560348    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/262b6401-e723-497a-9637-5d80e4d722d9-maas-api-openshift-ca" pod "262b6401-e723-497a-9637-5d80e4d722d9" (UID: "262b6401-e723-497a-9637-5d80e4d722d9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:52:27.562212 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:27.562187    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/262b6401-e723-497a-9637-5d80e4d722d9-maas-api-xks-ca" pod "262b6401-e723-497a-9637-5d80e4d722d9" (UID: "262b6401-e723-497a-9637-5d80e4d722d9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:52:27.562332 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:27.562305    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/262b6401-e723-497a-9637-5d80e4d722d9-kube-api-access-7kssf" pod "262b6401-e723-497a-9637-5d80e4d722d9" (UID: "262b6401-e723-497a-9637-5d80e4d722d9"). InnerVolumeSpecName "kube-api-access-7kssf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:52:27.662217 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:27.662182    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/262b6401-e723-497a-9637-5d80e4d722d9-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:27.662217 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:27.662217    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/262b6401-e723-497a-9637-5d80e4d722d9-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:27.662465 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:27.662233    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kssf\" (UniqueName: \"kubernetes.io/projected/262b6401-e723-497a-9637-5d80e4d722d9-kube-api-access-7kssf\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:27.781974 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:27.781939    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-541d2bf7-w25bt"]
Oct 30 18:52:27.782757 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:27.782738    4525 cpu_manager.go:414] "containerMap: removing container" podUID="262b6401-e723-497a-9637-5d80e4d722d9" containerName="revoke-keys"
Oct 30 18:52:27.782757 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:27.782758    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="262b6401-e723-497a-9637-5d80e4d722d9" containerName="revoke-keys"
Oct 30 18:52:27.782854 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:27.782847    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="262b6401-e723-497a-9637-5d80e4d722d9" containerName="revoke-keys"
Oct 30 18:52:27.787100 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:27.787076    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-541d2bf7-w25bt"
Oct 30 18:52:27.800035 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:27.799997    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-541d2bf7-w25bt"]
Oct 30 18:52:27.972707 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:27.970220    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/328d84f1-7f29-4e6c-b62d-df7a0709914a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-541d2bf7-w25bt\" (UID: \"328d84f1-7f29-4e6c-b62d-df7a0709914a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-541d2bf7-w25bt"
Oct 30 18:52:27.972707 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:27.970500    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4l8qs\" (UniqueName: \"kubernetes.io/projected/328d84f1-7f29-4e6c-b62d-df7a0709914a-kube-api-access-4l8qs\") pod \"maas-api-revoke-sub-541d2bf7-w25bt\" (UID: \"328d84f1-7f29-4e6c-b62d-df7a0709914a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-541d2bf7-w25bt"
Oct 30 18:52:27.972707 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:27.970615    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/328d84f1-7f29-4e6c-b62d-df7a0709914a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-541d2bf7-w25bt\" (UID: \"328d84f1-7f29-4e6c-b62d-df7a0709914a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-541d2bf7-w25bt"
Oct 30 18:52:28.071967 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:28.071936    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l8qs\" (UniqueName: \"kubernetes.io/projected/328d84f1-7f29-4e6c-b62d-df7a0709914a-kube-api-access-4l8qs\") pod \"maas-api-revoke-sub-541d2bf7-w25bt\" (UID: \"328d84f1-7f29-4e6c-b62d-df7a0709914a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-541d2bf7-w25bt"
Oct 30 18:52:28.072163 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:28.072024    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/328d84f1-7f29-4e6c-b62d-df7a0709914a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-541d2bf7-w25bt\" (UID: \"328d84f1-7f29-4e6c-b62d-df7a0709914a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-541d2bf7-w25bt"
Oct 30 18:52:28.072163 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:28.072086    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/328d84f1-7f29-4e6c-b62d-df7a0709914a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-541d2bf7-w25bt\" (UID: \"328d84f1-7f29-4e6c-b62d-df7a0709914a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-541d2bf7-w25bt"
Oct 30 18:52:28.072815 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:28.072787    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/328d84f1-7f29-4e6c-b62d-df7a0709914a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-541d2bf7-w25bt\" (UID: \"328d84f1-7f29-4e6c-b62d-df7a0709914a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-541d2bf7-w25bt"
Oct 30 18:52:28.074566 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:28.074547    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/328d84f1-7f29-4e6c-b62d-df7a0709914a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-541d2bf7-w25bt\" (UID: \"328d84f1-7f29-4e6c-b62d-df7a0709914a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-541d2bf7-w25bt"
Oct 30 18:52:28.082413 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:28.082385    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l8qs\" (UniqueName: \"kubernetes.io/projected/328d84f1-7f29-4e6c-b62d-df7a0709914a-kube-api-access-4l8qs\") pod \"maas-api-revoke-sub-541d2bf7-w25bt\" (UID: \"328d84f1-7f29-4e6c-b62d-df7a0709914a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-541d2bf7-w25bt"
Oct 30 18:52:28.129331 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:28.129299    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-541d2bf7-w25bt"
Oct 30 18:52:28.194980 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:28.194641    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-073ea9f1-pnfqr"]
Oct 30 18:52:28.204788 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:28.204763    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-073ea9f1-pnfqr"
Oct 30 18:52:28.211706 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:28.211544    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-073ea9f1-pnfqr"]
Oct 30 18:52:28.294671 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:28.294597    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6926d01d-cqhns"
Oct 30 18:52:28.294671 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:28.294621    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6926d01d-cqhns" event={"ID":"262b6401-e723-497a-9637-5d80e4d722d9","Type":"ContainerDied","Data":"de1f942c215849d2dc1dfca1c067f3da9c5ff8125e03215710b7ef7e7458134a"}
Oct 30 18:52:28.294867 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:28.294672    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de1f942c215849d2dc1dfca1c067f3da9c5ff8125e03215710b7ef7e7458134a"
Oct 30 18:52:28.377011 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:28.376979    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rff75\" (UniqueName: \"kubernetes.io/projected/84f528c3-fc82-45a6-9c25-f72970220236-kube-api-access-rff75\") pod \"maas-api-revoke-sub-073ea9f1-pnfqr\" (UID: \"84f528c3-fc82-45a6-9c25-f72970220236\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-073ea9f1-pnfqr"
Oct 30 18:52:28.377174 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:28.377101    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84f528c3-fc82-45a6-9c25-f72970220236-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-073ea9f1-pnfqr\" (UID: \"84f528c3-fc82-45a6-9c25-f72970220236\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-073ea9f1-pnfqr"
Oct 30 18:52:28.377174 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:28.377146    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84f528c3-fc82-45a6-9c25-f72970220236-maas-api-xks-ca\") pod \"maas-api-revoke-sub-073ea9f1-pnfqr\" (UID: \"84f528c3-fc82-45a6-9c25-f72970220236\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-073ea9f1-pnfqr"
Oct 30 18:52:28.412640 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:28.412613    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-541d2bf7-w25bt"]
Oct 30 18:52:28.412879 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:52:28.412825    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod328d84f1_7f29_4e6c_b62d_df7a0709914a.slice/crio-f4df21929c841181df4e06e8f297ba98a44cc948871676a6935bd97fc67c204a WatchSource:0}: Error finding container f4df21929c841181df4e06e8f297ba98a44cc948871676a6935bd97fc67c204a: Status 404 returned error can't find the container with id f4df21929c841181df4e06e8f297ba98a44cc948871676a6935bd97fc67c204a
Oct 30 18:52:28.478730 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:28.478697    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rff75\" (UniqueName: \"kubernetes.io/projected/84f528c3-fc82-45a6-9c25-f72970220236-kube-api-access-rff75\") pod \"maas-api-revoke-sub-073ea9f1-pnfqr\" (UID: \"84f528c3-fc82-45a6-9c25-f72970220236\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-073ea9f1-pnfqr"
Oct 30 18:52:28.479129 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:28.478806    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84f528c3-fc82-45a6-9c25-f72970220236-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-073ea9f1-pnfqr\" (UID: \"84f528c3-fc82-45a6-9c25-f72970220236\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-073ea9f1-pnfqr"
Oct 30 18:52:28.479129 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:28.478847    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84f528c3-fc82-45a6-9c25-f72970220236-maas-api-xks-ca\") pod \"maas-api-revoke-sub-073ea9f1-pnfqr\" (UID: \"84f528c3-fc82-45a6-9c25-f72970220236\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-073ea9f1-pnfqr"
Oct 30 18:52:28.479573 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:28.479552    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84f528c3-fc82-45a6-9c25-f72970220236-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-073ea9f1-pnfqr\" (UID: \"84f528c3-fc82-45a6-9c25-f72970220236\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-073ea9f1-pnfqr"
Oct 30 18:52:28.481471 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:28.481453    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84f528c3-fc82-45a6-9c25-f72970220236-maas-api-xks-ca\") pod \"maas-api-revoke-sub-073ea9f1-pnfqr\" (UID: \"84f528c3-fc82-45a6-9c25-f72970220236\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-073ea9f1-pnfqr"
Oct 30 18:52:28.486136 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:28.486116    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rff75\" (UniqueName: \"kubernetes.io/projected/84f528c3-fc82-45a6-9c25-f72970220236-kube-api-access-rff75\") pod \"maas-api-revoke-sub-073ea9f1-pnfqr\" (UID: \"84f528c3-fc82-45a6-9c25-f72970220236\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-073ea9f1-pnfqr"
Oct 30 18:52:28.528544 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:28.528506    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-073ea9f1-pnfqr"
Oct 30 18:52:28.846995 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:52:28.846968    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84f528c3_fc82_45a6_9c25_f72970220236.slice/crio-d37dc32ad28658f5e161d64ddbf8caedfdd7b1ca03e43bc01fe8051d9ce4e60b WatchSource:0}: Error finding container d37dc32ad28658f5e161d64ddbf8caedfdd7b1ca03e43bc01fe8051d9ce4e60b: Status 404 returned error can't find the container with id d37dc32ad28658f5e161d64ddbf8caedfdd7b1ca03e43bc01fe8051d9ce4e60b
Oct 30 18:52:28.847748 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:28.847724    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-073ea9f1-pnfqr"]
Oct 30 18:52:29.303968 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:29.303921    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-073ea9f1-pnfqr" event={"ID":"84f528c3-fc82-45a6-9c25-f72970220236","Type":"ContainerStarted","Data":"53518b6cc290c11fb287151812d2643097bcda6b78886d17ffd452caa2028415"}
Oct 30 18:52:29.304150 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:29.303984    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-073ea9f1-pnfqr" event={"ID":"84f528c3-fc82-45a6-9c25-f72970220236","Type":"ContainerStarted","Data":"d37dc32ad28658f5e161d64ddbf8caedfdd7b1ca03e43bc01fe8051d9ce4e60b"}
Oct 30 18:52:29.305534 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:29.305510    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-541d2bf7-w25bt" event={"ID":"328d84f1-7f29-4e6c-b62d-df7a0709914a","Type":"ContainerStarted","Data":"6bbdef8a51e66f03eceac71630ca234f33ced4c0e637e1c41f5177ab01e5ba93"}
Oct 30 18:52:29.305660 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:29.305548    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-541d2bf7-w25bt" event={"ID":"328d84f1-7f29-4e6c-b62d-df7a0709914a","Type":"ContainerStarted","Data":"f4df21929c841181df4e06e8f297ba98a44cc948871676a6935bd97fc67c204a"}
Oct 30 18:52:29.323928 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:29.323880    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-073ea9f1-pnfqr" podStartSLOduration=1.323867686 podStartE2EDuration="1.323867686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:52:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:29.318228769 +0000 UTC m=+971.943898733" watchObservedRunningTime="2026-10-30 18:52:29.323867686 +0000 UTC m=+971.949537686"
Oct 30 18:52:29.339653 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:29.339609    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-541d2bf7-w25bt" podStartSLOduration=2.339595798 podStartE2EDuration="2.339595798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:52:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:29.336055896 +0000 UTC m=+971.961725860" watchObservedRunningTime="2026-10-30 18:52:29.339595798 +0000 UTC m=+971.965265762"
Oct 30 18:52:30.316286 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:30.316209    4525 generic.go:356] "Generic (PLEG): container finished" podID="328d84f1-7f29-4e6c-b62d-df7a0709914a" containerID="6bbdef8a51e66f03eceac71630ca234f33ced4c0e637e1c41f5177ab01e5ba93" exitCode=0
Oct 30 18:52:30.316661 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:30.316296    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-541d2bf7-w25bt" event={"ID":"328d84f1-7f29-4e6c-b62d-df7a0709914a","Type":"ContainerDied","Data":"6bbdef8a51e66f03eceac71630ca234f33ced4c0e637e1c41f5177ab01e5ba93"}
Oct 30 18:52:30.318108 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:30.318076    4525 generic.go:356] "Generic (PLEG): container finished" podID="84f528c3-fc82-45a6-9c25-f72970220236" containerID="53518b6cc290c11fb287151812d2643097bcda6b78886d17ffd452caa2028415" exitCode=0
Oct 30 18:52:30.318246 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:30.318158    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-073ea9f1-pnfqr" event={"ID":"84f528c3-fc82-45a6-9c25-f72970220236","Type":"ContainerDied","Data":"53518b6cc290c11fb287151812d2643097bcda6b78886d17ffd452caa2028415"}
Oct 30 18:52:31.274444 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:31.274407    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c4ef7c1-bfk6v"]
Oct 30 18:52:31.278999 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:31.278971    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c4ef7c1-bfk6v"
Oct 30 18:52:31.287897 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:31.287870    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c4ef7c1-bfk6v"]
Oct 30 18:52:31.325126 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:31.325095    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46wjc\" (UniqueName: \"kubernetes.io/projected/04ac81d7-49c8-4c5a-bf2d-f6740ac448ba-kube-api-access-46wjc\") pod \"maas-api-revoke-sub-6c4ef7c1-bfk6v\" (UID: \"04ac81d7-49c8-4c5a-bf2d-f6740ac448ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c4ef7c1-bfk6v"
Oct 30 18:52:31.325541 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:31.325265    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04ac81d7-49c8-4c5a-bf2d-f6740ac448ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c4ef7c1-bfk6v\" (UID: \"04ac81d7-49c8-4c5a-bf2d-f6740ac448ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c4ef7c1-bfk6v"
Oct 30 18:52:31.325541 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:31.325289    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04ac81d7-49c8-4c5a-bf2d-f6740ac448ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c4ef7c1-bfk6v\" (UID: \"04ac81d7-49c8-4c5a-bf2d-f6740ac448ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c4ef7c1-bfk6v"
Oct 30 18:52:31.427082 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:31.427052    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04ac81d7-49c8-4c5a-bf2d-f6740ac448ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c4ef7c1-bfk6v\" (UID: \"04ac81d7-49c8-4c5a-bf2d-f6740ac448ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c4ef7c1-bfk6v"
Oct 30 18:52:31.427237 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:31.427097    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04ac81d7-49c8-4c5a-bf2d-f6740ac448ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c4ef7c1-bfk6v\" (UID: \"04ac81d7-49c8-4c5a-bf2d-f6740ac448ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c4ef7c1-bfk6v"
Oct 30 18:52:31.427237 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:31.427229    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46wjc\" (UniqueName: \"kubernetes.io/projected/04ac81d7-49c8-4c5a-bf2d-f6740ac448ba-kube-api-access-46wjc\") pod \"maas-api-revoke-sub-6c4ef7c1-bfk6v\" (UID: \"04ac81d7-49c8-4c5a-bf2d-f6740ac448ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c4ef7c1-bfk6v"
Oct 30 18:52:31.428343 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:31.428268    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04ac81d7-49c8-4c5a-bf2d-f6740ac448ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c4ef7c1-bfk6v\" (UID: \"04ac81d7-49c8-4c5a-bf2d-f6740ac448ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c4ef7c1-bfk6v"
Oct 30 18:52:31.430322 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:31.430294    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04ac81d7-49c8-4c5a-bf2d-f6740ac448ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c4ef7c1-bfk6v\" (UID: \"04ac81d7-49c8-4c5a-bf2d-f6740ac448ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c4ef7c1-bfk6v"
Oct 30 18:52:31.436476 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:31.436428    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46wjc\" (UniqueName: \"kubernetes.io/projected/04ac81d7-49c8-4c5a-bf2d-f6740ac448ba-kube-api-access-46wjc\") pod \"maas-api-revoke-sub-6c4ef7c1-bfk6v\" (UID: \"04ac81d7-49c8-4c5a-bf2d-f6740ac448ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c4ef7c1-bfk6v"
Oct 30 18:52:31.490797 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:31.490767    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-541d2bf7-w25bt"
Oct 30 18:52:31.527982 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:31.527913    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/328d84f1-7f29-4e6c-b62d-df7a0709914a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/328d84f1-7f29-4e6c-b62d-df7a0709914a-maas-api-xks-ca\") pod \"328d84f1-7f29-4e6c-b62d-df7a0709914a\" (UID: \"328d84f1-7f29-4e6c-b62d-df7a0709914a\") "
Oct 30 18:52:31.527982 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:31.527975    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/328d84f1-7f29-4e6c-b62d-df7a0709914a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/328d84f1-7f29-4e6c-b62d-df7a0709914a-maas-api-openshift-ca\") pod \"328d84f1-7f29-4e6c-b62d-df7a0709914a\" (UID: \"328d84f1-7f29-4e6c-b62d-df7a0709914a\") "
Oct 30 18:52:31.528169 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:31.528003    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/328d84f1-7f29-4e6c-b62d-df7a0709914a-kube-api-access-4l8qs\" (UniqueName: \"kubernetes.io/projected/328d84f1-7f29-4e6c-b62d-df7a0709914a-kube-api-access-4l8qs\") pod \"328d84f1-7f29-4e6c-b62d-df7a0709914a\" (UID: \"328d84f1-7f29-4e6c-b62d-df7a0709914a\") "
Oct 30 18:52:31.528377 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:31.528350    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/328d84f1-7f29-4e6c-b62d-df7a0709914a-maas-api-openshift-ca" pod "328d84f1-7f29-4e6c-b62d-df7a0709914a" (UID: "328d84f1-7f29-4e6c-b62d-df7a0709914a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:52:31.528509 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:31.528489    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/328d84f1-7f29-4e6c-b62d-df7a0709914a-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:31.531515 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:31.530825    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/328d84f1-7f29-4e6c-b62d-df7a0709914a-kube-api-access-4l8qs" pod "328d84f1-7f29-4e6c-b62d-df7a0709914a" (UID: "328d84f1-7f29-4e6c-b62d-df7a0709914a"). InnerVolumeSpecName "kube-api-access-4l8qs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:52:31.531515 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:31.531167    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/328d84f1-7f29-4e6c-b62d-df7a0709914a-maas-api-xks-ca" pod "328d84f1-7f29-4e6c-b62d-df7a0709914a" (UID: "328d84f1-7f29-4e6c-b62d-df7a0709914a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:52:31.536041 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:31.536020    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-073ea9f1-pnfqr"
Oct 30 18:52:31.597872 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:31.597834    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c4ef7c1-bfk6v"
Oct 30 18:52:31.629574 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:31.629544    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84f528c3-fc82-45a6-9c25-f72970220236-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84f528c3-fc82-45a6-9c25-f72970220236-maas-api-xks-ca\") pod \"84f528c3-fc82-45a6-9c25-f72970220236\" (UID: \"84f528c3-fc82-45a6-9c25-f72970220236\") "
Oct 30 18:52:31.629761 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:31.629718    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84f528c3-fc82-45a6-9c25-f72970220236-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84f528c3-fc82-45a6-9c25-f72970220236-maas-api-openshift-ca\") pod \"84f528c3-fc82-45a6-9c25-f72970220236\" (UID: \"84f528c3-fc82-45a6-9c25-f72970220236\") "
Oct 30 18:52:31.629761 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:31.629759    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84f528c3-fc82-45a6-9c25-f72970220236-kube-api-access-rff75\" (UniqueName: \"kubernetes.io/projected/84f528c3-fc82-45a6-9c25-f72970220236-kube-api-access-rff75\") pod \"84f528c3-fc82-45a6-9c25-f72970220236\" (UID: \"84f528c3-fc82-45a6-9c25-f72970220236\") "
Oct 30 18:52:31.630263 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:31.630210    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/328d84f1-7f29-4e6c-b62d-df7a0709914a-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:31.630263 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:31.630217    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84f528c3-fc82-45a6-9c25-f72970220236-maas-api-openshift-ca" pod "84f528c3-fc82-45a6-9c25-f72970220236" (UID: "84f528c3-fc82-45a6-9c25-f72970220236"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:52:31.630263 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:31.630236    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4l8qs\" (UniqueName: \"kubernetes.io/projected/328d84f1-7f29-4e6c-b62d-df7a0709914a-kube-api-access-4l8qs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:31.631999 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:31.631969    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84f528c3-fc82-45a6-9c25-f72970220236-maas-api-xks-ca" pod "84f528c3-fc82-45a6-9c25-f72970220236" (UID: "84f528c3-fc82-45a6-9c25-f72970220236"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:52:31.632492 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:31.632464    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84f528c3-fc82-45a6-9c25-f72970220236-kube-api-access-rff75" pod "84f528c3-fc82-45a6-9c25-f72970220236" (UID: "84f528c3-fc82-45a6-9c25-f72970220236"). InnerVolumeSpecName "kube-api-access-rff75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:52:31.731012 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:31.730978    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84f528c3-fc82-45a6-9c25-f72970220236-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:31.731012 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:31.731007    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84f528c3-fc82-45a6-9c25-f72970220236-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:31.731204 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:31.731022    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rff75\" (UniqueName: \"kubernetes.io/projected/84f528c3-fc82-45a6-9c25-f72970220236-kube-api-access-rff75\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:31.883095 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:52:31.883060    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04ac81d7_49c8_4c5a_bf2d_f6740ac448ba.slice/crio-9517f075b31308bf209a400179f7cdf26a3472ffa311f87cb2b110c28fcdaa27 WatchSource:0}: Error finding container 9517f075b31308bf209a400179f7cdf26a3472ffa311f87cb2b110c28fcdaa27: Status 404 returned error can't find the container with id 9517f075b31308bf209a400179f7cdf26a3472ffa311f87cb2b110c28fcdaa27
Oct 30 18:52:31.884047 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:31.884023    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c4ef7c1-bfk6v"]
Oct 30 18:52:32.336968 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:32.336942    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-073ea9f1-pnfqr"
Oct 30 18:52:32.337391 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:32.336978    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-073ea9f1-pnfqr" event={"ID":"84f528c3-fc82-45a6-9c25-f72970220236","Type":"ContainerDied","Data":"d37dc32ad28658f5e161d64ddbf8caedfdd7b1ca03e43bc01fe8051d9ce4e60b"}
Oct 30 18:52:32.337391 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:32.337028    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d37dc32ad28658f5e161d64ddbf8caedfdd7b1ca03e43bc01fe8051d9ce4e60b"
Oct 30 18:52:32.338824 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:32.338803    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c4ef7c1-bfk6v" event={"ID":"04ac81d7-49c8-4c5a-bf2d-f6740ac448ba","Type":"ContainerStarted","Data":"b68366aeb4e054341958f0f94410c05cdd75bef33f89982d47e30f3bfdf350a3"}
Oct 30 18:52:32.338945 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:32.338842    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c4ef7c1-bfk6v" event={"ID":"04ac81d7-49c8-4c5a-bf2d-f6740ac448ba","Type":"ContainerStarted","Data":"9517f075b31308bf209a400179f7cdf26a3472ffa311f87cb2b110c28fcdaa27"}
Oct 30 18:52:32.340235 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:32.340220    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-541d2bf7-w25bt"
Oct 30 18:52:32.340319 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:32.340255    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-541d2bf7-w25bt" event={"ID":"328d84f1-7f29-4e6c-b62d-df7a0709914a","Type":"ContainerDied","Data":"f4df21929c841181df4e06e8f297ba98a44cc948871676a6935bd97fc67c204a"}
Oct 30 18:52:32.340319 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:32.340294    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4df21929c841181df4e06e8f297ba98a44cc948871676a6935bd97fc67c204a"
Oct 30 18:52:32.357077 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:32.357029    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c4ef7c1-bfk6v" podStartSLOduration=1.357016009 podStartE2EDuration="1.357016009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:52:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:32.353056582 +0000 UTC m=+974.978726546" watchObservedRunningTime="2026-10-30 18:52:32.357016009 +0000 UTC m=+974.982685973"
Oct 30 18:52:33.350153 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:33.350115    4525 generic.go:356] "Generic (PLEG): container finished" podID="04ac81d7-49c8-4c5a-bf2d-f6740ac448ba" containerID="b68366aeb4e054341958f0f94410c05cdd75bef33f89982d47e30f3bfdf350a3" exitCode=0
Oct 30 18:52:33.350538 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:33.350167    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c4ef7c1-bfk6v" event={"ID":"04ac81d7-49c8-4c5a-bf2d-f6740ac448ba","Type":"ContainerDied","Data":"b68366aeb4e054341958f0f94410c05cdd75bef33f89982d47e30f3bfdf350a3"}
Oct 30 18:52:34.522406 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:34.522381    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c4ef7c1-bfk6v"
Oct 30 18:52:34.567233 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:34.567194    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04ac81d7-49c8-4c5a-bf2d-f6740ac448ba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04ac81d7-49c8-4c5a-bf2d-f6740ac448ba-maas-api-xks-ca\") pod \"04ac81d7-49c8-4c5a-bf2d-f6740ac448ba\" (UID: \"04ac81d7-49c8-4c5a-bf2d-f6740ac448ba\") "
Oct 30 18:52:34.567424 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:34.567406    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04ac81d7-49c8-4c5a-bf2d-f6740ac448ba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04ac81d7-49c8-4c5a-bf2d-f6740ac448ba-maas-api-openshift-ca\") pod \"04ac81d7-49c8-4c5a-bf2d-f6740ac448ba\" (UID: \"04ac81d7-49c8-4c5a-bf2d-f6740ac448ba\") "
Oct 30 18:52:34.567497 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:34.567481    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04ac81d7-49c8-4c5a-bf2d-f6740ac448ba-kube-api-access-46wjc\" (UniqueName: \"kubernetes.io/projected/04ac81d7-49c8-4c5a-bf2d-f6740ac448ba-kube-api-access-46wjc\") pod \"04ac81d7-49c8-4c5a-bf2d-f6740ac448ba\" (UID: \"04ac81d7-49c8-4c5a-bf2d-f6740ac448ba\") "
Oct 30 18:52:34.567842 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:34.567810    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04ac81d7-49c8-4c5a-bf2d-f6740ac448ba-maas-api-openshift-ca" pod "04ac81d7-49c8-4c5a-bf2d-f6740ac448ba" (UID: "04ac81d7-49c8-4c5a-bf2d-f6740ac448ba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:52:34.568260 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:34.568237    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04ac81d7-49c8-4c5a-bf2d-f6740ac448ba-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:34.569570 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:34.569542    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04ac81d7-49c8-4c5a-bf2d-f6740ac448ba-maas-api-xks-ca" pod "04ac81d7-49c8-4c5a-bf2d-f6740ac448ba" (UID: "04ac81d7-49c8-4c5a-bf2d-f6740ac448ba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:52:34.570128 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:34.570104    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04ac81d7-49c8-4c5a-bf2d-f6740ac448ba-kube-api-access-46wjc" pod "04ac81d7-49c8-4c5a-bf2d-f6740ac448ba" (UID: "04ac81d7-49c8-4c5a-bf2d-f6740ac448ba"). InnerVolumeSpecName "kube-api-access-46wjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:52:34.669484 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:34.669402    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46wjc\" (UniqueName: \"kubernetes.io/projected/04ac81d7-49c8-4c5a-bf2d-f6740ac448ba-kube-api-access-46wjc\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:34.669484 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:34.669462    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04ac81d7-49c8-4c5a-bf2d-f6740ac448ba-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:35.367527 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:35.367487    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c4ef7c1-bfk6v"
Oct 30 18:52:35.367527 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:35.367516    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c4ef7c1-bfk6v" event={"ID":"04ac81d7-49c8-4c5a-bf2d-f6740ac448ba","Type":"ContainerDied","Data":"9517f075b31308bf209a400179f7cdf26a3472ffa311f87cb2b110c28fcdaa27"}
Oct 30 18:52:35.367753 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:35.367575    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9517f075b31308bf209a400179f7cdf26a3472ffa311f87cb2b110c28fcdaa27"
Oct 30 18:52:37.072416 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:37.072373    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2ea8d05-9xc2z"]
Oct 30 18:52:37.073153 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:37.073131    4525 cpu_manager.go:414] "containerMap: removing container" podUID="84f528c3-fc82-45a6-9c25-f72970220236" containerName="revoke-keys"
Oct 30 18:52:37.073258 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:37.073155    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84f528c3-fc82-45a6-9c25-f72970220236" containerName="revoke-keys"
Oct 30 18:52:37.073258 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:37.073183    4525 cpu_manager.go:414] "containerMap: removing container" podUID="328d84f1-7f29-4e6c-b62d-df7a0709914a" containerName="revoke-keys"
Oct 30 18:52:37.073258 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:37.073191    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="328d84f1-7f29-4e6c-b62d-df7a0709914a" containerName="revoke-keys"
Oct 30 18:52:37.073258 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:37.073215    4525 cpu_manager.go:414] "containerMap: removing container" podUID="04ac81d7-49c8-4c5a-bf2d-f6740ac448ba" containerName="revoke-keys"
Oct 30 18:52:37.073258 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:37.073223    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04ac81d7-49c8-4c5a-bf2d-f6740ac448ba" containerName="revoke-keys"
Oct 30 18:52:37.073506 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:37.073351    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="84f528c3-fc82-45a6-9c25-f72970220236" containerName="revoke-keys"
Oct 30 18:52:37.073506 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:37.073367    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="328d84f1-7f29-4e6c-b62d-df7a0709914a" containerName="revoke-keys"
Oct 30 18:52:37.073506 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:37.073380    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="04ac81d7-49c8-4c5a-bf2d-f6740ac448ba" containerName="revoke-keys"
Oct 30 18:52:37.078234 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:37.078210    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2ea8d05-9xc2z"
Oct 30 18:52:37.082484 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:37.082455    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:52:37.083451 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:37.083413    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94hcn\""
Oct 30 18:52:37.087857 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:37.087834    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2ea8d05-9xc2z"]
Oct 30 18:52:37.200852 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:37.200811    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bd81572-1116-4a66-bbc1-a08ce150a2ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2ea8d05-9xc2z\" (UID: \"8bd81572-1116-4a66-bbc1-a08ce150a2ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2ea8d05-9xc2z"
Oct 30 18:52:37.201137 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:37.201107    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bd81572-1116-4a66-bbc1-a08ce150a2ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2ea8d05-9xc2z\" (UID: \"8bd81572-1116-4a66-bbc1-a08ce150a2ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2ea8d05-9xc2z"
Oct 30 18:52:37.201199 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:37.201163    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9bzm\" (UniqueName: \"kubernetes.io/projected/8bd81572-1116-4a66-bbc1-a08ce150a2ba-kube-api-access-q9bzm\") pod \"maas-api-revoke-sub-e2ea8d05-9xc2z\" (UID: \"8bd81572-1116-4a66-bbc1-a08ce150a2ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2ea8d05-9xc2z"
Oct 30 18:52:37.302637 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:37.302599    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bd81572-1116-4a66-bbc1-a08ce150a2ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2ea8d05-9xc2z\" (UID: \"8bd81572-1116-4a66-bbc1-a08ce150a2ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2ea8d05-9xc2z"
Oct 30 18:52:37.302637 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:37.302639    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9bzm\" (UniqueName: \"kubernetes.io/projected/8bd81572-1116-4a66-bbc1-a08ce150a2ba-kube-api-access-q9bzm\") pod \"maas-api-revoke-sub-e2ea8d05-9xc2z\" (UID: \"8bd81572-1116-4a66-bbc1-a08ce150a2ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2ea8d05-9xc2z"
Oct 30 18:52:37.302909 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:37.302734    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bd81572-1116-4a66-bbc1-a08ce150a2ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2ea8d05-9xc2z\" (UID: \"8bd81572-1116-4a66-bbc1-a08ce150a2ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2ea8d05-9xc2z"
Oct 30 18:52:37.303504 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:37.303428    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bd81572-1116-4a66-bbc1-a08ce150a2ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2ea8d05-9xc2z\" (UID: \"8bd81572-1116-4a66-bbc1-a08ce150a2ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2ea8d05-9xc2z"
Oct 30 18:52:37.307620 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:37.307592    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bd81572-1116-4a66-bbc1-a08ce150a2ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2ea8d05-9xc2z\" (UID: \"8bd81572-1116-4a66-bbc1-a08ce150a2ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2ea8d05-9xc2z"
Oct 30 18:52:37.312664 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:37.312638    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9bzm\" (UniqueName: \"kubernetes.io/projected/8bd81572-1116-4a66-bbc1-a08ce150a2ba-kube-api-access-q9bzm\") pod \"maas-api-revoke-sub-e2ea8d05-9xc2z\" (UID: \"8bd81572-1116-4a66-bbc1-a08ce150a2ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2ea8d05-9xc2z"
Oct 30 18:52:37.395471 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:37.395434    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2ea8d05-9xc2z"
Oct 30 18:52:37.744268 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:37.744238    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2ea8d05-9xc2z"]
Oct 30 18:52:37.744506 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:52:37.744473    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bd81572_1116_4a66_bbc1_a08ce150a2ba.slice/crio-2bba2ea8b42362c54b4e55a20a9607d8ba186b90c49234cdaf00de60df329369 WatchSource:0}: Error finding container 2bba2ea8b42362c54b4e55a20a9607d8ba186b90c49234cdaf00de60df329369: Status 404 returned error can't find the container with id 2bba2ea8b42362c54b4e55a20a9607d8ba186b90c49234cdaf00de60df329369
Oct 30 18:52:38.155707 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:38.154793    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-642ba060-xf9kw"]
Oct 30 18:52:38.175745 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:38.171922    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-642ba060-xf9kw"]
Oct 30 18:52:38.175745 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:38.172054    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-642ba060-xf9kw"
Oct 30 18:52:38.252037 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:38.251934    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0d69f71-d7aa-4ab2-94fe-72582bbec250-maas-api-xks-ca\") pod \"maas-api-revoke-sub-642ba060-xf9kw\" (UID: \"f0d69f71-d7aa-4ab2-94fe-72582bbec250\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-642ba060-xf9kw"
Oct 30 18:52:38.252354 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:38.252331    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0d69f71-d7aa-4ab2-94fe-72582bbec250-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-642ba060-xf9kw\" (UID: \"f0d69f71-d7aa-4ab2-94fe-72582bbec250\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-642ba060-xf9kw"
Oct 30 18:52:38.252743 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:38.252675    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx2hp\" (UniqueName: \"kubernetes.io/projected/f0d69f71-d7aa-4ab2-94fe-72582bbec250-kube-api-access-dx2hp\") pod \"maas-api-revoke-sub-642ba060-xf9kw\" (UID: \"f0d69f71-d7aa-4ab2-94fe-72582bbec250\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-642ba060-xf9kw"
Oct 30 18:52:38.354569 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:38.354529    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0d69f71-d7aa-4ab2-94fe-72582bbec250-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-642ba060-xf9kw\" (UID: \"f0d69f71-d7aa-4ab2-94fe-72582bbec250\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-642ba060-xf9kw"
Oct 30 18:52:38.354782 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:38.354674    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx2hp\" (UniqueName: \"kubernetes.io/projected/f0d69f71-d7aa-4ab2-94fe-72582bbec250-kube-api-access-dx2hp\") pod \"maas-api-revoke-sub-642ba060-xf9kw\" (UID: \"f0d69f71-d7aa-4ab2-94fe-72582bbec250\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-642ba060-xf9kw"
Oct 30 18:52:38.354871 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:38.354814    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0d69f71-d7aa-4ab2-94fe-72582bbec250-maas-api-xks-ca\") pod \"maas-api-revoke-sub-642ba060-xf9kw\" (UID: \"f0d69f71-d7aa-4ab2-94fe-72582bbec250\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-642ba060-xf9kw"
Oct 30 18:52:38.356303 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:38.356276    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0d69f71-d7aa-4ab2-94fe-72582bbec250-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-642ba060-xf9kw\" (UID: \"f0d69f71-d7aa-4ab2-94fe-72582bbec250\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-642ba060-xf9kw"
Oct 30 18:52:38.363117 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:38.363049    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0d69f71-d7aa-4ab2-94fe-72582bbec250-maas-api-xks-ca\") pod \"maas-api-revoke-sub-642ba060-xf9kw\" (UID: \"f0d69f71-d7aa-4ab2-94fe-72582bbec250\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-642ba060-xf9kw"
Oct 30 18:52:38.367449 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:38.367404    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx2hp\" (UniqueName: \"kubernetes.io/projected/f0d69f71-d7aa-4ab2-94fe-72582bbec250-kube-api-access-dx2hp\") pod \"maas-api-revoke-sub-642ba060-xf9kw\" (UID: \"f0d69f71-d7aa-4ab2-94fe-72582bbec250\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-642ba060-xf9kw"
Oct 30 18:52:38.403568 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:38.403519    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2ea8d05-9xc2z" event={"ID":"8bd81572-1116-4a66-bbc1-a08ce150a2ba","Type":"ContainerStarted","Data":"c7f26c3ac62fc10ff8dc1446b88fd4f7e3d948f7c66e5b9c2f5c72d11b2cb720"}
Oct 30 18:52:38.403770 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:38.403595    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2ea8d05-9xc2z" event={"ID":"8bd81572-1116-4a66-bbc1-a08ce150a2ba","Type":"ContainerStarted","Data":"2bba2ea8b42362c54b4e55a20a9607d8ba186b90c49234cdaf00de60df329369"}
Oct 30 18:52:38.426720 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:38.426490    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2ea8d05-9xc2z" podStartSLOduration=1.426469917 podStartE2EDuration="1.426469917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:52:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:38.421165 +0000 UTC m=+981.046834965" watchObservedRunningTime="2026-10-30 18:52:38.426469917 +0000 UTC m=+981.052139882"
Oct 30 18:52:38.505625 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:38.504925    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-642ba060-xf9kw"
Oct 30 18:52:39.063830 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:39.062298    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9384137-s9nhl"]
Oct 30 18:52:39.122196 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:39.121895    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9384137-s9nhl"
Oct 30 18:52:39.206248 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:39.202150    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9384137-s9nhl"]
Oct 30 18:52:39.402482 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:39.384598    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/756ed288-3ee5-4f9a-9ae3-114ec19838c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9384137-s9nhl\" (UID: \"756ed288-3ee5-4f9a-9ae3-114ec19838c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9384137-s9nhl"
Oct 30 18:52:39.402482 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:39.385238    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/756ed288-3ee5-4f9a-9ae3-114ec19838c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9384137-s9nhl\" (UID: \"756ed288-3ee5-4f9a-9ae3-114ec19838c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9384137-s9nhl"
Oct 30 18:52:39.402482 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:39.385292    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hjls\" (UniqueName: \"kubernetes.io/projected/756ed288-3ee5-4f9a-9ae3-114ec19838c6-kube-api-access-7hjls\") pod \"maas-api-revoke-sub-b9384137-s9nhl\" (UID: \"756ed288-3ee5-4f9a-9ae3-114ec19838c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9384137-s9nhl"
Oct 30 18:52:39.490837 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:39.485579    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-642ba060-xf9kw" event={"ID":"f0d69f71-d7aa-4ab2-94fe-72582bbec250","Type":"ContainerStarted","Data":"0cb3e35405c32d6b0d4a67e9dfd68173d5b49569f6a8ad73acf55244e18b6ab3"}
Oct 30 18:52:39.490837 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:39.486303    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/756ed288-3ee5-4f9a-9ae3-114ec19838c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9384137-s9nhl\" (UID: \"756ed288-3ee5-4f9a-9ae3-114ec19838c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9384137-s9nhl"
Oct 30 18:52:39.490837 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:39.486461    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/756ed288-3ee5-4f9a-9ae3-114ec19838c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9384137-s9nhl\" (UID: \"756ed288-3ee5-4f9a-9ae3-114ec19838c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9384137-s9nhl"
Oct 30 18:52:39.490837 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:39.486502    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hjls\" (UniqueName: \"kubernetes.io/projected/756ed288-3ee5-4f9a-9ae3-114ec19838c6-kube-api-access-7hjls\") pod \"maas-api-revoke-sub-b9384137-s9nhl\" (UID: \"756ed288-3ee5-4f9a-9ae3-114ec19838c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9384137-s9nhl"
Oct 30 18:52:39.504318 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:39.504254    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/756ed288-3ee5-4f9a-9ae3-114ec19838c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9384137-s9nhl\" (UID: \"756ed288-3ee5-4f9a-9ae3-114ec19838c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9384137-s9nhl"
Oct 30 18:52:39.561708 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:39.551951    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-642ba060-xf9kw"]
Oct 30 18:52:39.619213 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:39.619155    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/756ed288-3ee5-4f9a-9ae3-114ec19838c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9384137-s9nhl\" (UID: \"756ed288-3ee5-4f9a-9ae3-114ec19838c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9384137-s9nhl"
Oct 30 18:52:39.675460 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:39.675380    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hjls\" (UniqueName: \"kubernetes.io/projected/756ed288-3ee5-4f9a-9ae3-114ec19838c6-kube-api-access-7hjls\") pod \"maas-api-revoke-sub-b9384137-s9nhl\" (UID: \"756ed288-3ee5-4f9a-9ae3-114ec19838c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9384137-s9nhl"
Oct 30 18:52:39.858133 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:39.858102    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9384137-s9nhl"
Oct 30 18:52:39.941708 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:39.935892    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr" podUID="82bbbf85-ed93-48d2-9f17-49ac1f31c0a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:52:39.964187 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:39.944622    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts" podUID="be6d4969-285b-4da0-883e-4847f2cbf3af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:52:39.964187 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:39.944911    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj" podUID="84c99c49-1d60-4176-bda4-cad29c7d801c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:52:39.964187 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:39.944622    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv" podUID="595230e3-55f3-43eb-9a7b-6d89250af9b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:52:39.964187 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:39.946621    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw" podUID="ac6852fd-01be-4d69-8c9b-0c591e50b889" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:52:40.318000 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:40.316803    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t" podUID="424fca14-6d8a-4663-86ec-0c91165f34ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:52:40.475609 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:40.474910    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb" podUID="12ebc8b9-c794-47b5-b87b-a63c4bd993e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:52:40.529928 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:40.529887    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-642ba060-xf9kw" event={"ID":"f0d69f71-d7aa-4ab2-94fe-72582bbec250","Type":"ContainerStarted","Data":"589f258e5d1defd5a31215f460fe4236574cc45040b86ffcec0d76eecb6d80b7"}
Oct 30 18:52:40.546235 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:40.546202    4525 generic.go:356] "Generic (PLEG): container finished" podID="8bd81572-1116-4a66-bbc1-a08ce150a2ba" containerID="c7f26c3ac62fc10ff8dc1446b88fd4f7e3d948f7c66e5b9c2f5c72d11b2cb720" exitCode=0
Oct 30 18:52:40.546881 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:40.546590    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2ea8d05-9xc2z" event={"ID":"8bd81572-1116-4a66-bbc1-a08ce150a2ba","Type":"ContainerDied","Data":"c7f26c3ac62fc10ff8dc1446b88fd4f7e3d948f7c66e5b9c2f5c72d11b2cb720"}
Oct 30 18:52:40.813293 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:40.813031    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-642ba060-xf9kw" podStartSLOduration=2.813006255 podStartE2EDuration="2.813006255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:52:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:40.572757568 +0000 UTC m=+983.198427535" watchObservedRunningTime="2026-10-30 18:52:40.813006255 +0000 UTC m=+983.438676220"
Oct 30 18:52:40.996228 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:40.945567    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr" podUID="82bbbf85-ed93-48d2-9f17-49ac1f31c0a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 30 18:52:40.996228 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:40.947809    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr"
Oct 30 18:52:40.996228 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:40.969616    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts" podUID="be6d4969-285b-4da0-883e-4847f2cbf3af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Oct 30 18:52:40.996228 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:40.970492    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj" podUID="84c99c49-1d60-4176-bda4-cad29c7d801c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 18:52:40.996228 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:40.971246    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv" podUID="595230e3-55f3-43eb-9a7b-6d89250af9b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:52:41.023713 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:41.020716    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts"
Oct 30 18:52:41.023713 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:41.020820    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj"
Oct 30 18:52:41.023713 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:41.020885    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv"
Oct 30 18:52:41.274789 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:41.271818    4525 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv" podUID="595230e3-55f3-43eb-9a7b-6d89250af9b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:52:41.407722 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:41.397346    4525 patch_prober.go:29] interesting pod/e2e-worker-w4-0458df-openshift-default-7c86d79f79-zvv2m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 18:52:41.407722 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:41.397434    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-zvv2m" podUID="7c2697b3-2f09-4c2e-a6b6-abd10d93835c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:52:41.438134 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:41.438090    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw" podUID="ac6852fd-01be-4d69-8c9b-0c591e50b889" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 18:52:41.438533 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:41.438512    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw"
Oct 30 18:52:41.633640 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:41.633593    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb" podUID="12ebc8b9-c794-47b5-b87b-a63c4bd993e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 18:52:41.719852 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:41.666172    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t" podUID="424fca14-6d8a-4663-86ec-0c91165f34ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:52:41.955989 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:41.955955    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb"
Oct 30 18:52:41.973589 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:41.973545    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t"
Oct 30 18:52:41.976707 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:41.976655    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv" podUID="595230e3-55f3-43eb-9a7b-6d89250af9b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:52:42.075759 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:42.075717    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj" podUID="84c99c49-1d60-4176-bda4-cad29c7d801c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 18:52:42.098706 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:42.076893    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts" podUID="be6d4969-285b-4da0-883e-4847f2cbf3af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Oct 30 18:52:42.100274 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:42.082919    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr" podUID="82bbbf85-ed93-48d2-9f17-49ac1f31c0a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 30 18:52:42.477880 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:42.477842    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9384137-s9nhl"]
Oct 30 18:52:42.665185 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:42.665145    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb" podUID="12ebc8b9-c794-47b5-b87b-a63c4bd993e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 18:52:42.780242 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:42.780140    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t" podUID="424fca14-6d8a-4663-86ec-0c91165f34ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:52:42.918741 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:42.902166    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9384137-s9nhl" event={"ID":"756ed288-3ee5-4f9a-9ae3-114ec19838c6","Type":"ContainerStarted","Data":"5026ae56a10ff18a4e1e4f79051148c9c54c853e154c891da65db7437c80e1be"}
Oct 30 18:52:42.918741 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:42.780671    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw" podUID="ac6852fd-01be-4d69-8c9b-0c591e50b889" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 18:52:43.020708 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:43.016132    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv" podUID="595230e3-55f3-43eb-9a7b-6d89250af9b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:52:43.113763 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:43.098837    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj" podUID="84c99c49-1d60-4176-bda4-cad29c7d801c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 18:52:43.113763 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:43.108328    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr" podUID="82bbbf85-ed93-48d2-9f17-49ac1f31c0a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 30 18:52:43.113763 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:43.108803    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts" podUID="be6d4969-285b-4da0-883e-4847f2cbf3af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:52:43.128707 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:43.125121    4525 generic.go:356] "Generic (PLEG): container finished" podID="f0d69f71-d7aa-4ab2-94fe-72582bbec250" containerID="589f258e5d1defd5a31215f460fe4236574cc45040b86ffcec0d76eecb6d80b7" exitCode=0
Oct 30 18:52:43.128707 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:43.126052    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-642ba060-xf9kw" event={"ID":"f0d69f71-d7aa-4ab2-94fe-72582bbec250","Type":"ContainerDied","Data":"589f258e5d1defd5a31215f460fe4236574cc45040b86ffcec0d76eecb6d80b7"}
Oct 30 18:52:43.162723 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:43.159944    4525 patch_prober.go:29] interesting pod/e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 18:52:43.162723 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:43.160034    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp" podUID="99aac9ad-b8dc-46cf-ac3f-69a814014bd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:52:43.667717 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:43.667155    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb" podUID="12ebc8b9-c794-47b5-b87b-a63c4bd993e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 18:52:43.949727 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:43.949190    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t" podUID="424fca14-6d8a-4663-86ec-0c91165f34ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:52:43.949727 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:43.949608    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw" podUID="ac6852fd-01be-4d69-8c9b-0c591e50b889" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:52:44.004393 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:44.004359    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv"
Oct 30 18:52:44.176007 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:44.141105    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts" podUID="be6d4969-285b-4da0-883e-4847f2cbf3af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:52:44.176560 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:44.144585    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj" podUID="84c99c49-1d60-4176-bda4-cad29c7d801c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:52:44.177345 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:44.144665    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr" podUID="82bbbf85-ed93-48d2-9f17-49ac1f31c0a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:52:44.307352 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:44.307310    4525 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts" podUID="be6d4969-285b-4da0-883e-4847f2cbf3af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:52:44.496767 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:44.492526    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb"
Oct 30 18:52:44.604082 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:44.603989    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-812953cf-w4g6v"]
Oct 30 18:52:44.809274 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:44.791268    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-812953cf-w4g6v"
Oct 30 18:52:44.809274 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:44.809239    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f19f3cd6-d6f8-4f67-9bc1-76acf8cffb17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-812953cf-w4g6v\" (UID: \"f19f3cd6-d6f8-4f67-9bc1-76acf8cffb17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-812953cf-w4g6v"
Oct 30 18:52:44.809854 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:44.809468    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nrzn\" (UniqueName: \"kubernetes.io/projected/f19f3cd6-d6f8-4f67-9bc1-76acf8cffb17-kube-api-access-4nrzn\") pod \"maas-api-revoke-sub-812953cf-w4g6v\" (UID: \"f19f3cd6-d6f8-4f67-9bc1-76acf8cffb17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-812953cf-w4g6v"
Oct 30 18:52:44.809854 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:44.809584    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f19f3cd6-d6f8-4f67-9bc1-76acf8cffb17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-812953cf-w4g6v\" (UID: \"f19f3cd6-d6f8-4f67-9bc1-76acf8cffb17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-812953cf-w4g6v"
Oct 30 18:52:44.812049 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:44.812027    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj"
Oct 30 18:52:44.817071 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:44.816121    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9384137-s9nhl" event={"ID":"756ed288-3ee5-4f9a-9ae3-114ec19838c6","Type":"ContainerStarted","Data":"dcd1416cb55f1679e4324af778c237c54e4f8702c3de01040cc2da7ea7497767"}
Oct 30 18:52:44.817071 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:44.816216    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t"
Oct 30 18:52:44.817071 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:44.816340    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw"
Oct 30 18:52:44.817071 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:44.816391    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts"
Oct 30 18:52:44.817071 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:44.816438    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr"
Oct 30 18:52:44.818980 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:44.818952    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-812953cf-w4g6v"]
Oct 30 18:52:44.837877 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:44.834752    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-txf2f"]
Oct 30 18:52:44.841989 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:44.841965    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-txf2f"
Oct 30 18:52:44.846826 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:44.846804    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-txf2f"]
Oct 30 18:52:44.863717 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:44.863660    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2ea8d05-9xc2z"
Oct 30 18:52:44.911917 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:44.911884    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bd81572-1116-4a66-bbc1-a08ce150a2ba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bd81572-1116-4a66-bbc1-a08ce150a2ba-maas-api-openshift-ca\") pod \"8bd81572-1116-4a66-bbc1-a08ce150a2ba\" (UID: \"8bd81572-1116-4a66-bbc1-a08ce150a2ba\") "
Oct 30 18:52:44.912573 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:44.912544    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bd81572-1116-4a66-bbc1-a08ce150a2ba-kube-api-access-q9bzm\" (UniqueName: \"kubernetes.io/projected/8bd81572-1116-4a66-bbc1-a08ce150a2ba-kube-api-access-q9bzm\") pod \"8bd81572-1116-4a66-bbc1-a08ce150a2ba\" (UID: \"8bd81572-1116-4a66-bbc1-a08ce150a2ba\") "
Oct 30 18:52:44.912718 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:44.912631    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bd81572-1116-4a66-bbc1-a08ce150a2ba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bd81572-1116-4a66-bbc1-a08ce150a2ba-maas-api-xks-ca\") pod \"8bd81572-1116-4a66-bbc1-a08ce150a2ba\" (UID: \"8bd81572-1116-4a66-bbc1-a08ce150a2ba\") "
Oct 30 18:52:44.912995 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:44.912971    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nrzn\" (UniqueName: \"kubernetes.io/projected/f19f3cd6-d6f8-4f67-9bc1-76acf8cffb17-kube-api-access-4nrzn\") pod \"maas-api-revoke-sub-812953cf-w4g6v\" (UID: \"f19f3cd6-d6f8-4f67-9bc1-76acf8cffb17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-812953cf-w4g6v"
Oct 30 18:52:44.913084 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:44.913050    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f94f5889-fce0-46bc-a055-1c5e5e51d6a6-istio-envoy\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-txf2f\" (UID: \"f94f5889-fce0-46bc-a055-1c5e5e51d6a6\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-txf2f"
Oct 30 18:52:44.913145 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:44.913108    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f19f3cd6-d6f8-4f67-9bc1-76acf8cffb17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-812953cf-w4g6v\" (UID: \"f19f3cd6-d6f8-4f67-9bc1-76acf8cffb17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-812953cf-w4g6v"
Oct 30 18:52:44.913201 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:44.913142    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfptz\" (UniqueName: \"kubernetes.io/projected/f94f5889-fce0-46bc-a055-1c5e5e51d6a6-kube-api-access-jfptz\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-txf2f\" (UID: \"f94f5889-fce0-46bc-a055-1c5e5e51d6a6\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-txf2f"
Oct 30 18:52:44.913259 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:44.913227    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f94f5889-fce0-46bc-a055-1c5e5e51d6a6-workload-certs\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-txf2f\" (UID: \"f94f5889-fce0-46bc-a055-1c5e5e51d6a6\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-txf2f"
Oct 30 18:52:44.913315 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:44.913282    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f94f5889-fce0-46bc-a055-1c5e5e51d6a6-istiod-ca-cert\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-txf2f\" (UID: \"f94f5889-fce0-46bc-a055-1c5e5e51d6a6\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-txf2f"
Oct 30 18:52:44.913436 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:44.913416    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f94f5889-fce0-46bc-a055-1c5e5e51d6a6-workload-socket\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-txf2f\" (UID: \"f94f5889-fce0-46bc-a055-1c5e5e51d6a6\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-txf2f"
Oct 30 18:52:44.913508 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:44.913475    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f19f3cd6-d6f8-4f67-9bc1-76acf8cffb17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-812953cf-w4g6v\" (UID: \"f19f3cd6-d6f8-4f67-9bc1-76acf8cffb17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-812953cf-w4g6v"
Oct 30 18:52:44.913568 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:44.913527    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f94f5889-fce0-46bc-a055-1c5e5e51d6a6-istio-data\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-txf2f\" (UID: \"f94f5889-fce0-46bc-a055-1c5e5e51d6a6\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-txf2f"
Oct 30 18:52:44.913661 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:44.913644    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f94f5889-fce0-46bc-a055-1c5e5e51d6a6-istio-podinfo\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-txf2f\" (UID: \"f94f5889-fce0-46bc-a055-1c5e5e51d6a6\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-txf2f"
Oct 30 18:52:44.913753 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:44.913734    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f94f5889-fce0-46bc-a055-1c5e5e51d6a6-istio-token\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-txf2f\" (UID: \"f94f5889-fce0-46bc-a055-1c5e5e51d6a6\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-txf2f"
Oct 30 18:52:44.913939 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:44.913920    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f94f5889-fce0-46bc-a055-1c5e5e51d6a6-credential-socket\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-txf2f\" (UID: \"f94f5889-fce0-46bc-a055-1c5e5e51d6a6\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-txf2f"
Oct 30 18:52:44.914534 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:44.914508    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bd81572-1116-4a66-bbc1-a08ce150a2ba-maas-api-openshift-ca" pod "8bd81572-1116-4a66-bbc1-a08ce150a2ba" (UID: "8bd81572-1116-4a66-bbc1-a08ce150a2ba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:52:44.934215 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:44.925694    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f19f3cd6-d6f8-4f67-9bc1-76acf8cffb17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-812953cf-w4g6v\" (UID: \"f19f3cd6-d6f8-4f67-9bc1-76acf8cffb17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-812953cf-w4g6v"
Oct 30 18:52:44.935532 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:44.935475    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nrzn\" (UniqueName: \"kubernetes.io/projected/f19f3cd6-d6f8-4f67-9bc1-76acf8cffb17-kube-api-access-4nrzn\") pod \"maas-api-revoke-sub-812953cf-w4g6v\" (UID: \"f19f3cd6-d6f8-4f67-9bc1-76acf8cffb17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-812953cf-w4g6v"
Oct 30 18:52:44.952090 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:44.951999    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f19f3cd6-d6f8-4f67-9bc1-76acf8cffb17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-812953cf-w4g6v\" (UID: \"f19f3cd6-d6f8-4f67-9bc1-76acf8cffb17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-812953cf-w4g6v"
Oct 30 18:52:44.953256 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:44.953218    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bd81572-1116-4a66-bbc1-a08ce150a2ba-maas-api-xks-ca" pod "8bd81572-1116-4a66-bbc1-a08ce150a2ba" (UID: "8bd81572-1116-4a66-bbc1-a08ce150a2ba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:52:44.956011 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:44.955973    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bd81572-1116-4a66-bbc1-a08ce150a2ba-kube-api-access-q9bzm" pod "8bd81572-1116-4a66-bbc1-a08ce150a2ba" (UID: "8bd81572-1116-4a66-bbc1-a08ce150a2ba"). InnerVolumeSpecName "kube-api-access-q9bzm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:52:44.992516 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:44.991756    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9384137-s9nhl" podStartSLOduration=6.991735273 podStartE2EDuration="6.991735273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:52:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:44.954448537 +0000 UTC m=+987.580118497" watchObservedRunningTime="2026-10-30 18:52:44.991735273 +0000 UTC m=+987.617405237"
Oct 30 18:52:45.015667 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:45.015578    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f94f5889-fce0-46bc-a055-1c5e5e51d6a6-istio-envoy\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-txf2f\" (UID: \"f94f5889-fce0-46bc-a055-1c5e5e51d6a6\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-txf2f"
Oct 30 18:52:45.015667 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:45.015659    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfptz\" (UniqueName: \"kubernetes.io/projected/f94f5889-fce0-46bc-a055-1c5e5e51d6a6-kube-api-access-jfptz\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-txf2f\" (UID: \"f94f5889-fce0-46bc-a055-1c5e5e51d6a6\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-txf2f"
Oct 30 18:52:45.015909 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:45.015718    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f94f5889-fce0-46bc-a055-1c5e5e51d6a6-workload-certs\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-txf2f\" (UID: \"f94f5889-fce0-46bc-a055-1c5e5e51d6a6\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-txf2f"
Oct 30 18:52:45.015909 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:45.015755    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f94f5889-fce0-46bc-a055-1c5e5e51d6a6-istiod-ca-cert\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-txf2f\" (UID: \"f94f5889-fce0-46bc-a055-1c5e5e51d6a6\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-txf2f"
Oct 30 18:52:45.015909 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:45.015820    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f94f5889-fce0-46bc-a055-1c5e5e51d6a6-workload-socket\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-txf2f\" (UID: \"f94f5889-fce0-46bc-a055-1c5e5e51d6a6\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-txf2f"
Oct 30 18:52:45.015909 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:45.015855    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f94f5889-fce0-46bc-a055-1c5e5e51d6a6-istio-data\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-txf2f\" (UID: \"f94f5889-fce0-46bc-a055-1c5e5e51d6a6\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-txf2f"
Oct 30 18:52:45.016126 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:45.015912    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f94f5889-fce0-46bc-a055-1c5e5e51d6a6-istio-podinfo\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-txf2f\" (UID: \"f94f5889-fce0-46bc-a055-1c5e5e51d6a6\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-txf2f"
Oct 30 18:52:45.016126 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:45.015936    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f94f5889-fce0-46bc-a055-1c5e5e51d6a6-istio-token\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-txf2f\" (UID: \"f94f5889-fce0-46bc-a055-1c5e5e51d6a6\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-txf2f"
Oct 30 18:52:45.016126 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:45.016041    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f94f5889-fce0-46bc-a055-1c5e5e51d6a6-credential-socket\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-txf2f\" (UID: \"f94f5889-fce0-46bc-a055-1c5e5e51d6a6\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-txf2f"
Oct 30 18:52:45.016285 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:45.016156    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9bzm\" (UniqueName: \"kubernetes.io/projected/8bd81572-1116-4a66-bbc1-a08ce150a2ba-kube-api-access-q9bzm\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:45.016285 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:45.016173    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bd81572-1116-4a66-bbc1-a08ce150a2ba-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:45.016285 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:45.016190    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bd81572-1116-4a66-bbc1-a08ce150a2ba-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:45.016611 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:45.016579    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f94f5889-fce0-46bc-a055-1c5e5e51d6a6-credential-socket\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-txf2f\" (UID: \"f94f5889-fce0-46bc-a055-1c5e5e51d6a6\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-txf2f"
Oct 30 18:52:45.016878 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:45.016858    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f94f5889-fce0-46bc-a055-1c5e5e51d6a6-workload-socket\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-txf2f\" (UID: \"f94f5889-fce0-46bc-a055-1c5e5e51d6a6\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-txf2f"
Oct 30 18:52:45.017118 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:45.017098    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f94f5889-fce0-46bc-a055-1c5e5e51d6a6-istio-data\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-txf2f\" (UID: \"f94f5889-fce0-46bc-a055-1c5e5e51d6a6\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-txf2f"
Oct 30 18:52:45.018237 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:45.017905    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f94f5889-fce0-46bc-a055-1c5e5e51d6a6-istiod-ca-cert\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-txf2f\" (UID: \"f94f5889-fce0-46bc-a055-1c5e5e51d6a6\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-txf2f"
Oct 30 18:52:45.018237 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:45.018179    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f94f5889-fce0-46bc-a055-1c5e5e51d6a6-workload-certs\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-txf2f\" (UID: \"f94f5889-fce0-46bc-a055-1c5e5e51d6a6\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-txf2f"
Oct 30 18:52:45.019614 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:45.019594    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f94f5889-fce0-46bc-a055-1c5e5e51d6a6-istio-envoy\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-txf2f\" (UID: \"f94f5889-fce0-46bc-a055-1c5e5e51d6a6\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-txf2f"
Oct 30 18:52:45.020263 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:45.020222    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f94f5889-fce0-46bc-a055-1c5e5e51d6a6-istio-podinfo\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-txf2f\" (UID: \"f94f5889-fce0-46bc-a055-1c5e5e51d6a6\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-txf2f"
Oct 30 18:52:45.026449 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:45.026404    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfptz\" (UniqueName: \"kubernetes.io/projected/f94f5889-fce0-46bc-a055-1c5e5e51d6a6-kube-api-access-jfptz\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-txf2f\" (UID: \"f94f5889-fce0-46bc-a055-1c5e5e51d6a6\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-txf2f"
Oct 30 18:52:45.026968 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:45.026929    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f94f5889-fce0-46bc-a055-1c5e5e51d6a6-istio-token\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-txf2f\" (UID: \"f94f5889-fce0-46bc-a055-1c5e5e51d6a6\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-txf2f"
Oct 30 18:52:45.048291 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:45.048270    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-642ba060-xf9kw"
Oct 30 18:52:45.116546 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:45.116517    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0d69f71-d7aa-4ab2-94fe-72582bbec250-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0d69f71-d7aa-4ab2-94fe-72582bbec250-maas-api-openshift-ca\") pod \"f0d69f71-d7aa-4ab2-94fe-72582bbec250\" (UID: \"f0d69f71-d7aa-4ab2-94fe-72582bbec250\") "
Oct 30 18:52:45.116746 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:45.116653    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0d69f71-d7aa-4ab2-94fe-72582bbec250-kube-api-access-dx2hp\" (UniqueName: \"kubernetes.io/projected/f0d69f71-d7aa-4ab2-94fe-72582bbec250-kube-api-access-dx2hp\") pod \"f0d69f71-d7aa-4ab2-94fe-72582bbec250\" (UID: \"f0d69f71-d7aa-4ab2-94fe-72582bbec250\") "
Oct 30 18:52:45.116817 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:45.116794    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0d69f71-d7aa-4ab2-94fe-72582bbec250-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0d69f71-d7aa-4ab2-94fe-72582bbec250-maas-api-xks-ca\") pod \"f0d69f71-d7aa-4ab2-94fe-72582bbec250\" (UID: \"f0d69f71-d7aa-4ab2-94fe-72582bbec250\") "
Oct 30 18:52:45.116908 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:45.116886    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0d69f71-d7aa-4ab2-94fe-72582bbec250-maas-api-openshift-ca" pod "f0d69f71-d7aa-4ab2-94fe-72582bbec250" (UID: "f0d69f71-d7aa-4ab2-94fe-72582bbec250"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:52:45.117196 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:45.117180    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0d69f71-d7aa-4ab2-94fe-72582bbec250-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:45.118916 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:45.118891    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0d69f71-d7aa-4ab2-94fe-72582bbec250-kube-api-access-dx2hp" pod "f0d69f71-d7aa-4ab2-94fe-72582bbec250" (UID: "f0d69f71-d7aa-4ab2-94fe-72582bbec250"). InnerVolumeSpecName "kube-api-access-dx2hp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:52:45.119026 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:45.119002    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0d69f71-d7aa-4ab2-94fe-72582bbec250-maas-api-xks-ca" pod "f0d69f71-d7aa-4ab2-94fe-72582bbec250" (UID: "f0d69f71-d7aa-4ab2-94fe-72582bbec250"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:52:45.148260 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:45.148227    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-812953cf-w4g6v"
Oct 30 18:52:45.195590 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:45.195557    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-txf2f"
Oct 30 18:52:45.219806 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:45.219777    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dx2hp\" (UniqueName: \"kubernetes.io/projected/f0d69f71-d7aa-4ab2-94fe-72582bbec250-kube-api-access-dx2hp\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:45.219911 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:45.219814    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0d69f71-d7aa-4ab2-94fe-72582bbec250-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:45.431174 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:52:45.431136    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf19f3cd6_d6f8_4f67_9bc1_76acf8cffb17.slice/crio-dd02981c0994d70d087b6fd0e8e3f625f4bed506c1f94c266b56fca56cc4f063 WatchSource:0}: Error finding container dd02981c0994d70d087b6fd0e8e3f625f4bed506c1f94c266b56fca56cc4f063: Status 404 returned error can't find the container with id dd02981c0994d70d087b6fd0e8e3f625f4bed506c1f94c266b56fca56cc4f063
Oct 30 18:52:45.432184 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:45.432154    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-812953cf-w4g6v"]
Oct 30 18:52:45.497446 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:52:45.497414    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf94f5889_fce0_46bc_a055_1c5e5e51d6a6.slice/crio-05db23b54e8cf57cda893d1ba6028c2273f24b744e3ea47f2e907e2553054563 WatchSource:0}: Error finding container 05db23b54e8cf57cda893d1ba6028c2273f24b744e3ea47f2e907e2553054563: Status 404 returned error can't find the container with id 05db23b54e8cf57cda893d1ba6028c2273f24b744e3ea47f2e907e2553054563
Oct 30 18:52:45.498125 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:45.498057    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-txf2f"]
Oct 30 18:52:45.500007 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:45.499978    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:52:45.500097 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:45.500038    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:52:45.500097 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:45.500065    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:52:45.827063 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:45.826940    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-txf2f" event={"ID":"f94f5889-fce0-46bc-a055-1c5e5e51d6a6","Type":"ContainerStarted","Data":"bc2cbbfe6b46c07059660f1c6a406daa26774e90ab7a95fb4c51c6ab06012878"}
Oct 30 18:52:45.827655 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:45.827609    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-txf2f" event={"ID":"f94f5889-fce0-46bc-a055-1c5e5e51d6a6","Type":"ContainerStarted","Data":"05db23b54e8cf57cda893d1ba6028c2273f24b744e3ea47f2e907e2553054563"}
Oct 30 18:52:45.828977 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:45.828952    4525 generic.go:356] "Generic (PLEG): container finished" podID="756ed288-3ee5-4f9a-9ae3-114ec19838c6" containerID="dcd1416cb55f1679e4324af778c237c54e4f8702c3de01040cc2da7ea7497767" exitCode=0
Oct 30 18:52:45.829115 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:45.829095    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9384137-s9nhl" event={"ID":"756ed288-3ee5-4f9a-9ae3-114ec19838c6","Type":"ContainerDied","Data":"dcd1416cb55f1679e4324af778c237c54e4f8702c3de01040cc2da7ea7497767"}
Oct 30 18:52:45.830805 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:45.830775    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-642ba060-xf9kw"
Oct 30 18:52:45.830916 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:45.830787    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-642ba060-xf9kw" event={"ID":"f0d69f71-d7aa-4ab2-94fe-72582bbec250","Type":"ContainerDied","Data":"0cb3e35405c32d6b0d4a67e9dfd68173d5b49569f6a8ad73acf55244e18b6ab3"}
Oct 30 18:52:45.830981 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:45.830923    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0cb3e35405c32d6b0d4a67e9dfd68173d5b49569f6a8ad73acf55244e18b6ab3"
Oct 30 18:52:45.833224 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:45.832868    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-812953cf-w4g6v" event={"ID":"f19f3cd6-d6f8-4f67-9bc1-76acf8cffb17","Type":"ContainerStarted","Data":"e965e38955c96400cbd2ab3af5ee488d9ca95d19f24e3b9bc585667d04b75b07"}
Oct 30 18:52:45.833224 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:45.832939    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-812953cf-w4g6v" event={"ID":"f19f3cd6-d6f8-4f67-9bc1-76acf8cffb17","Type":"ContainerStarted","Data":"dd02981c0994d70d087b6fd0e8e3f625f4bed506c1f94c266b56fca56cc4f063"}
Oct 30 18:52:45.834695 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:45.834655    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2ea8d05-9xc2z" event={"ID":"8bd81572-1116-4a66-bbc1-a08ce150a2ba","Type":"ContainerDied","Data":"2bba2ea8b42362c54b4e55a20a9607d8ba186b90c49234cdaf00de60df329369"}
Oct 30 18:52:45.834784 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:45.834734    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2bba2ea8b42362c54b4e55a20a9607d8ba186b90c49234cdaf00de60df329369"
Oct 30 18:52:45.834784 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:45.834667    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2ea8d05-9xc2z"
Oct 30 18:52:45.852001 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:45.851956    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-txf2f" podStartSLOduration=2.851939505 podStartE2EDuration="2.851939505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:52:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:45.84869887 +0000 UTC m=+988.474368834" watchObservedRunningTime="2026-10-30 18:52:45.851939505 +0000 UTC m=+988.477609469"
Oct 30 18:52:45.909195 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:45.892383    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-812953cf-w4g6v" podStartSLOduration=2.892365393 podStartE2EDuration="2.892365393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:52:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:45.883838299 +0000 UTC m=+988.509508263" watchObservedRunningTime="2026-10-30 18:52:45.892365393 +0000 UTC m=+988.518035358"
Oct 30 18:52:46.213271 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:46.211166    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-txf2f"
Oct 30 18:52:46.214707 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:46.213834    4525 patch_prober.go:29] interesting pod/e2e-worker-w0-9de872-openshift-default-77cf8df555-txf2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 30 18:52:46.214707 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:46.213914    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-txf2f" podUID="f94f5889-fce0-46bc-a055-1c5e5e51d6a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 30 18:52:46.844410 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:46.844378    4525 generic.go:356] "Generic (PLEG): container finished" podID="f19f3cd6-d6f8-4f67-9bc1-76acf8cffb17" containerID="e965e38955c96400cbd2ab3af5ee488d9ca95d19f24e3b9bc585667d04b75b07" exitCode=0
Oct 30 18:52:46.844920 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:46.844460    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-812953cf-w4g6v" event={"ID":"f19f3cd6-d6f8-4f67-9bc1-76acf8cffb17","Type":"ContainerDied","Data":"e965e38955c96400cbd2ab3af5ee488d9ca95d19f24e3b9bc585667d04b75b07"}
Oct 30 18:52:47.061829 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:47.061806    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9384137-s9nhl"
Oct 30 18:52:47.145617 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:47.145587    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/756ed288-3ee5-4f9a-9ae3-114ec19838c6-kube-api-access-7hjls\" (UniqueName: \"kubernetes.io/projected/756ed288-3ee5-4f9a-9ae3-114ec19838c6-kube-api-access-7hjls\") pod \"756ed288-3ee5-4f9a-9ae3-114ec19838c6\" (UID: \"756ed288-3ee5-4f9a-9ae3-114ec19838c6\") "
Oct 30 18:52:47.145790 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:47.145640    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/756ed288-3ee5-4f9a-9ae3-114ec19838c6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/756ed288-3ee5-4f9a-9ae3-114ec19838c6-maas-api-openshift-ca\") pod \"756ed288-3ee5-4f9a-9ae3-114ec19838c6\" (UID: \"756ed288-3ee5-4f9a-9ae3-114ec19838c6\") "
Oct 30 18:52:47.145790 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:47.145728    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/756ed288-3ee5-4f9a-9ae3-114ec19838c6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/756ed288-3ee5-4f9a-9ae3-114ec19838c6-maas-api-xks-ca\") pod \"756ed288-3ee5-4f9a-9ae3-114ec19838c6\" (UID: \"756ed288-3ee5-4f9a-9ae3-114ec19838c6\") "
Oct 30 18:52:47.146028 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:47.146005    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/756ed288-3ee5-4f9a-9ae3-114ec19838c6-maas-api-openshift-ca" pod "756ed288-3ee5-4f9a-9ae3-114ec19838c6" (UID: "756ed288-3ee5-4f9a-9ae3-114ec19838c6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:52:47.146245 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:47.146230    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/756ed288-3ee5-4f9a-9ae3-114ec19838c6-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:47.147977 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:47.147952    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/756ed288-3ee5-4f9a-9ae3-114ec19838c6-kube-api-access-7hjls" pod "756ed288-3ee5-4f9a-9ae3-114ec19838c6" (UID: "756ed288-3ee5-4f9a-9ae3-114ec19838c6"). InnerVolumeSpecName "kube-api-access-7hjls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:52:47.148080 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:47.148008    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/756ed288-3ee5-4f9a-9ae3-114ec19838c6-maas-api-xks-ca" pod "756ed288-3ee5-4f9a-9ae3-114ec19838c6" (UID: "756ed288-3ee5-4f9a-9ae3-114ec19838c6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:52:47.201717 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:47.201184    4525 patch_prober.go:29] interesting pod/e2e-worker-w0-9de872-openshift-default-77cf8df555-txf2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 30 18:52:47.201717 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:47.201264    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-txf2f" podUID="f94f5889-fce0-46bc-a055-1c5e5e51d6a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 30 18:52:47.247798 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:47.247767    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hjls\" (UniqueName: \"kubernetes.io/projected/756ed288-3ee5-4f9a-9ae3-114ec19838c6-kube-api-access-7hjls\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:47.247798 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:47.247798    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/756ed288-3ee5-4f9a-9ae3-114ec19838c6-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:47.425071 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:52:47.424986    4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d49l" containerName="extractor"
Oct 30 18:52:47.855582 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:47.855561    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9384137-s9nhl"
Oct 30 18:52:47.856104 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:47.855601    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9384137-s9nhl" event={"ID":"756ed288-3ee5-4f9a-9ae3-114ec19838c6","Type":"ContainerDied","Data":"5026ae56a10ff18a4e1e4f79051148c9c54c853e154c891da65db7437c80e1be"}
Oct 30 18:52:47.856104 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:47.855647    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5026ae56a10ff18a4e1e4f79051148c9c54c853e154c891da65db7437c80e1be"
Oct 30 18:52:48.121502 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:48.121426    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-812953cf-w4g6v"
Oct 30 18:52:48.166159 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:48.166120    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f19f3cd6-d6f8-4f67-9bc1-76acf8cffb17-kube-api-access-4nrzn\" (UniqueName: \"kubernetes.io/projected/f19f3cd6-d6f8-4f67-9bc1-76acf8cffb17-kube-api-access-4nrzn\") pod \"f19f3cd6-d6f8-4f67-9bc1-76acf8cffb17\" (UID: \"f19f3cd6-d6f8-4f67-9bc1-76acf8cffb17\") "
Oct 30 18:52:48.166339 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:48.166220    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f19f3cd6-d6f8-4f67-9bc1-76acf8cffb17-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f19f3cd6-d6f8-4f67-9bc1-76acf8cffb17-maas-api-xks-ca\") pod \"f19f3cd6-d6f8-4f67-9bc1-76acf8cffb17\" (UID: \"f19f3cd6-d6f8-4f67-9bc1-76acf8cffb17\") "
Oct 30 18:52:48.166339 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:48.166261    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f19f3cd6-d6f8-4f67-9bc1-76acf8cffb17-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f19f3cd6-d6f8-4f67-9bc1-76acf8cffb17-maas-api-openshift-ca\") pod \"f19f3cd6-d6f8-4f67-9bc1-76acf8cffb17\" (UID: \"f19f3cd6-d6f8-4f67-9bc1-76acf8cffb17\") "
Oct 30 18:52:48.166919 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:48.166888    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f19f3cd6-d6f8-4f67-9bc1-76acf8cffb17-maas-api-openshift-ca" pod "f19f3cd6-d6f8-4f67-9bc1-76acf8cffb17" (UID: "f19f3cd6-d6f8-4f67-9bc1-76acf8cffb17"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:52:48.168903 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:48.168877    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f19f3cd6-d6f8-4f67-9bc1-76acf8cffb17-maas-api-xks-ca" pod "f19f3cd6-d6f8-4f67-9bc1-76acf8cffb17" (UID: "f19f3cd6-d6f8-4f67-9bc1-76acf8cffb17"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:52:48.170292 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:48.170261    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f19f3cd6-d6f8-4f67-9bc1-76acf8cffb17-kube-api-access-4nrzn" pod "f19f3cd6-d6f8-4f67-9bc1-76acf8cffb17" (UID: "f19f3cd6-d6f8-4f67-9bc1-76acf8cffb17"). InnerVolumeSpecName "kube-api-access-4nrzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:52:48.207515 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:48.207144    4525 patch_prober.go:29] interesting pod/e2e-worker-w0-9de872-openshift-default-77cf8df555-txf2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 30 18:52:48.207515 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:48.207208    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-txf2f" podUID="f94f5889-fce0-46bc-a055-1c5e5e51d6a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 30 18:52:48.268217 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:48.268185    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f19f3cd6-d6f8-4f67-9bc1-76acf8cffb17-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:48.268217 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:48.268216    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f19f3cd6-d6f8-4f67-9bc1-76acf8cffb17-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:48.268431 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:48.268229    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nrzn\" (UniqueName: \"kubernetes.io/projected/f19f3cd6-d6f8-4f67-9bc1-76acf8cffb17-kube-api-access-4nrzn\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:52:48.866221 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:48.866188    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-812953cf-w4g6v"
Oct 30 18:52:48.866221 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:48.866212    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-812953cf-w4g6v" event={"ID":"f19f3cd6-d6f8-4f67-9bc1-76acf8cffb17","Type":"ContainerDied","Data":"dd02981c0994d70d087b6fd0e8e3f625f4bed506c1f94c266b56fca56cc4f063"}
Oct 30 18:52:48.866698 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:48.866266    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd02981c0994d70d087b6fd0e8e3f625f4bed506c1f94c266b56fca56cc4f063"
Oct 30 18:52:49.200901 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:49.200816    4525 patch_prober.go:29] interesting pod/e2e-worker-w0-9de872-openshift-default-77cf8df555-txf2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 30 18:52:49.200901 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:49.200882    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-txf2f" podUID="f94f5889-fce0-46bc-a055-1c5e5e51d6a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 30 18:52:50.199708 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:50.198040    4525 patch_prober.go:29] interesting pod/e2e-worker-w0-9de872-openshift-default-77cf8df555-txf2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 30 18:52:50.199708 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:50.198118    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-txf2f" podUID="f94f5889-fce0-46bc-a055-1c5e5e51d6a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 30 18:52:51.197708 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:51.196038    4525 patch_prober.go:29] interesting pod/e2e-worker-w0-9de872-openshift-default-77cf8df555-txf2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 30 18:52:51.197708 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:51.196101    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-txf2f" podUID="f94f5889-fce0-46bc-a055-1c5e5e51d6a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 30 18:52:51.856479 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:51.856442    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-87293f-openshift-default-7c9c9cbc7f-vw6fm"]
Oct 30 18:52:51.857305 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:51.857281    4525 cpu_manager.go:414] "containerMap: removing container" podUID="f19f3cd6-d6f8-4f67-9bc1-76acf8cffb17" containerName="revoke-keys"
Oct 30 18:52:51.857394 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:51.857306    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f19f3cd6-d6f8-4f67-9bc1-76acf8cffb17" containerName="revoke-keys"
Oct 30 18:52:51.857394 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:51.857321    4525 cpu_manager.go:414] "containerMap: removing container" podUID="8bd81572-1116-4a66-bbc1-a08ce150a2ba" containerName="revoke-keys"
Oct 30 18:52:51.857394 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:51.857328    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bd81572-1116-4a66-bbc1-a08ce150a2ba" containerName="revoke-keys"
Oct 30 18:52:51.857394 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:51.857370    4525 cpu_manager.go:414] "containerMap: removing container" podUID="f0d69f71-d7aa-4ab2-94fe-72582bbec250" containerName="revoke-keys"
Oct 30 18:52:51.857394 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:51.857378    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0d69f71-d7aa-4ab2-94fe-72582bbec250" containerName="revoke-keys"
Oct 30 18:52:51.857394 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:51.857390    4525 cpu_manager.go:414] "containerMap: removing container" podUID="756ed288-3ee5-4f9a-9ae3-114ec19838c6" containerName="revoke-keys"
Oct 30 18:52:51.857766 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:51.857397    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="756ed288-3ee5-4f9a-9ae3-114ec19838c6" containerName="revoke-keys"
Oct 30 18:52:51.857766 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:51.857538    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0d69f71-d7aa-4ab2-94fe-72582bbec250" containerName="revoke-keys"
Oct 30 18:52:51.857766 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:51.857549    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="756ed288-3ee5-4f9a-9ae3-114ec19838c6" containerName="revoke-keys"
Oct 30 18:52:51.857766 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:51.857566    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="f19f3cd6-d6f8-4f67-9bc1-76acf8cffb17" containerName="revoke-keys"
Oct 30 18:52:51.857766 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:51.857575    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bd81572-1116-4a66-bbc1-a08ce150a2ba" containerName="revoke-keys"
Oct 30 18:52:51.862155 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:51.862133    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-87293f-openshift-default-7c9c9cbc7f-vw6fm"
Oct 30 18:52:51.883016 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:51.882977    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-87293f-openshift-default-dockercfg-qmjzk\""
Oct 30 18:52:51.886392 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:51.886369    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-87293f-openshift-default-7c9c9cbc7f-vw6fm"]
Oct 30 18:52:51.941498 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:51.941448    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf796179-ce62-433b-87b8-a8bb1405c70b-istio-envoy\") pod \"e2e-api-a-87293f-openshift-default-7c9c9cbc7f-vw6fm\" (UID: \"cf796179-ce62-433b-87b8-a8bb1405c70b\") " pod="openshift-ingress/e2e-api-a-87293f-openshift-default-7c9c9cbc7f-vw6fm"
Oct 30 18:52:51.941671 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:51.941508    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf796179-ce62-433b-87b8-a8bb1405c70b-istiod-ca-cert\") pod \"e2e-api-a-87293f-openshift-default-7c9c9cbc7f-vw6fm\" (UID: \"cf796179-ce62-433b-87b8-a8bb1405c70b\") " pod="openshift-ingress/e2e-api-a-87293f-openshift-default-7c9c9cbc7f-vw6fm"
Oct 30 18:52:51.941671 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:51.941540    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf796179-ce62-433b-87b8-a8bb1405c70b-credential-socket\") pod \"e2e-api-a-87293f-openshift-default-7c9c9cbc7f-vw6fm\" (UID: \"cf796179-ce62-433b-87b8-a8bb1405c70b\") " pod="openshift-ingress/e2e-api-a-87293f-openshift-default-7c9c9cbc7f-vw6fm"
Oct 30 18:52:51.941795 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:51.941718    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf796179-ce62-433b-87b8-a8bb1405c70b-workload-certs\") pod \"e2e-api-a-87293f-openshift-default-7c9c9cbc7f-vw6fm\" (UID: \"cf796179-ce62-433b-87b8-a8bb1405c70b\") " pod="openshift-ingress/e2e-api-a-87293f-openshift-default-7c9c9cbc7f-vw6fm"
Oct 30 18:52:51.941795 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:51.941768    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf796179-ce62-433b-87b8-a8bb1405c70b-istio-podinfo\") pod \"e2e-api-a-87293f-openshift-default-7c9c9cbc7f-vw6fm\" (UID: \"cf796179-ce62-433b-87b8-a8bb1405c70b\") " pod="openshift-ingress/e2e-api-a-87293f-openshift-default-7c9c9cbc7f-vw6fm"
Oct 30 18:52:51.941864 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:51.941792    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtwft\" (UniqueName: \"kubernetes.io/projected/cf796179-ce62-433b-87b8-a8bb1405c70b-kube-api-access-vtwft\") pod \"e2e-api-a-87293f-openshift-default-7c9c9cbc7f-vw6fm\" (UID: \"cf796179-ce62-433b-87b8-a8bb1405c70b\") " pod="openshift-ingress/e2e-api-a-87293f-openshift-default-7c9c9cbc7f-vw6fm"
Oct 30 18:52:51.941864 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:51.941820    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf796179-ce62-433b-87b8-a8bb1405c70b-istio-data\") pod \"e2e-api-a-87293f-openshift-default-7c9c9cbc7f-vw6fm\" (UID: \"cf796179-ce62-433b-87b8-a8bb1405c70b\") " pod="openshift-ingress/e2e-api-a-87293f-openshift-default-7c9c9cbc7f-vw6fm"
Oct 30 18:52:51.941933 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:51.941882    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf796179-ce62-433b-87b8-a8bb1405c70b-workload-socket\") pod \"e2e-api-a-87293f-openshift-default-7c9c9cbc7f-vw6fm\" (UID: \"cf796179-ce62-433b-87b8-a8bb1405c70b\") " pod="openshift-ingress/e2e-api-a-87293f-openshift-default-7c9c9cbc7f-vw6fm"
Oct 30 18:52:51.942014 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:51.941950    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf796179-ce62-433b-87b8-a8bb1405c70b-istio-token\") pod \"e2e-api-a-87293f-openshift-default-7c9c9cbc7f-vw6fm\" (UID: \"cf796179-ce62-433b-87b8-a8bb1405c70b\") " pod="openshift-ingress/e2e-api-a-87293f-openshift-default-7c9c9cbc7f-vw6fm"
Oct 30 18:52:52.059835 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:52.059800    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf796179-ce62-433b-87b8-a8bb1405c70b-workload-socket\") pod \"e2e-api-a-87293f-openshift-default-7c9c9cbc7f-vw6fm\" (UID: \"cf796179-ce62-433b-87b8-a8bb1405c70b\") " pod="openshift-ingress/e2e-api-a-87293f-openshift-default-7c9c9cbc7f-vw6fm"
Oct 30 18:52:52.060140 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:52.060119    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf796179-ce62-433b-87b8-a8bb1405c70b-istio-token\") pod \"e2e-api-a-87293f-openshift-default-7c9c9cbc7f-vw6fm\" (UID: \"cf796179-ce62-433b-87b8-a8bb1405c70b\") " pod="openshift-ingress/e2e-api-a-87293f-openshift-default-7c9c9cbc7f-vw6fm"
Oct 30 18:52:52.060315 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:52.060300    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf796179-ce62-433b-87b8-a8bb1405c70b-istio-envoy\") pod \"e2e-api-a-87293f-openshift-default-7c9c9cbc7f-vw6fm\" (UID: \"cf796179-ce62-433b-87b8-a8bb1405c70b\") " pod="openshift-ingress/e2e-api-a-87293f-openshift-default-7c9c9cbc7f-vw6fm"
Oct 30 18:52:52.060425 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:52.060411    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf796179-ce62-433b-87b8-a8bb1405c70b-istiod-ca-cert\") pod \"e2e-api-a-87293f-openshift-default-7c9c9cbc7f-vw6fm\" (UID: \"cf796179-ce62-433b-87b8-a8bb1405c70b\") " pod="openshift-ingress/e2e-api-a-87293f-openshift-default-7c9c9cbc7f-vw6fm"
Oct 30 18:52:52.060531 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:52.060517    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf796179-ce62-433b-87b8-a8bb1405c70b-credential-socket\") pod \"e2e-api-a-87293f-openshift-default-7c9c9cbc7f-vw6fm\" (UID: \"cf796179-ce62-433b-87b8-a8bb1405c70b\") " pod="openshift-ingress/e2e-api-a-87293f-openshift-default-7c9c9cbc7f-vw6fm"
Oct 30 18:52:52.060841 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:52.060805    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf796179-ce62-433b-87b8-a8bb1405c70b-workload-certs\") pod \"e2e-api-a-87293f-openshift-default-7c9c9cbc7f-vw6fm\" (UID: \"cf796179-ce62-433b-87b8-a8bb1405c70b\") " pod="openshift-ingress/e2e-api-a-87293f-openshift-default-7c9c9cbc7f-vw6fm"
Oct 30 18:52:52.061032 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:52.061012    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf796179-ce62-433b-87b8-a8bb1405c70b-istio-podinfo\") pod \"e2e-api-a-87293f-openshift-default-7c9c9cbc7f-vw6fm\" (UID: \"cf796179-ce62-433b-87b8-a8bb1405c70b\") " pod="openshift-ingress/e2e-api-a-87293f-openshift-default-7c9c9cbc7f-vw6fm"
Oct 30 18:52:52.061391 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:52.061181    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf796179-ce62-433b-87b8-a8bb1405c70b-workload-socket\") pod \"e2e-api-a-87293f-openshift-default-7c9c9cbc7f-vw6fm\" (UID: \"cf796179-ce62-433b-87b8-a8bb1405c70b\") " pod="openshift-ingress/e2e-api-a-87293f-openshift-default-7c9c9cbc7f-vw6fm"
Oct 30 18:52:52.061391 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:52.061233    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtwft\" (UniqueName: \"kubernetes.io/projected/cf796179-ce62-433b-87b8-a8bb1405c70b-kube-api-access-vtwft\") pod \"e2e-api-a-87293f-openshift-default-7c9c9cbc7f-vw6fm\" (UID: \"cf796179-ce62-433b-87b8-a8bb1405c70b\") " pod="openshift-ingress/e2e-api-a-87293f-openshift-default-7c9c9cbc7f-vw6fm"
Oct 30 18:52:52.061391 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:52.061322    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf796179-ce62-433b-87b8-a8bb1405c70b-istio-data\") pod \"e2e-api-a-87293f-openshift-default-7c9c9cbc7f-vw6fm\" (UID: \"cf796179-ce62-433b-87b8-a8bb1405c70b\") " pod="openshift-ingress/e2e-api-a-87293f-openshift-default-7c9c9cbc7f-vw6fm"
Oct 30 18:52:52.062140 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:52.061439    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf796179-ce62-433b-87b8-a8bb1405c70b-credential-socket\") pod \"e2e-api-a-87293f-openshift-default-7c9c9cbc7f-vw6fm\" (UID: \"cf796179-ce62-433b-87b8-a8bb1405c70b\") " pod="openshift-ingress/e2e-api-a-87293f-openshift-default-7c9c9cbc7f-vw6fm"
Oct 30 18:52:52.066642 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:52.066615    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf796179-ce62-433b-87b8-a8bb1405c70b-istiod-ca-cert\") pod \"e2e-api-a-87293f-openshift-default-7c9c9cbc7f-vw6fm\" (UID: \"cf796179-ce62-433b-87b8-a8bb1405c70b\") " pod="openshift-ingress/e2e-api-a-87293f-openshift-default-7c9c9cbc7f-vw6fm"
Oct 30 18:52:52.067091 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:52.067068    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf796179-ce62-433b-87b8-a8bb1405c70b-istio-data\") pod \"e2e-api-a-87293f-openshift-default-7c9c9cbc7f-vw6fm\" (UID: \"cf796179-ce62-433b-87b8-a8bb1405c70b\") " pod="openshift-ingress/e2e-api-a-87293f-openshift-default-7c9c9cbc7f-vw6fm"
Oct 30 18:52:52.068449 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:52.068403    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf796179-ce62-433b-87b8-a8bb1405c70b-workload-certs\") pod \"e2e-api-a-87293f-openshift-default-7c9c9cbc7f-vw6fm\" (UID: \"cf796179-ce62-433b-87b8-a8bb1405c70b\") " pod="openshift-ingress/e2e-api-a-87293f-openshift-default-7c9c9cbc7f-vw6fm"
Oct 30 18:52:52.072593 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:52.072549    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf796179-ce62-433b-87b8-a8bb1405c70b-istio-envoy\") pod \"e2e-api-a-87293f-openshift-default-7c9c9cbc7f-vw6fm\" (UID: \"cf796179-ce62-433b-87b8-a8bb1405c70b\") " pod="openshift-ingress/e2e-api-a-87293f-openshift-default-7c9c9cbc7f-vw6fm"
Oct 30 18:52:52.073600 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:52.073141    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf796179-ce62-433b-87b8-a8bb1405c70b-istio-token\") pod \"e2e-api-a-87293f-openshift-default-7c9c9cbc7f-vw6fm\" (UID: \"cf796179-ce62-433b-87b8-a8bb1405c70b\") " pod="openshift-ingress/e2e-api-a-87293f-openshift-default-7c9c9cbc7f-vw6fm"
Oct 30 18:52:52.076448 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:52.076360    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtwft\" (UniqueName: \"kubernetes.io/projected/cf796179-ce62-433b-87b8-a8bb1405c70b-kube-api-access-vtwft\") pod \"e2e-api-a-87293f-openshift-default-7c9c9cbc7f-vw6fm\" (UID: \"cf796179-ce62-433b-87b8-a8bb1405c70b\") " pod="openshift-ingress/e2e-api-a-87293f-openshift-default-7c9c9cbc7f-vw6fm"
Oct 30 18:52:52.080737 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:52.079153    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf796179-ce62-433b-87b8-a8bb1405c70b-istio-podinfo\") pod \"e2e-api-a-87293f-openshift-default-7c9c9cbc7f-vw6fm\" (UID: \"cf796179-ce62-433b-87b8-a8bb1405c70b\") " pod="openshift-ingress/e2e-api-a-87293f-openshift-default-7c9c9cbc7f-vw6fm"
Oct 30 18:52:52.178562 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:52.178471    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-87293f-openshift-default-7c9c9cbc7f-vw6fm"
Oct 30 18:52:52.196798 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:52.196753    4525 patch_prober.go:29] interesting pod/e2e-worker-w0-9de872-openshift-default-77cf8df555-txf2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 30 18:52:52.197596 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:52.197019    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-txf2f" podUID="f94f5889-fce0-46bc-a055-1c5e5e51d6a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 30 18:52:52.490616 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:52:52.490581    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf796179_ce62_433b_87b8_a8bb1405c70b.slice/crio-4862e37873a2eb78da592424b01c21b7cb1d7536676de9faf0edbbe14445b5bc WatchSource:0}: Error finding container 4862e37873a2eb78da592424b01c21b7cb1d7536676de9faf0edbbe14445b5bc: Status 404 returned error can't find the container with id 4862e37873a2eb78da592424b01c21b7cb1d7536676de9faf0edbbe14445b5bc
Oct 30 18:52:52.493193 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:52.493160    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:52:52.493306 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:52.493231    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:52:52.493306 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:52.493257    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:52:52.494591 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:52.494543    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-87293f-openshift-default-7c9c9cbc7f-vw6fm"]
Oct 30 18:52:52.926402 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:52.926368    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-87293f-openshift-default-7c9c9cbc7f-vw6fm" event={"ID":"cf796179-ce62-433b-87b8-a8bb1405c70b","Type":"ContainerStarted","Data":"67aa60478f70c3fa5eb0938ee01dd809a741e22324dd53c360be0b09d59cb8f8"}
Oct 30 18:52:52.926930 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:52.926431    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-87293f-openshift-default-7c9c9cbc7f-vw6fm" event={"ID":"cf796179-ce62-433b-87b8-a8bb1405c70b","Type":"ContainerStarted","Data":"4862e37873a2eb78da592424b01c21b7cb1d7536676de9faf0edbbe14445b5bc"}
Oct 30 18:52:52.950930 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:52.950871    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-87293f-openshift-default-7c9c9cbc7f-vw6fm" podStartSLOduration=1.950856747 podStartE2EDuration="1.950856747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:52:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:52.945161793 +0000 UTC m=+995.570831756" watchObservedRunningTime="2026-10-30 18:52:52.950856747 +0000 UTC m=+995.576526710"
Oct 30 18:52:53.186513 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:53.182021    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-87293f-openshift-default-7c9c9cbc7f-vw6fm"
Oct 30 18:52:53.199151 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:53.197332    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-87293f-openshift-default-7c9c9cbc7f-vw6fm"
Oct 30 18:52:53.481596 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:53.481517    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-txf2f"
Oct 30 18:52:53.482298 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:53.482266    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-txf2f"
Oct 30 18:52:53.486704 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:53.483761    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-txf2f"
Oct 30 18:52:53.934455 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:53.934424    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-87293f-openshift-default-7c9c9cbc7f-vw6fm"
Oct 30 18:52:53.935396 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:53.935376    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-87293f-openshift-default-7c9c9cbc7f-vw6fm"
Oct 30 18:52:59.352922 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:59.352885    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2"]
Oct 30 18:52:59.359090 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:59.359064    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2"
Oct 30 18:52:59.371154 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:59.371123    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2"]
Oct 30 18:52:59.401659 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:59.401619    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72b6833a-13cf-4fe8-9b8d-26f5decea7f6-workload-socket\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2\" (UID: \"72b6833a-13cf-4fe8-9b8d-26f5decea7f6\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2"
Oct 30 18:52:59.401659 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:59.401664    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72b6833a-13cf-4fe8-9b8d-26f5decea7f6-istio-podinfo\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2\" (UID: \"72b6833a-13cf-4fe8-9b8d-26f5decea7f6\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2"
Oct 30 18:52:59.401957 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:59.401819    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72b6833a-13cf-4fe8-9b8d-26f5decea7f6-istio-data\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2\" (UID: \"72b6833a-13cf-4fe8-9b8d-26f5decea7f6\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2"
Oct 30 18:52:59.401957 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:59.401865    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72b6833a-13cf-4fe8-9b8d-26f5decea7f6-credential-socket\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2\" (UID: \"72b6833a-13cf-4fe8-9b8d-26f5decea7f6\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2"
Oct 30 18:52:59.402092 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:59.401960    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72b6833a-13cf-4fe8-9b8d-26f5decea7f6-istio-envoy\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2\" (UID: \"72b6833a-13cf-4fe8-9b8d-26f5decea7f6\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2"
Oct 30 18:52:59.402092 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:59.402076    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72b6833a-13cf-4fe8-9b8d-26f5decea7f6-istiod-ca-cert\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2\" (UID: \"72b6833a-13cf-4fe8-9b8d-26f5decea7f6\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2"
Oct 30 18:52:59.402204 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:59.402146    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72b6833a-13cf-4fe8-9b8d-26f5decea7f6-istio-token\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2\" (UID: \"72b6833a-13cf-4fe8-9b8d-26f5decea7f6\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2"
Oct 30 18:52:59.402204 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:59.402174    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldgfp\" (UniqueName: \"kubernetes.io/projected/72b6833a-13cf-4fe8-9b8d-26f5decea7f6-kube-api-access-ldgfp\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2\" (UID: \"72b6833a-13cf-4fe8-9b8d-26f5decea7f6\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2"
Oct 30 18:52:59.402319 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:59.402203    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72b6833a-13cf-4fe8-9b8d-26f5decea7f6-workload-certs\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2\" (UID: \"72b6833a-13cf-4fe8-9b8d-26f5decea7f6\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2"
Oct 30 18:52:59.503694 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:59.503650    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72b6833a-13cf-4fe8-9b8d-26f5decea7f6-istio-envoy\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2\" (UID: \"72b6833a-13cf-4fe8-9b8d-26f5decea7f6\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2"
Oct 30 18:52:59.503877 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:59.503791    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72b6833a-13cf-4fe8-9b8d-26f5decea7f6-istiod-ca-cert\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2\" (UID: \"72b6833a-13cf-4fe8-9b8d-26f5decea7f6\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2"
Oct 30 18:52:59.503877 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:59.503833    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72b6833a-13cf-4fe8-9b8d-26f5decea7f6-istio-token\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2\" (UID: \"72b6833a-13cf-4fe8-9b8d-26f5decea7f6\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2"
Oct 30 18:52:59.503877 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:59.503851    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldgfp\" (UniqueName: \"kubernetes.io/projected/72b6833a-13cf-4fe8-9b8d-26f5decea7f6-kube-api-access-ldgfp\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2\" (UID: \"72b6833a-13cf-4fe8-9b8d-26f5decea7f6\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2"
Oct 30 18:52:59.503877 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:59.503871    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72b6833a-13cf-4fe8-9b8d-26f5decea7f6-workload-certs\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2\" (UID: \"72b6833a-13cf-4fe8-9b8d-26f5decea7f6\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2"
Oct 30 18:52:59.504088 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:59.503958    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72b6833a-13cf-4fe8-9b8d-26f5decea7f6-workload-socket\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2\" (UID: \"72b6833a-13cf-4fe8-9b8d-26f5decea7f6\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2"
Oct 30 18:52:59.504088 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:59.503982    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72b6833a-13cf-4fe8-9b8d-26f5decea7f6-istio-podinfo\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2\" (UID: \"72b6833a-13cf-4fe8-9b8d-26f5decea7f6\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2"
Oct 30 18:52:59.504088 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:59.504021    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72b6833a-13cf-4fe8-9b8d-26f5decea7f6-istio-data\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2\" (UID: \"72b6833a-13cf-4fe8-9b8d-26f5decea7f6\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2"
Oct 30 18:52:59.504088 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:59.504049    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72b6833a-13cf-4fe8-9b8d-26f5decea7f6-credential-socket\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2\" (UID: \"72b6833a-13cf-4fe8-9b8d-26f5decea7f6\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2"
Oct 30 18:52:59.504403 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:59.504377    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72b6833a-13cf-4fe8-9b8d-26f5decea7f6-workload-certs\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2\" (UID: \"72b6833a-13cf-4fe8-9b8d-26f5decea7f6\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2"
Oct 30 18:52:59.504500 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:59.504474    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72b6833a-13cf-4fe8-9b8d-26f5decea7f6-workload-socket\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2\" (UID: \"72b6833a-13cf-4fe8-9b8d-26f5decea7f6\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2"
Oct 30 18:52:59.504571 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:59.504484    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72b6833a-13cf-4fe8-9b8d-26f5decea7f6-istio-data\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2\" (UID: \"72b6833a-13cf-4fe8-9b8d-26f5decea7f6\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2"
Oct 30 18:52:59.504630 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:59.504563    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72b6833a-13cf-4fe8-9b8d-26f5decea7f6-credential-socket\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2\" (UID: \"72b6833a-13cf-4fe8-9b8d-26f5decea7f6\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2"
Oct 30 18:52:59.504726 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:59.504699    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72b6833a-13cf-4fe8-9b8d-26f5decea7f6-istiod-ca-cert\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2\" (UID: \"72b6833a-13cf-4fe8-9b8d-26f5decea7f6\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2"
Oct 30 18:52:59.506553 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:59.506522    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72b6833a-13cf-4fe8-9b8d-26f5decea7f6-istio-envoy\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2\" (UID: \"72b6833a-13cf-4fe8-9b8d-26f5decea7f6\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2"
Oct 30 18:52:59.506736 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:59.506715    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72b6833a-13cf-4fe8-9b8d-26f5decea7f6-istio-podinfo\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2\" (UID: \"72b6833a-13cf-4fe8-9b8d-26f5decea7f6\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2"
Oct 30 18:52:59.513144 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:59.513111    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72b6833a-13cf-4fe8-9b8d-26f5decea7f6-istio-token\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2\" (UID: \"72b6833a-13cf-4fe8-9b8d-26f5decea7f6\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2"
Oct 30 18:52:59.513341 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:59.513318    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldgfp\" (UniqueName: \"kubernetes.io/projected/72b6833a-13cf-4fe8-9b8d-26f5decea7f6-kube-api-access-ldgfp\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2\" (UID: \"72b6833a-13cf-4fe8-9b8d-26f5decea7f6\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2"
Oct 30 18:52:59.698699 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:52:59.698652    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2"
Oct 30 18:53:00.067399 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:00.066637    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:53:00.067399 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:00.066743    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:53:00.067399 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:00.066786    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:53:00.102502 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:00.102419    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2"]
Oct 30 18:53:01.110728 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:01.107158    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2" event={"ID":"72b6833a-13cf-4fe8-9b8d-26f5decea7f6","Type":"ContainerStarted","Data":"43f2859aefd0804546efa7b62f01106635812344a55c17fea614a262775e3519"}
Oct 30 18:53:01.110728 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:01.107227    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2" event={"ID":"72b6833a-13cf-4fe8-9b8d-26f5decea7f6","Type":"ContainerStarted","Data":"6f3189413c195fc986a19abfa6e22f949197cfb92653f5473d11cff6bb312221"}
Oct 30 18:53:01.161709 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:01.161309    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2" podStartSLOduration=2.161273552 podStartE2EDuration="2.161273552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:52:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:53:01.157302283 +0000 UTC m=+1003.782972282" watchObservedRunningTime="2026-10-30 18:53:01.161273552 +0000 UTC m=+1003.786943517"
Oct 30 18:53:01.712706 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:01.711991    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2"
Oct 30 18:53:01.712706 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:01.712175    4525 patch_prober.go:29] interesting pod/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Oct 30 18:53:01.712706 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:01.712224    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2" podUID="72b6833a-13cf-4fe8-9b8d-26f5decea7f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Oct 30 18:53:01.892708 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:01.891793    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t" podUID="424fca14-6d8a-4663-86ec-0c91165f34ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:53:02.097773 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:02.096785    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj" podUID="84c99c49-1d60-4176-bda4-cad29c7d801c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:53:02.115719 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:02.105977    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr" podUID="82bbbf85-ed93-48d2-9f17-49ac1f31c0a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 30 18:53:02.115719 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:02.106360    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts" podUID="be6d4969-285b-4da0-883e-4847f2cbf3af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:53:02.394833 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:02.394795    4525 patch_prober.go:29] interesting pod/e2e-worker-w3-2889dd-openshift-default-84768d69f7-gtdr6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 18:53:02.402715 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:02.395056    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-gtdr6" podUID="01a5c107-dcf4-4ca5-8c9d-57c40814206d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:53:02.784770 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:02.705409    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb" podUID="12ebc8b9-c794-47b5-b87b-a63c4bd993e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 18:53:02.784770 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:02.731418    4525 patch_prober.go:29] interesting pod/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Oct 30 18:53:02.784770 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:02.731475    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2" podUID="72b6833a-13cf-4fe8-9b8d-26f5decea7f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Oct 30 18:53:02.784770 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:02.756225    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw" podUID="ac6852fd-01be-4d69-8c9b-0c591e50b889" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 18:53:02.881720 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:02.878974    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv" podUID="595230e3-55f3-43eb-9a7b-6d89250af9b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:53:02.996713 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:02.893546    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t" podUID="424fca14-6d8a-4663-86ec-0c91165f34ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:53:03.022708 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:02.997383    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t"
Oct 30 18:53:03.107769 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:03.105904    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj" podUID="84c99c49-1d60-4176-bda4-cad29c7d801c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:53:03.157708 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:03.144141    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj"
Oct 30 18:53:03.173486 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:03.173437    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts" podUID="be6d4969-285b-4da0-883e-4847f2cbf3af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:53:03.174354 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:03.174322    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr" podUID="82bbbf85-ed93-48d2-9f17-49ac1f31c0a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 30 18:53:03.200097 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:03.200063    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts"
Oct 30 18:53:03.200405 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:03.200387    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr"
Oct 30 18:53:03.527768 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:03.456106    4525 patch_prober.go:29] interesting pod/e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 18:53:03.698708 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:03.664883    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j" podUID="6061ede7-981c-4c07-acfc-04be1cae252d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:53:03.753891 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:03.753853    4525 patch_prober.go:29] interesting pod/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Oct 30 18:53:03.762017 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:03.761977    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2" podUID="72b6833a-13cf-4fe8-9b8d-26f5decea7f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Oct 30 18:53:03.762272 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:03.760201    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw" podUID="ac6852fd-01be-4d69-8c9b-0c591e50b889" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:53:03.872758 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:03.853775    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb" podUID="12ebc8b9-c794-47b5-b87b-a63c4bd993e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:53:03.904708 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:03.889234    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv" podUID="595230e3-55f3-43eb-9a7b-6d89250af9b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:53:03.922216 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:03.922181    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw"
Oct 30 18:53:03.922549 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:03.922530    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb"
Oct 30 18:53:03.932324 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:03.932294    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv"
Oct 30 18:53:04.016851 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:04.016804    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t" podUID="424fca14-6d8a-4663-86ec-0c91165f34ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 18:53:04.150764 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:04.131371    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj" podUID="84c99c49-1d60-4176-bda4-cad29c7d801c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 18:53:04.177904 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:04.177856    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts" podUID="be6d4969-285b-4da0-883e-4847f2cbf3af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:53:04.218847 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:04.218792    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr" podUID="82bbbf85-ed93-48d2-9f17-49ac1f31c0a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:53:04.478742 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:04.476965    4525 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts" podUID="be6d4969-285b-4da0-883e-4847f2cbf3af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:53:04.765362 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:04.765265    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw" podUID="ac6852fd-01be-4d69-8c9b-0c591e50b889" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:53:04.807399 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:04.807359    4525 patch_prober.go:29] interesting pod/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Oct 30 18:53:04.807706 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:04.807640    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2" podUID="72b6833a-13cf-4fe8-9b8d-26f5decea7f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Oct 30 18:53:04.911709 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:04.893369    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb" podUID="12ebc8b9-c794-47b5-b87b-a63c4bd993e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:53:04.912291 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:04.912243    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv" podUID="595230e3-55f3-43eb-9a7b-6d89250af9b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:53:05.029309 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:05.029200    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t" podUID="424fca14-6d8a-4663-86ec-0c91165f34ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 18:53:05.172548 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:05.172504    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj" podUID="84c99c49-1d60-4176-bda4-cad29c7d801c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 18:53:05.224391 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:05.223587    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts" podUID="be6d4969-285b-4da0-883e-4847f2cbf3af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Oct 30 18:53:05.249732 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:05.225115    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a325894f-sm5gh"]
Oct 30 18:53:05.276321 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:05.250049    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr" podUID="82bbbf85-ed93-48d2-9f17-49ac1f31c0a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:53:05.297767 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:05.282401    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a325894f-sm5gh"
Oct 30 18:53:05.308839 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:05.308134    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94hcn\""
Oct 30 18:53:05.308839 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:05.308225    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:53:05.391193 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:05.391164    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a325894f-sm5gh"]
Oct 30 18:53:05.521383 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:05.499299    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2438ddf-fbd5-4d40-b271-cbdb10efd74f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a325894f-sm5gh\" (UID: \"a2438ddf-fbd5-4d40-b271-cbdb10efd74f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a325894f-sm5gh"
Oct 30 18:53:05.521948 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:05.521919    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2438ddf-fbd5-4d40-b271-cbdb10efd74f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a325894f-sm5gh\" (UID: \"a2438ddf-fbd5-4d40-b271-cbdb10efd74f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a325894f-sm5gh"
Oct 30 18:53:05.522192 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:05.522173    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2dc4\" (UniqueName: \"kubernetes.io/projected/a2438ddf-fbd5-4d40-b271-cbdb10efd74f-kube-api-access-b2dc4\") pod \"maas-api-revoke-sub-a325894f-sm5gh\" (UID: \"a2438ddf-fbd5-4d40-b271-cbdb10efd74f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a325894f-sm5gh"
Oct 30 18:53:05.624000 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:05.623963    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2438ddf-fbd5-4d40-b271-cbdb10efd74f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a325894f-sm5gh\" (UID: \"a2438ddf-fbd5-4d40-b271-cbdb10efd74f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a325894f-sm5gh"
Oct 30 18:53:05.663706 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:05.663069    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2dc4\" (UniqueName: \"kubernetes.io/projected/a2438ddf-fbd5-4d40-b271-cbdb10efd74f-kube-api-access-b2dc4\") pod \"maas-api-revoke-sub-a325894f-sm5gh\" (UID: \"a2438ddf-fbd5-4d40-b271-cbdb10efd74f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a325894f-sm5gh"
Oct 30 18:53:05.663706 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:05.663322    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2438ddf-fbd5-4d40-b271-cbdb10efd74f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a325894f-sm5gh\" (UID: \"a2438ddf-fbd5-4d40-b271-cbdb10efd74f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a325894f-sm5gh"
Oct 30 18:53:05.680720 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:05.666139    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2438ddf-fbd5-4d40-b271-cbdb10efd74f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a325894f-sm5gh\" (UID: \"a2438ddf-fbd5-4d40-b271-cbdb10efd74f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a325894f-sm5gh"
Oct 30 18:53:05.764806 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:05.762839    4525 patch_prober.go:29] interesting pod/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Oct 30 18:53:05.764806 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:05.762906    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2" podUID="72b6833a-13cf-4fe8-9b8d-26f5decea7f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Oct 30 18:53:05.771807 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:05.771757    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw" podUID="ac6852fd-01be-4d69-8c9b-0c591e50b889" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:53:05.801706 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:05.772712    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2438ddf-fbd5-4d40-b271-cbdb10efd74f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a325894f-sm5gh\" (UID: \"a2438ddf-fbd5-4d40-b271-cbdb10efd74f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a325894f-sm5gh"
Oct 30 18:53:05.888765 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:05.886377    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2dc4\" (UniqueName: \"kubernetes.io/projected/a2438ddf-fbd5-4d40-b271-cbdb10efd74f-kube-api-access-b2dc4\") pod \"maas-api-revoke-sub-a325894f-sm5gh\" (UID: \"a2438ddf-fbd5-4d40-b271-cbdb10efd74f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a325894f-sm5gh"
Oct 30 18:53:05.904704 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:05.902756    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb" podUID="12ebc8b9-c794-47b5-b87b-a63c4bd993e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:53:05.970719 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:05.918604    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv"
Oct 30 18:53:06.097709 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:06.097522    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t" podUID="424fca14-6d8a-4663-86ec-0c91165f34ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 18:53:06.185763 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:06.178026    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj" podUID="84c99c49-1d60-4176-bda4-cad29c7d801c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 18:53:06.219879 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:06.219847    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a325894f-sm5gh"
Oct 30 18:53:06.301706 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:06.293886    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr" podUID="82bbbf85-ed93-48d2-9f17-49ac1f31c0a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:53:06.328739 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:06.321747    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts" podUID="be6d4969-285b-4da0-883e-4847f2cbf3af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:53:06.858765 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:06.854831    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw" podUID="ac6852fd-01be-4d69-8c9b-0c591e50b889" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 18:53:06.940842 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:06.940791    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv" podUID="595230e3-55f3-43eb-9a7b-6d89250af9b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:53:06.972023 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:06.971973    4525 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb" podUID="12ebc8b9-c794-47b5-b87b-a63c4bd993e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": context deadline exceeded"
Oct 30 18:53:07.159145 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:07.119798    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb" podUID="12ebc8b9-c794-47b5-b87b-a63c4bd993e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 18:53:07.160937 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:07.120746    4525 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-68cb9b4b95-dhclt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 18:53:07.172092 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:07.172044    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-dhclt" podUID="0e0636e7-6f61-4843-844d-fb9d7bb527da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:53:07.172320 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:07.121008    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t" podUID="424fca14-6d8a-4663-86ec-0c91165f34ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:53:07.172767 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:07.137870    4525 patch_prober.go:29] interesting pod/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Oct 30 18:53:07.172994 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:07.172962    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2" podUID="72b6833a-13cf-4fe8-9b8d-26f5decea7f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Oct 30 18:53:07.178768 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:07.178612    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj" podUID="84c99c49-1d60-4176-bda4-cad29c7d801c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:53:07.319577 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:07.317597    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr" podUID="82bbbf85-ed93-48d2-9f17-49ac1f31c0a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:53:07.336858 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:07.335864    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj"
Oct 30 18:53:07.368445 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:07.343840    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts" podUID="be6d4969-285b-4da0-883e-4847f2cbf3af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:53:07.512342 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:07.512310    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t"
Oct 30 18:53:07.794761 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:07.792811    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts"
Oct 30 18:53:07.812427 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:07.811422    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw"
Oct 30 18:53:07.829813 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:07.813441    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb"
Oct 30 18:53:07.829813 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:07.815248    4525 patch_prober.go:29] interesting pod/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Oct 30 18:53:07.829813 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:07.815321    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2" podUID="72b6833a-13cf-4fe8-9b8d-26f5decea7f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Oct 30 18:53:07.832707 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:07.831920    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr"
Oct 30 18:53:07.838894 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:07.838871    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a325894f-sm5gh"]
Oct 30 18:53:07.839189 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:07.839165    4525 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr" podUID="82bbbf85-ed93-48d2-9f17-49ac1f31c0a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:53:08.093157 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:08.093064    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a325894f-sm5gh" event={"ID":"a2438ddf-fbd5-4d40-b271-cbdb10efd74f","Type":"ContainerStarted","Data":"186b176e3b05e90727dca97431a88913b488b625598ed45aa497a401c240a11e"}
Oct 30 18:53:08.700757 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:08.699095    4525 patch_prober.go:29] interesting pod/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Oct 30 18:53:08.700757 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:08.699173    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2" podUID="72b6833a-13cf-4fe8-9b8d-26f5decea7f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Oct 30 18:53:09.122178 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:09.122139    4525 generic.go:356] "Generic (PLEG): container finished" podID="a2438ddf-fbd5-4d40-b271-cbdb10efd74f" containerID="ad786fa88c15e58611c19ae6a84a8025eedc972c43c4fd1bea5a05a92424661f" exitCode=0
Oct 30 18:53:09.122395 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:09.122369    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a325894f-sm5gh" event={"ID":"a2438ddf-fbd5-4d40-b271-cbdb10efd74f","Type":"ContainerDied","Data":"ad786fa88c15e58611c19ae6a84a8025eedc972c43c4fd1bea5a05a92424661f"}
Oct 30 18:53:09.710625 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:09.709418    4525 patch_prober.go:29] interesting pod/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Oct 30 18:53:09.710625 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:09.709483    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2" podUID="72b6833a-13cf-4fe8-9b8d-26f5decea7f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Oct 30 18:53:10.354664 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:10.354640    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a325894f-sm5gh"
Oct 30 18:53:10.475056 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:10.474966    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2438ddf-fbd5-4d40-b271-cbdb10efd74f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2438ddf-fbd5-4d40-b271-cbdb10efd74f-maas-api-openshift-ca\") pod \"a2438ddf-fbd5-4d40-b271-cbdb10efd74f\" (UID: \"a2438ddf-fbd5-4d40-b271-cbdb10efd74f\") "
Oct 30 18:53:10.475209 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:10.475152    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2438ddf-fbd5-4d40-b271-cbdb10efd74f-kube-api-access-b2dc4\" (UniqueName: \"kubernetes.io/projected/a2438ddf-fbd5-4d40-b271-cbdb10efd74f-kube-api-access-b2dc4\") pod \"a2438ddf-fbd5-4d40-b271-cbdb10efd74f\" (UID: \"a2438ddf-fbd5-4d40-b271-cbdb10efd74f\") "
Oct 30 18:53:10.475282 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:10.475263    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2438ddf-fbd5-4d40-b271-cbdb10efd74f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2438ddf-fbd5-4d40-b271-cbdb10efd74f-maas-api-xks-ca\") pod \"a2438ddf-fbd5-4d40-b271-cbdb10efd74f\" (UID: \"a2438ddf-fbd5-4d40-b271-cbdb10efd74f\") "
Oct 30 18:53:10.475346 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:10.475316    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2438ddf-fbd5-4d40-b271-cbdb10efd74f-maas-api-openshift-ca" pod "a2438ddf-fbd5-4d40-b271-cbdb10efd74f" (UID: "a2438ddf-fbd5-4d40-b271-cbdb10efd74f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:53:10.475790 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:10.475769    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2438ddf-fbd5-4d40-b271-cbdb10efd74f-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:10.477509 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:10.477483    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2438ddf-fbd5-4d40-b271-cbdb10efd74f-maas-api-xks-ca" pod "a2438ddf-fbd5-4d40-b271-cbdb10efd74f" (UID: "a2438ddf-fbd5-4d40-b271-cbdb10efd74f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:53:10.477575 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:10.477524    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2438ddf-fbd5-4d40-b271-cbdb10efd74f-kube-api-access-b2dc4" pod "a2438ddf-fbd5-4d40-b271-cbdb10efd74f" (UID: "a2438ddf-fbd5-4d40-b271-cbdb10efd74f"). InnerVolumeSpecName "kube-api-access-b2dc4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:53:10.577426 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:10.577388    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2438ddf-fbd5-4d40-b271-cbdb10efd74f-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:10.577426 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:10.577422    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2dc4\" (UniqueName: \"kubernetes.io/projected/a2438ddf-fbd5-4d40-b271-cbdb10efd74f-kube-api-access-b2dc4\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:10.705130 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:10.704129    4525 patch_prober.go:29] interesting pod/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Oct 30 18:53:10.705130 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:10.704197    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2" podUID="72b6833a-13cf-4fe8-9b8d-26f5decea7f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Oct 30 18:53:11.142745 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:11.142707    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a325894f-sm5gh" event={"ID":"a2438ddf-fbd5-4d40-b271-cbdb10efd74f","Type":"ContainerDied","Data":"186b176e3b05e90727dca97431a88913b488b625598ed45aa497a401c240a11e"}
Oct 30 18:53:11.142745 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:11.142737    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a325894f-sm5gh"
Oct 30 18:53:11.143190 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:11.142769    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="186b176e3b05e90727dca97431a88913b488b625598ed45aa497a401c240a11e"
Oct 30 18:53:11.895070 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:11.895036    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2"
Oct 30 18:53:11.898308 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:11.898270    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2"
Oct 30 18:53:11.898734 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:11.898709    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2"
Oct 30 18:53:13.602270 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:13.602149    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-zv786"]
Oct 30 18:53:13.602888 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:13.602772    4525 cpu_manager.go:414] "containerMap: removing container" podUID="a2438ddf-fbd5-4d40-b271-cbdb10efd74f" containerName="revoke-keys"
Oct 30 18:53:13.602888 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:13.602789    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2438ddf-fbd5-4d40-b271-cbdb10efd74f" containerName="revoke-keys"
Oct 30 18:53:13.602974 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:13.602946    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2438ddf-fbd5-4d40-b271-cbdb10efd74f" containerName="revoke-keys"
Oct 30 18:53:13.607170 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:13.607149    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-zv786"
Oct 30 18:53:13.630751 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:13.630724    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-zv786"]
Oct 30 18:53:13.718445 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:13.718401    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f3c674d-b2c1-4a9a-afa1-5bfd33999e99-workload-certs\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-zv786\" (UID: \"4f3c674d-b2c1-4a9a-afa1-5bfd33999e99\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-zv786"
Oct 30 18:53:13.718621 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:13.718572    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f3c674d-b2c1-4a9a-afa1-5bfd33999e99-istiod-ca-cert\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-zv786\" (UID: \"4f3c674d-b2c1-4a9a-afa1-5bfd33999e99\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-zv786"
Oct 30 18:53:13.718669 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:13.718653    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f3c674d-b2c1-4a9a-afa1-5bfd33999e99-istio-data\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-zv786\" (UID: \"4f3c674d-b2c1-4a9a-afa1-5bfd33999e99\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-zv786"
Oct 30 18:53:13.718737 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:13.718696    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f3c674d-b2c1-4a9a-afa1-5bfd33999e99-istio-envoy\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-zv786\" (UID: \"4f3c674d-b2c1-4a9a-afa1-5bfd33999e99\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-zv786"
Oct 30 18:53:13.718775 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:13.718735    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f3c674d-b2c1-4a9a-afa1-5bfd33999e99-credential-socket\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-zv786\" (UID: \"4f3c674d-b2c1-4a9a-afa1-5bfd33999e99\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-zv786"
Oct 30 18:53:13.718775 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:13.718754    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f3c674d-b2c1-4a9a-afa1-5bfd33999e99-istio-token\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-zv786\" (UID: \"4f3c674d-b2c1-4a9a-afa1-5bfd33999e99\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-zv786"
Oct 30 18:53:13.718929 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:13.718908    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f3c674d-b2c1-4a9a-afa1-5bfd33999e99-workload-socket\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-zv786\" (UID: \"4f3c674d-b2c1-4a9a-afa1-5bfd33999e99\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-zv786"
Oct 30 18:53:13.718964 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:13.718945    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q28jp\" (UniqueName: \"kubernetes.io/projected/4f3c674d-b2c1-4a9a-afa1-5bfd33999e99-kube-api-access-q28jp\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-zv786\" (UID: \"4f3c674d-b2c1-4a9a-afa1-5bfd33999e99\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-zv786"
Oct 30 18:53:13.718997 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:13.718965    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f3c674d-b2c1-4a9a-afa1-5bfd33999e99-istio-podinfo\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-zv786\" (UID: \"4f3c674d-b2c1-4a9a-afa1-5bfd33999e99\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-zv786"
Oct 30 18:53:13.820849 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:13.820808    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f3c674d-b2c1-4a9a-afa1-5bfd33999e99-workload-socket\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-zv786\" (UID: \"4f3c674d-b2c1-4a9a-afa1-5bfd33999e99\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-zv786"
Oct 30 18:53:13.820849 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:13.820854    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q28jp\" (UniqueName: \"kubernetes.io/projected/4f3c674d-b2c1-4a9a-afa1-5bfd33999e99-kube-api-access-q28jp\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-zv786\" (UID: \"4f3c674d-b2c1-4a9a-afa1-5bfd33999e99\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-zv786"
Oct 30 18:53:13.821125 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:13.820875    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f3c674d-b2c1-4a9a-afa1-5bfd33999e99-istio-podinfo\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-zv786\" (UID: \"4f3c674d-b2c1-4a9a-afa1-5bfd33999e99\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-zv786"
Oct 30 18:53:13.821125 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:13.820912    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f3c674d-b2c1-4a9a-afa1-5bfd33999e99-workload-certs\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-zv786\" (UID: \"4f3c674d-b2c1-4a9a-afa1-5bfd33999e99\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-zv786"
Oct 30 18:53:13.821125 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:13.821000    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f3c674d-b2c1-4a9a-afa1-5bfd33999e99-istiod-ca-cert\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-zv786\" (UID: \"4f3c674d-b2c1-4a9a-afa1-5bfd33999e99\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-zv786"
Oct 30 18:53:13.821125 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:13.821053    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f3c674d-b2c1-4a9a-afa1-5bfd33999e99-istio-data\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-zv786\" (UID: \"4f3c674d-b2c1-4a9a-afa1-5bfd33999e99\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-zv786"
Oct 30 18:53:13.821125 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:13.821083    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f3c674d-b2c1-4a9a-afa1-5bfd33999e99-istio-envoy\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-zv786\" (UID: \"4f3c674d-b2c1-4a9a-afa1-5bfd33999e99\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-zv786"
Oct 30 18:53:13.821391 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:13.821139    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f3c674d-b2c1-4a9a-afa1-5bfd33999e99-credential-socket\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-zv786\" (UID: \"4f3c674d-b2c1-4a9a-afa1-5bfd33999e99\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-zv786"
Oct 30 18:53:13.821391 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:13.821164    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f3c674d-b2c1-4a9a-afa1-5bfd33999e99-istio-token\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-zv786\" (UID: \"4f3c674d-b2c1-4a9a-afa1-5bfd33999e99\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-zv786"
Oct 30 18:53:13.821391 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:13.821349    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f3c674d-b2c1-4a9a-afa1-5bfd33999e99-workload-socket\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-zv786\" (UID: \"4f3c674d-b2c1-4a9a-afa1-5bfd33999e99\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-zv786"
Oct 30 18:53:13.821391 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:13.821378    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f3c674d-b2c1-4a9a-afa1-5bfd33999e99-workload-certs\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-zv786\" (UID: \"4f3c674d-b2c1-4a9a-afa1-5bfd33999e99\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-zv786"
Oct 30 18:53:13.821661 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:13.821510    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f3c674d-b2c1-4a9a-afa1-5bfd33999e99-istio-data\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-zv786\" (UID: \"4f3c674d-b2c1-4a9a-afa1-5bfd33999e99\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-zv786"
Oct 30 18:53:13.821744 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:13.821729    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f3c674d-b2c1-4a9a-afa1-5bfd33999e99-credential-socket\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-zv786\" (UID: \"4f3c674d-b2c1-4a9a-afa1-5bfd33999e99\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-zv786"
Oct 30 18:53:13.821963 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:13.821923    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f3c674d-b2c1-4a9a-afa1-5bfd33999e99-istiod-ca-cert\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-zv786\" (UID: \"4f3c674d-b2c1-4a9a-afa1-5bfd33999e99\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-zv786"
Oct 30 18:53:13.823621 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:13.823594    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f3c674d-b2c1-4a9a-afa1-5bfd33999e99-istio-envoy\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-zv786\" (UID: \"4f3c674d-b2c1-4a9a-afa1-5bfd33999e99\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-zv786"
Oct 30 18:53:13.823809 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:13.823788    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f3c674d-b2c1-4a9a-afa1-5bfd33999e99-istio-podinfo\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-zv786\" (UID: \"4f3c674d-b2c1-4a9a-afa1-5bfd33999e99\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-zv786"
Oct 30 18:53:13.828524 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:13.828500    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f3c674d-b2c1-4a9a-afa1-5bfd33999e99-istio-token\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-zv786\" (UID: \"4f3c674d-b2c1-4a9a-afa1-5bfd33999e99\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-zv786"
Oct 30 18:53:13.828643 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:13.828623    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q28jp\" (UniqueName: \"kubernetes.io/projected/4f3c674d-b2c1-4a9a-afa1-5bfd33999e99-kube-api-access-q28jp\") pod \"e2e-worker-w0-9de872-openshift-default-77cf8df555-zv786\" (UID: \"4f3c674d-b2c1-4a9a-afa1-5bfd33999e99\") " pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-zv786"
Oct 30 18:53:13.930835 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:13.930796    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-zv786"
Oct 30 18:53:14.222724 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:14.222674    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-zv786"]
Oct 30 18:53:14.223364 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:53:14.223332    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f3c674d_b2c1_4a9a_afa1_5bfd33999e99.slice/crio-0f100e3b2dbefb2628a82c66e0d9456ba0a6698df3c5be9abc9fd4f1013ecb3a WatchSource:0}: Error finding container 0f100e3b2dbefb2628a82c66e0d9456ba0a6698df3c5be9abc9fd4f1013ecb3a: Status 404 returned error can't find the container with id 0f100e3b2dbefb2628a82c66e0d9456ba0a6698df3c5be9abc9fd4f1013ecb3a
Oct 30 18:53:14.234015 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:14.233956    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:53:14.234240 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:14.234212    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:53:14.234335 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:14.234313    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:53:15.212071 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:15.212030    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-zv786" event={"ID":"4f3c674d-b2c1-4a9a-afa1-5bfd33999e99","Type":"ContainerStarted","Data":"e1d959ee8e2f4a6eba684605cf5f2200ae0605953dff477153b7b806f5df67ae"}
Oct 30 18:53:15.212504 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:15.212126    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-zv786" event={"ID":"4f3c674d-b2c1-4a9a-afa1-5bfd33999e99","Type":"ContainerStarted","Data":"0f100e3b2dbefb2628a82c66e0d9456ba0a6698df3c5be9abc9fd4f1013ecb3a"}
Oct 30 18:53:15.233641 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:15.233576    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-zv786" podStartSLOduration=2.233561036 podStartE2EDuration="2.233561036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:53:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:53:15.23023178 +0000 UTC m=+1017.855901744" watchObservedRunningTime="2026-10-30 18:53:15.233561036 +0000 UTC m=+1017.859230999"
Oct 30 18:53:15.935937 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:15.935853    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-zv786"
Oct 30 18:53:15.944731 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:15.942842    4525 patch_prober.go:29] interesting pod/e2e-worker-w0-9de872-openshift-default-77cf8df555-zv786 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Oct 30 18:53:15.944731 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:15.942905    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-zv786" podUID="4f3c674d-b2c1-4a9a-afa1-5bfd33999e99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Oct 30 18:53:16.931520 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:16.931483    4525 patch_prober.go:29] interesting pod/e2e-worker-w0-9de872-openshift-default-77cf8df555-zv786 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Oct 30 18:53:16.931981 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:16.931554    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-zv786" podUID="4f3c674d-b2c1-4a9a-afa1-5bfd33999e99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Oct 30 18:53:17.422790 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:53:17.422749    4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d49l" containerName="extractor"
Oct 30 18:53:17.932082 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:17.932040    4525 patch_prober.go:29] interesting pod/e2e-worker-w0-9de872-openshift-default-77cf8df555-zv786 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Oct 30 18:53:17.932488 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:17.932102    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-zv786" podUID="4f3c674d-b2c1-4a9a-afa1-5bfd33999e99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Oct 30 18:53:18.931236 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:18.931199    4525 patch_prober.go:29] interesting pod/e2e-worker-w0-9de872-openshift-default-77cf8df555-zv786 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Oct 30 18:53:18.931447 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:18.931273    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-zv786" podUID="4f3c674d-b2c1-4a9a-afa1-5bfd33999e99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Oct 30 18:53:19.931397 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:19.931367    4525 patch_prober.go:29] interesting pod/e2e-worker-w0-9de872-openshift-default-77cf8df555-zv786 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Oct 30 18:53:19.931817 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:19.931421    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-zv786" podUID="4f3c674d-b2c1-4a9a-afa1-5bfd33999e99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Oct 30 18:53:20.931759 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:20.931717    4525 patch_prober.go:29] interesting pod/e2e-worker-w0-9de872-openshift-default-77cf8df555-zv786 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Oct 30 18:53:20.932254 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:20.931782    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-zv786" podUID="4f3c674d-b2c1-4a9a-afa1-5bfd33999e99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Oct 30 18:53:21.116771 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:21.115138    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-066928-openshift-default-55ddf5c76f-pzlmw"]
Oct 30 18:53:21.133296 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:21.121425    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-066928-openshift-default-55ddf5c76f-pzlmw"
Oct 30 18:53:21.133296 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:21.131300    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-066928-openshift-default-55ddf5c76f-pzlmw"]
Oct 30 18:53:21.136667 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:21.136642    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-066928-openshift-default-dockercfg-fjphv\""
Oct 30 18:53:21.299469 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:21.298764    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9ff1283-3fc7-463f-b3b7-fafdd56f2349-istio-data\") pod \"e2e-api-b-066928-openshift-default-55ddf5c76f-pzlmw\" (UID: \"a9ff1283-3fc7-463f-b3b7-fafdd56f2349\") " pod="openshift-ingress/e2e-api-b-066928-openshift-default-55ddf5c76f-pzlmw"
Oct 30 18:53:21.299929 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:21.299900    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9ff1283-3fc7-463f-b3b7-fafdd56f2349-credential-socket\") pod \"e2e-api-b-066928-openshift-default-55ddf5c76f-pzlmw\" (UID: \"a9ff1283-3fc7-463f-b3b7-fafdd56f2349\") " pod="openshift-ingress/e2e-api-b-066928-openshift-default-55ddf5c76f-pzlmw"
Oct 30 18:53:21.300267 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:21.300247    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9ff1283-3fc7-463f-b3b7-fafdd56f2349-istio-podinfo\") pod \"e2e-api-b-066928-openshift-default-55ddf5c76f-pzlmw\" (UID: \"a9ff1283-3fc7-463f-b3b7-fafdd56f2349\") " pod="openshift-ingress/e2e-api-b-066928-openshift-default-55ddf5c76f-pzlmw"
Oct 30 18:53:21.300477 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:21.300458    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9ff1283-3fc7-463f-b3b7-fafdd56f2349-istio-envoy\") pod \"e2e-api-b-066928-openshift-default-55ddf5c76f-pzlmw\" (UID: \"a9ff1283-3fc7-463f-b3b7-fafdd56f2349\") " pod="openshift-ingress/e2e-api-b-066928-openshift-default-55ddf5c76f-pzlmw"
Oct 30 18:53:21.300941 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:21.300921    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9ff1283-3fc7-463f-b3b7-fafdd56f2349-istio-token\") pod \"e2e-api-b-066928-openshift-default-55ddf5c76f-pzlmw\" (UID: \"a9ff1283-3fc7-463f-b3b7-fafdd56f2349\") " pod="openshift-ingress/e2e-api-b-066928-openshift-default-55ddf5c76f-pzlmw"
Oct 30 18:53:21.301134 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:21.301116    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhpkq\" (UniqueName: \"kubernetes.io/projected/a9ff1283-3fc7-463f-b3b7-fafdd56f2349-kube-api-access-mhpkq\") pod \"e2e-api-b-066928-openshift-default-55ddf5c76f-pzlmw\" (UID: \"a9ff1283-3fc7-463f-b3b7-fafdd56f2349\") " pod="openshift-ingress/e2e-api-b-066928-openshift-default-55ddf5c76f-pzlmw"
Oct 30 18:53:21.301298 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:21.301282    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9ff1283-3fc7-463f-b3b7-fafdd56f2349-workload-socket\") pod \"e2e-api-b-066928-openshift-default-55ddf5c76f-pzlmw\" (UID: \"a9ff1283-3fc7-463f-b3b7-fafdd56f2349\") " pod="openshift-ingress/e2e-api-b-066928-openshift-default-55ddf5c76f-pzlmw"
Oct 30 18:53:21.301447 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:21.301432    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9ff1283-3fc7-463f-b3b7-fafdd56f2349-istiod-ca-cert\") pod \"e2e-api-b-066928-openshift-default-55ddf5c76f-pzlmw\" (UID: \"a9ff1283-3fc7-463f-b3b7-fafdd56f2349\") " pod="openshift-ingress/e2e-api-b-066928-openshift-default-55ddf5c76f-pzlmw"
Oct 30 18:53:21.301626 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:21.301609    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9ff1283-3fc7-463f-b3b7-fafdd56f2349-workload-certs\") pod \"e2e-api-b-066928-openshift-default-55ddf5c76f-pzlmw\" (UID: \"a9ff1283-3fc7-463f-b3b7-fafdd56f2349\") " pod="openshift-ingress/e2e-api-b-066928-openshift-default-55ddf5c76f-pzlmw"
Oct 30 18:53:21.414828 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:21.410556    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9ff1283-3fc7-463f-b3b7-fafdd56f2349-istio-data\") pod \"e2e-api-b-066928-openshift-default-55ddf5c76f-pzlmw\" (UID: \"a9ff1283-3fc7-463f-b3b7-fafdd56f2349\") " pod="openshift-ingress/e2e-api-b-066928-openshift-default-55ddf5c76f-pzlmw"
Oct 30 18:53:21.414828 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:21.410720    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9ff1283-3fc7-463f-b3b7-fafdd56f2349-credential-socket\") pod \"e2e-api-b-066928-openshift-default-55ddf5c76f-pzlmw\" (UID: \"a9ff1283-3fc7-463f-b3b7-fafdd56f2349\") " pod="openshift-ingress/e2e-api-b-066928-openshift-default-55ddf5c76f-pzlmw"
Oct 30 18:53:21.414828 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:21.410747    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9ff1283-3fc7-463f-b3b7-fafdd56f2349-istio-podinfo\") pod \"e2e-api-b-066928-openshift-default-55ddf5c76f-pzlmw\" (UID: \"a9ff1283-3fc7-463f-b3b7-fafdd56f2349\") " pod="openshift-ingress/e2e-api-b-066928-openshift-default-55ddf5c76f-pzlmw"
Oct 30 18:53:21.414828 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:21.410880    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9ff1283-3fc7-463f-b3b7-fafdd56f2349-istio-envoy\") pod \"e2e-api-b-066928-openshift-default-55ddf5c76f-pzlmw\" (UID: \"a9ff1283-3fc7-463f-b3b7-fafdd56f2349\") " pod="openshift-ingress/e2e-api-b-066928-openshift-default-55ddf5c76f-pzlmw"
Oct 30 18:53:21.414828 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:21.411003    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9ff1283-3fc7-463f-b3b7-fafdd56f2349-istio-token\") pod \"e2e-api-b-066928-openshift-default-55ddf5c76f-pzlmw\" (UID: \"a9ff1283-3fc7-463f-b3b7-fafdd56f2349\") " pod="openshift-ingress/e2e-api-b-066928-openshift-default-55ddf5c76f-pzlmw"
Oct 30 18:53:21.414828 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:21.411033    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhpkq\" (UniqueName: \"kubernetes.io/projected/a9ff1283-3fc7-463f-b3b7-fafdd56f2349-kube-api-access-mhpkq\") pod \"e2e-api-b-066928-openshift-default-55ddf5c76f-pzlmw\" (UID: \"a9ff1283-3fc7-463f-b3b7-fafdd56f2349\") " pod="openshift-ingress/e2e-api-b-066928-openshift-default-55ddf5c76f-pzlmw"
Oct 30 18:53:21.414828 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:21.411061    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9ff1283-3fc7-463f-b3b7-fafdd56f2349-workload-socket\") pod \"e2e-api-b-066928-openshift-default-55ddf5c76f-pzlmw\" (UID: \"a9ff1283-3fc7-463f-b3b7-fafdd56f2349\") " pod="openshift-ingress/e2e-api-b-066928-openshift-default-55ddf5c76f-pzlmw"
Oct 30 18:53:21.414828 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:21.411085    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9ff1283-3fc7-463f-b3b7-fafdd56f2349-istiod-ca-cert\") pod \"e2e-api-b-066928-openshift-default-55ddf5c76f-pzlmw\" (UID: \"a9ff1283-3fc7-463f-b3b7-fafdd56f2349\") " pod="openshift-ingress/e2e-api-b-066928-openshift-default-55ddf5c76f-pzlmw"
Oct 30 18:53:21.414828 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:21.411144    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9ff1283-3fc7-463f-b3b7-fafdd56f2349-workload-certs\") pod \"e2e-api-b-066928-openshift-default-55ddf5c76f-pzlmw\" (UID: \"a9ff1283-3fc7-463f-b3b7-fafdd56f2349\") " pod="openshift-ingress/e2e-api-b-066928-openshift-default-55ddf5c76f-pzlmw"
Oct 30 18:53:21.414828 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:21.411641    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9ff1283-3fc7-463f-b3b7-fafdd56f2349-workload-certs\") pod \"e2e-api-b-066928-openshift-default-55ddf5c76f-pzlmw\" (UID: \"a9ff1283-3fc7-463f-b3b7-fafdd56f2349\") " pod="openshift-ingress/e2e-api-b-066928-openshift-default-55ddf5c76f-pzlmw"
Oct 30 18:53:21.414828 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:21.413236    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9ff1283-3fc7-463f-b3b7-fafdd56f2349-istio-data\") pod \"e2e-api-b-066928-openshift-default-55ddf5c76f-pzlmw\" (UID: \"a9ff1283-3fc7-463f-b3b7-fafdd56f2349\") " pod="openshift-ingress/e2e-api-b-066928-openshift-default-55ddf5c76f-pzlmw"
Oct 30 18:53:21.414828 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:21.413436    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9ff1283-3fc7-463f-b3b7-fafdd56f2349-credential-socket\") pod \"e2e-api-b-066928-openshift-default-55ddf5c76f-pzlmw\" (UID: \"a9ff1283-3fc7-463f-b3b7-fafdd56f2349\") " pod="openshift-ingress/e2e-api-b-066928-openshift-default-55ddf5c76f-pzlmw"
Oct 30 18:53:21.415785 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:21.415033    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9ff1283-3fc7-463f-b3b7-fafdd56f2349-workload-socket\") pod \"e2e-api-b-066928-openshift-default-55ddf5c76f-pzlmw\" (UID: \"a9ff1283-3fc7-463f-b3b7-fafdd56f2349\") " pod="openshift-ingress/e2e-api-b-066928-openshift-default-55ddf5c76f-pzlmw"
Oct 30 18:53:21.415785 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:21.415634    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9ff1283-3fc7-463f-b3b7-fafdd56f2349-istiod-ca-cert\") pod \"e2e-api-b-066928-openshift-default-55ddf5c76f-pzlmw\" (UID: \"a9ff1283-3fc7-463f-b3b7-fafdd56f2349\") " pod="openshift-ingress/e2e-api-b-066928-openshift-default-55ddf5c76f-pzlmw"
Oct 30 18:53:21.431714 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:21.430456    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhpkq\" (UniqueName: \"kubernetes.io/projected/a9ff1283-3fc7-463f-b3b7-fafdd56f2349-kube-api-access-mhpkq\") pod \"e2e-api-b-066928-openshift-default-55ddf5c76f-pzlmw\" (UID: \"a9ff1283-3fc7-463f-b3b7-fafdd56f2349\") " pod="openshift-ingress/e2e-api-b-066928-openshift-default-55ddf5c76f-pzlmw"
Oct 30 18:53:21.457461 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:21.457348    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9ff1283-3fc7-463f-b3b7-fafdd56f2349-istio-token\") pod \"e2e-api-b-066928-openshift-default-55ddf5c76f-pzlmw\" (UID: \"a9ff1283-3fc7-463f-b3b7-fafdd56f2349\") " pod="openshift-ingress/e2e-api-b-066928-openshift-default-55ddf5c76f-pzlmw"
Oct 30 18:53:21.464737 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:21.461316    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9ff1283-3fc7-463f-b3b7-fafdd56f2349-istio-podinfo\") pod \"e2e-api-b-066928-openshift-default-55ddf5c76f-pzlmw\" (UID: \"a9ff1283-3fc7-463f-b3b7-fafdd56f2349\") " pod="openshift-ingress/e2e-api-b-066928-openshift-default-55ddf5c76f-pzlmw"
Oct 30 18:53:21.473092 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:21.473048    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9ff1283-3fc7-463f-b3b7-fafdd56f2349-istio-envoy\") pod \"e2e-api-b-066928-openshift-default-55ddf5c76f-pzlmw\" (UID: \"a9ff1283-3fc7-463f-b3b7-fafdd56f2349\") " pod="openshift-ingress/e2e-api-b-066928-openshift-default-55ddf5c76f-pzlmw"
Oct 30 18:53:21.640747 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:21.640002    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-7784e5-openshift-default-564776f9f-zm98k"]
Oct 30 18:53:21.649158 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:21.649131    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-7784e5-openshift-default-564776f9f-zm98k"
Oct 30 18:53:21.653533 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:21.653505    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-7784e5-openshift-default-564776f9f-zm98k"]
Oct 30 18:53:21.669741 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:21.664944    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-7784e5-openshift-default-dockercfg-47v78\""
Oct 30 18:53:21.754704 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:21.747964    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-066928-openshift-default-55ddf5c76f-pzlmw"
Oct 30 18:53:21.887704 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:21.885427    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97979488-7a76-4f4e-b330-49432c98bbec-istio-data\") pod \"e2e-api-b-7784e5-openshift-default-564776f9f-zm98k\" (UID: \"97979488-7a76-4f4e-b330-49432c98bbec\") " pod="openshift-ingress/e2e-api-b-7784e5-openshift-default-564776f9f-zm98k"
Oct 30 18:53:21.887704 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:21.885537    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97979488-7a76-4f4e-b330-49432c98bbec-workload-certs\") pod \"e2e-api-b-7784e5-openshift-default-564776f9f-zm98k\" (UID: \"97979488-7a76-4f4e-b330-49432c98bbec\") " pod="openshift-ingress/e2e-api-b-7784e5-openshift-default-564776f9f-zm98k"
Oct 30 18:53:21.887704 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:21.885563    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97979488-7a76-4f4e-b330-49432c98bbec-istio-token\") pod \"e2e-api-b-7784e5-openshift-default-564776f9f-zm98k\" (UID: \"97979488-7a76-4f4e-b330-49432c98bbec\") " pod="openshift-ingress/e2e-api-b-7784e5-openshift-default-564776f9f-zm98k"
Oct 30 18:53:21.887704 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:21.885594    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97979488-7a76-4f4e-b330-49432c98bbec-istio-podinfo\") pod \"e2e-api-b-7784e5-openshift-default-564776f9f-zm98k\" (UID: \"97979488-7a76-4f4e-b330-49432c98bbec\") " pod="openshift-ingress/e2e-api-b-7784e5-openshift-default-564776f9f-zm98k"
Oct 30 18:53:21.887704 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:21.885656    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97979488-7a76-4f4e-b330-49432c98bbec-workload-socket\") pod \"e2e-api-b-7784e5-openshift-default-564776f9f-zm98k\" (UID: \"97979488-7a76-4f4e-b330-49432c98bbec\") " pod="openshift-ingress/e2e-api-b-7784e5-openshift-default-564776f9f-zm98k"
Oct 30 18:53:21.887704 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:21.885824    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpx46\" (UniqueName: \"kubernetes.io/projected/97979488-7a76-4f4e-b330-49432c98bbec-kube-api-access-dpx46\") pod \"e2e-api-b-7784e5-openshift-default-564776f9f-zm98k\" (UID: \"97979488-7a76-4f4e-b330-49432c98bbec\") " pod="openshift-ingress/e2e-api-b-7784e5-openshift-default-564776f9f-zm98k"
Oct 30 18:53:21.887704 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:21.885923    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97979488-7a76-4f4e-b330-49432c98bbec-credential-socket\") pod \"e2e-api-b-7784e5-openshift-default-564776f9f-zm98k\" (UID: \"97979488-7a76-4f4e-b330-49432c98bbec\") " pod="openshift-ingress/e2e-api-b-7784e5-openshift-default-564776f9f-zm98k"
Oct 30 18:53:21.887704 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:21.885952    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97979488-7a76-4f4e-b330-49432c98bbec-istio-envoy\") pod \"e2e-api-b-7784e5-openshift-default-564776f9f-zm98k\" (UID: \"97979488-7a76-4f4e-b330-49432c98bbec\") " pod="openshift-ingress/e2e-api-b-7784e5-openshift-default-564776f9f-zm98k"
Oct 30 18:53:21.887704 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:21.885997    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97979488-7a76-4f4e-b330-49432c98bbec-istiod-ca-cert\") pod \"e2e-api-b-7784e5-openshift-default-564776f9f-zm98k\" (UID: \"97979488-7a76-4f4e-b330-49432c98bbec\") " pod="openshift-ingress/e2e-api-b-7784e5-openshift-default-564776f9f-zm98k"
Oct 30 18:53:21.988411 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:21.987765    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97979488-7a76-4f4e-b330-49432c98bbec-workload-certs\") pod \"e2e-api-b-7784e5-openshift-default-564776f9f-zm98k\" (UID: \"97979488-7a76-4f4e-b330-49432c98bbec\") " pod="openshift-ingress/e2e-api-b-7784e5-openshift-default-564776f9f-zm98k"
Oct 30 18:53:21.988411 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:21.987806    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97979488-7a76-4f4e-b330-49432c98bbec-istio-token\") pod \"e2e-api-b-7784e5-openshift-default-564776f9f-zm98k\" (UID: \"97979488-7a76-4f4e-b330-49432c98bbec\") " pod="openshift-ingress/e2e-api-b-7784e5-openshift-default-564776f9f-zm98k"
Oct 30 18:53:21.988411 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:21.987832    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97979488-7a76-4f4e-b330-49432c98bbec-istio-podinfo\") pod \"e2e-api-b-7784e5-openshift-default-564776f9f-zm98k\" (UID: \"97979488-7a76-4f4e-b330-49432c98bbec\") " pod="openshift-ingress/e2e-api-b-7784e5-openshift-default-564776f9f-zm98k"
Oct 30 18:53:21.988411 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:21.987876    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97979488-7a76-4f4e-b330-49432c98bbec-workload-socket\") pod \"e2e-api-b-7784e5-openshift-default-564776f9f-zm98k\" (UID: \"97979488-7a76-4f4e-b330-49432c98bbec\") " pod="openshift-ingress/e2e-api-b-7784e5-openshift-default-564776f9f-zm98k"
Oct 30 18:53:21.988411 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:21.987940    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpx46\" (UniqueName: \"kubernetes.io/projected/97979488-7a76-4f4e-b330-49432c98bbec-kube-api-access-dpx46\") pod \"e2e-api-b-7784e5-openshift-default-564776f9f-zm98k\" (UID: \"97979488-7a76-4f4e-b330-49432c98bbec\") " pod="openshift-ingress/e2e-api-b-7784e5-openshift-default-564776f9f-zm98k"
Oct 30 18:53:21.988411 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:21.987995    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97979488-7a76-4f4e-b330-49432c98bbec-credential-socket\") pod \"e2e-api-b-7784e5-openshift-default-564776f9f-zm98k\" (UID: \"97979488-7a76-4f4e-b330-49432c98bbec\") " pod="openshift-ingress/e2e-api-b-7784e5-openshift-default-564776f9f-zm98k"
Oct 30 18:53:21.988411 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:21.988015    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97979488-7a76-4f4e-b330-49432c98bbec-istio-envoy\") pod \"e2e-api-b-7784e5-openshift-default-564776f9f-zm98k\" (UID: \"97979488-7a76-4f4e-b330-49432c98bbec\") " pod="openshift-ingress/e2e-api-b-7784e5-openshift-default-564776f9f-zm98k"
Oct 30 18:53:21.988411 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:21.988048    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97979488-7a76-4f4e-b330-49432c98bbec-istiod-ca-cert\") pod \"e2e-api-b-7784e5-openshift-default-564776f9f-zm98k\" (UID: \"97979488-7a76-4f4e-b330-49432c98bbec\") " pod="openshift-ingress/e2e-api-b-7784e5-openshift-default-564776f9f-zm98k"
Oct 30 18:53:21.988411 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:21.988087    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97979488-7a76-4f4e-b330-49432c98bbec-istio-data\") pod \"e2e-api-b-7784e5-openshift-default-564776f9f-zm98k\" (UID: \"97979488-7a76-4f4e-b330-49432c98bbec\") " pod="openshift-ingress/e2e-api-b-7784e5-openshift-default-564776f9f-zm98k"
Oct 30 18:53:21.989368 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:21.988451    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97979488-7a76-4f4e-b330-49432c98bbec-istio-data\") pod \"e2e-api-b-7784e5-openshift-default-564776f9f-zm98k\" (UID: \"97979488-7a76-4f4e-b330-49432c98bbec\") " pod="openshift-ingress/e2e-api-b-7784e5-openshift-default-564776f9f-zm98k"
Oct 30 18:53:21.989368 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:21.988616    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97979488-7a76-4f4e-b330-49432c98bbec-workload-certs\") pod \"e2e-api-b-7784e5-openshift-default-564776f9f-zm98k\" (UID: \"97979488-7a76-4f4e-b330-49432c98bbec\") " pod="openshift-ingress/e2e-api-b-7784e5-openshift-default-564776f9f-zm98k"
Oct 30 18:53:21.989368 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:21.989319    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97979488-7a76-4f4e-b330-49432c98bbec-credential-socket\") pod \"e2e-api-b-7784e5-openshift-default-564776f9f-zm98k\" (UID: \"97979488-7a76-4f4e-b330-49432c98bbec\") " pod="openshift-ingress/e2e-api-b-7784e5-openshift-default-564776f9f-zm98k"
Oct 30 18:53:21.989531 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:21.989505    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97979488-7a76-4f4e-b330-49432c98bbec-workload-socket\") pod \"e2e-api-b-7784e5-openshift-default-564776f9f-zm98k\" (UID: \"97979488-7a76-4f4e-b330-49432c98bbec\") " pod="openshift-ingress/e2e-api-b-7784e5-openshift-default-564776f9f-zm98k"
Oct 30 18:53:21.990476 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:21.990453    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97979488-7a76-4f4e-b330-49432c98bbec-istiod-ca-cert\") pod \"e2e-api-b-7784e5-openshift-default-564776f9f-zm98k\" (UID: \"97979488-7a76-4f4e-b330-49432c98bbec\") " pod="openshift-ingress/e2e-api-b-7784e5-openshift-default-564776f9f-zm98k"
Oct 30 18:53:21.992009 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:21.991989    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97979488-7a76-4f4e-b330-49432c98bbec-istio-podinfo\") pod \"e2e-api-b-7784e5-openshift-default-564776f9f-zm98k\" (UID: \"97979488-7a76-4f4e-b330-49432c98bbec\") " pod="openshift-ingress/e2e-api-b-7784e5-openshift-default-564776f9f-zm98k"
Oct 30 18:53:21.992419 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:21.992397    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97979488-7a76-4f4e-b330-49432c98bbec-istio-envoy\") pod \"e2e-api-b-7784e5-openshift-default-564776f9f-zm98k\" (UID: \"97979488-7a76-4f4e-b330-49432c98bbec\") " pod="openshift-ingress/e2e-api-b-7784e5-openshift-default-564776f9f-zm98k"
Oct 30 18:53:21.998100 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:21.998063    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97979488-7a76-4f4e-b330-49432c98bbec-istio-token\") pod \"e2e-api-b-7784e5-openshift-default-564776f9f-zm98k\" (UID: \"97979488-7a76-4f4e-b330-49432c98bbec\") " pod="openshift-ingress/e2e-api-b-7784e5-openshift-default-564776f9f-zm98k"
Oct 30 18:53:21.998225 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:21.998151    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpx46\" (UniqueName: \"kubernetes.io/projected/97979488-7a76-4f4e-b330-49432c98bbec-kube-api-access-dpx46\") pod \"e2e-api-b-7784e5-openshift-default-564776f9f-zm98k\" (UID: \"97979488-7a76-4f4e-b330-49432c98bbec\") " pod="openshift-ingress/e2e-api-b-7784e5-openshift-default-564776f9f-zm98k"
Oct 30 18:53:22.050255 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:22.049388    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-zv786"
Oct 30 18:53:22.052288 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:22.051551    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-zv786"
Oct 30 18:53:22.053097 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:22.053050    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-zv786"
Oct 30 18:53:22.246841 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:53:22.246115    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9ff1283_3fc7_463f_b3b7_fafdd56f2349.slice/crio-797b6ac2e5e6ffd1e50e7e59d4748b9da3cb994aabacb1481136fa4da18a6461 WatchSource:0}: Error finding container 797b6ac2e5e6ffd1e50e7e59d4748b9da3cb994aabacb1481136fa4da18a6461: Status 404 returned error can't find the container with id 797b6ac2e5e6ffd1e50e7e59d4748b9da3cb994aabacb1481136fa4da18a6461
Oct 30 18:53:22.247926 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:22.247882    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-066928-openshift-default-55ddf5c76f-pzlmw"]
Oct 30 18:53:22.250254 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:22.249862    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:53:22.250254 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:22.249954    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:53:22.250254 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:22.249995    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:53:22.273156 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:22.273134    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-7784e5-openshift-default-564776f9f-zm98k"
Oct 30 18:53:22.411495 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:22.410979    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-066928-openshift-default-55ddf5c76f-pzlmw" event={"ID":"a9ff1283-3fc7-463f-b3b7-fafdd56f2349","Type":"ContainerStarted","Data":"797b6ac2e5e6ffd1e50e7e59d4748b9da3cb994aabacb1481136fa4da18a6461"}
Oct 30 18:53:22.620552 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:22.619841    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-7784e5-openshift-default-564776f9f-zm98k"]
Oct 30 18:53:22.651725 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:22.651365    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:53:22.651725 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:22.651489    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:53:22.651725 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:22.651553    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:53:23.419877 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:23.419836    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7784e5-openshift-default-564776f9f-zm98k" event={"ID":"97979488-7a76-4f4e-b330-49432c98bbec","Type":"ContainerStarted","Data":"c864ba5c8cf4b56772ab67170cbdd9ed6825d5373cae448336a7a11ba15b8609"}
Oct 30 18:53:23.420380 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:23.419927    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7784e5-openshift-default-564776f9f-zm98k" event={"ID":"97979488-7a76-4f4e-b330-49432c98bbec","Type":"ContainerStarted","Data":"fdc1f26257fc1b1eaf24ecd285a6391b55fe11ee51f9060fbb5abf202bafba78"}
Oct 30 18:53:23.422064 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:23.422033    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-066928-openshift-default-55ddf5c76f-pzlmw" event={"ID":"a9ff1283-3fc7-463f-b3b7-fafdd56f2349","Type":"ContainerStarted","Data":"5cfe5306c8da53186d023ff7953bf87c054d9bdea82fcdfad2919e3366f85faf"}
Oct 30 18:53:23.445167 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:23.445125    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-7784e5-openshift-default-564776f9f-zm98k" podStartSLOduration=2.445112607 podStartE2EDuration="2.445112607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:53:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:53:23.439761726 +0000 UTC m=+1026.065431691" watchObservedRunningTime="2026-10-30 18:53:23.445112607 +0000 UTC m=+1026.070782569"
Oct 30 18:53:23.464503 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:23.464452    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-066928-openshift-default-55ddf5c76f-pzlmw" podStartSLOduration=2.464440716 podStartE2EDuration="2.464440716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:53:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:53:23.459720276 +0000 UTC m=+1026.085390239" watchObservedRunningTime="2026-10-30 18:53:23.464440716 +0000 UTC m=+1026.090110679"
Oct 30 18:53:23.766769 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:23.762630    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-066928-openshift-default-55ddf5c76f-pzlmw"
Oct 30 18:53:23.797051 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:23.797017    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-066928-openshift-default-55ddf5c76f-pzlmw"
Oct 30 18:53:24.274246 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:24.274211    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-7784e5-openshift-default-564776f9f-zm98k"
Oct 30 18:53:24.280107 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:24.280081    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-7784e5-openshift-default-564776f9f-zm98k"
Oct 30 18:53:24.467276 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:24.467243    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-066928-openshift-default-55ddf5c76f-pzlmw"
Oct 30 18:53:24.468160 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:24.468139    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-7784e5-openshift-default-564776f9f-zm98k"
Oct 30 18:53:24.472301 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:24.472276    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-066928-openshift-default-55ddf5c76f-pzlmw"
Oct 30 18:53:24.473046 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:24.473026    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-7784e5-openshift-default-564776f9f-zm98k"
Oct 30 18:53:35.830250 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:35.830210    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-22qrl"]
Oct 30 18:53:35.834739 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:35.834715    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-22qrl"
Oct 30 18:53:35.843552 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:35.843524    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-f37d42-openshift-default-dockercfg-9hc5n\""
Oct 30 18:53:35.848404 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:35.848367    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-22qrl"]
Oct 30 18:53:35.990966 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:35.990926    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13c9b518-512f-4a88-ae9e-cd8e9fb247cb-workload-certs\") pod \"e2e-api-a-f37d42-openshift-default-7645c57f7c-22qrl\" (UID: \"13c9b518-512f-4a88-ae9e-cd8e9fb247cb\") " pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-22qrl"
Oct 30 18:53:35.991162 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:35.991028    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13c9b518-512f-4a88-ae9e-cd8e9fb247cb-istio-data\") pod \"e2e-api-a-f37d42-openshift-default-7645c57f7c-22qrl\" (UID: \"13c9b518-512f-4a88-ae9e-cd8e9fb247cb\") " pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-22qrl"
Oct 30 18:53:35.991162 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:35.991059    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smh8j\" (UniqueName: \"kubernetes.io/projected/13c9b518-512f-4a88-ae9e-cd8e9fb247cb-kube-api-access-smh8j\") pod \"e2e-api-a-f37d42-openshift-default-7645c57f7c-22qrl\" (UID: \"13c9b518-512f-4a88-ae9e-cd8e9fb247cb\") " pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-22qrl"
Oct 30 18:53:35.991285 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:35.991199    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13c9b518-512f-4a88-ae9e-cd8e9fb247cb-workload-socket\") pod \"e2e-api-a-f37d42-openshift-default-7645c57f7c-22qrl\" (UID: \"13c9b518-512f-4a88-ae9e-cd8e9fb247cb\") " pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-22qrl"
Oct 30 18:53:35.991343 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:35.991302    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13c9b518-512f-4a88-ae9e-cd8e9fb247cb-istiod-ca-cert\") pod \"e2e-api-a-f37d42-openshift-default-7645c57f7c-22qrl\" (UID: \"13c9b518-512f-4a88-ae9e-cd8e9fb247cb\") " pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-22qrl"
Oct 30 18:53:35.991343 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:35.991335    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13c9b518-512f-4a88-ae9e-cd8e9fb247cb-istio-podinfo\") pod \"e2e-api-a-f37d42-openshift-default-7645c57f7c-22qrl\" (UID: \"13c9b518-512f-4a88-ae9e-cd8e9fb247cb\") " pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-22qrl"
Oct 30 18:53:35.991433 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:35.991369    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13c9b518-512f-4a88-ae9e-cd8e9fb247cb-istio-envoy\") pod \"e2e-api-a-f37d42-openshift-default-7645c57f7c-22qrl\" (UID: \"13c9b518-512f-4a88-ae9e-cd8e9fb247cb\") " pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-22qrl"
Oct 30 18:53:35.991524 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:35.991502    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13c9b518-512f-4a88-ae9e-cd8e9fb247cb-istio-token\") pod \"e2e-api-a-f37d42-openshift-default-7645c57f7c-22qrl\" (UID: \"13c9b518-512f-4a88-ae9e-cd8e9fb247cb\") " pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-22qrl"
Oct 30 18:53:35.991559 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:35.991536    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13c9b518-512f-4a88-ae9e-cd8e9fb247cb-credential-socket\") pod \"e2e-api-a-f37d42-openshift-default-7645c57f7c-22qrl\" (UID: \"13c9b518-512f-4a88-ae9e-cd8e9fb247cb\") " pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-22qrl"
Oct 30 18:53:36.092649 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:36.092570    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13c9b518-512f-4a88-ae9e-cd8e9fb247cb-istio-token\") pod \"e2e-api-a-f37d42-openshift-default-7645c57f7c-22qrl\" (UID: \"13c9b518-512f-4a88-ae9e-cd8e9fb247cb\") " pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-22qrl"
Oct 30 18:53:36.092649 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:36.092609    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13c9b518-512f-4a88-ae9e-cd8e9fb247cb-credential-socket\") pod \"e2e-api-a-f37d42-openshift-default-7645c57f7c-22qrl\" (UID: \"13c9b518-512f-4a88-ae9e-cd8e9fb247cb\") " pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-22qrl"
Oct 30 18:53:36.092912 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:36.092716    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13c9b518-512f-4a88-ae9e-cd8e9fb247cb-workload-certs\") pod \"e2e-api-a-f37d42-openshift-default-7645c57f7c-22qrl\" (UID: \"13c9b518-512f-4a88-ae9e-cd8e9fb247cb\") " pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-22qrl"
Oct 30 18:53:36.092912 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:36.092737    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13c9b518-512f-4a88-ae9e-cd8e9fb247cb-istio-data\") pod \"e2e-api-a-f37d42-openshift-default-7645c57f7c-22qrl\" (UID: \"13c9b518-512f-4a88-ae9e-cd8e9fb247cb\") " pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-22qrl"
Oct 30 18:53:36.092912 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:36.092754    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smh8j\" (UniqueName: \"kubernetes.io/projected/13c9b518-512f-4a88-ae9e-cd8e9fb247cb-kube-api-access-smh8j\") pod \"e2e-api-a-f37d42-openshift-default-7645c57f7c-22qrl\" (UID: \"13c9b518-512f-4a88-ae9e-cd8e9fb247cb\") " pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-22qrl"
Oct 30 18:53:36.092912 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:36.092829    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13c9b518-512f-4a88-ae9e-cd8e9fb247cb-workload-socket\") pod \"e2e-api-a-f37d42-openshift-default-7645c57f7c-22qrl\" (UID: \"13c9b518-512f-4a88-ae9e-cd8e9fb247cb\") " pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-22qrl"
Oct 30 18:53:36.092912 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:36.092874    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13c9b518-512f-4a88-ae9e-cd8e9fb247cb-istiod-ca-cert\") pod \"e2e-api-a-f37d42-openshift-default-7645c57f7c-22qrl\" (UID: \"13c9b518-512f-4a88-ae9e-cd8e9fb247cb\") " pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-22qrl"
Oct 30 18:53:36.092912 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:36.092904    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13c9b518-512f-4a88-ae9e-cd8e9fb247cb-istio-podinfo\") pod \"e2e-api-a-f37d42-openshift-default-7645c57f7c-22qrl\" (UID: \"13c9b518-512f-4a88-ae9e-cd8e9fb247cb\") " pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-22qrl"
Oct 30 18:53:36.093215 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:36.092936    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13c9b518-512f-4a88-ae9e-cd8e9fb247cb-istio-envoy\") pod \"e2e-api-a-f37d42-openshift-default-7645c57f7c-22qrl\" (UID: \"13c9b518-512f-4a88-ae9e-cd8e9fb247cb\") " pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-22qrl"
Oct 30 18:53:36.093215 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:36.093153    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13c9b518-512f-4a88-ae9e-cd8e9fb247cb-credential-socket\") pod \"e2e-api-a-f37d42-openshift-default-7645c57f7c-22qrl\" (UID: \"13c9b518-512f-4a88-ae9e-cd8e9fb247cb\") " pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-22qrl"
Oct 30 18:53:36.093215 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:36.093188    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13c9b518-512f-4a88-ae9e-cd8e9fb247cb-istio-data\") pod \"e2e-api-a-f37d42-openshift-default-7645c57f7c-22qrl\" (UID: \"13c9b518-512f-4a88-ae9e-cd8e9fb247cb\") " pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-22qrl"
Oct 30 18:53:36.093377 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:36.093288    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13c9b518-512f-4a88-ae9e-cd8e9fb247cb-workload-socket\") pod \"e2e-api-a-f37d42-openshift-default-7645c57f7c-22qrl\" (UID: \"13c9b518-512f-4a88-ae9e-cd8e9fb247cb\") " pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-22qrl"
Oct 30 18:53:36.093494 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:36.093474    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13c9b518-512f-4a88-ae9e-cd8e9fb247cb-workload-certs\") pod \"e2e-api-a-f37d42-openshift-default-7645c57f7c-22qrl\" (UID: \"13c9b518-512f-4a88-ae9e-cd8e9fb247cb\") " pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-22qrl"
Oct 30 18:53:36.093551 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:36.093514    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13c9b518-512f-4a88-ae9e-cd8e9fb247cb-istiod-ca-cert\") pod \"e2e-api-a-f37d42-openshift-default-7645c57f7c-22qrl\" (UID: \"13c9b518-512f-4a88-ae9e-cd8e9fb247cb\") " pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-22qrl"
Oct 30 18:53:36.095406 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:36.095387    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13c9b518-512f-4a88-ae9e-cd8e9fb247cb-istio-envoy\") pod \"e2e-api-a-f37d42-openshift-default-7645c57f7c-22qrl\" (UID: \"13c9b518-512f-4a88-ae9e-cd8e9fb247cb\") " pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-22qrl"
Oct 30 18:53:36.095552 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:36.095534    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13c9b518-512f-4a88-ae9e-cd8e9fb247cb-istio-podinfo\") pod \"e2e-api-a-f37d42-openshift-default-7645c57f7c-22qrl\" (UID: \"13c9b518-512f-4a88-ae9e-cd8e9fb247cb\") " pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-22qrl"
Oct 30 18:53:36.101436 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:36.101406    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13c9b518-512f-4a88-ae9e-cd8e9fb247cb-istio-token\") pod \"e2e-api-a-f37d42-openshift-default-7645c57f7c-22qrl\" (UID: \"13c9b518-512f-4a88-ae9e-cd8e9fb247cb\") " pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-22qrl"
Oct 30 18:53:36.101590 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:36.101571    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smh8j\" (UniqueName: \"kubernetes.io/projected/13c9b518-512f-4a88-ae9e-cd8e9fb247cb-kube-api-access-smh8j\") pod \"e2e-api-a-f37d42-openshift-default-7645c57f7c-22qrl\" (UID: \"13c9b518-512f-4a88-ae9e-cd8e9fb247cb\") " pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-22qrl"
Oct 30 18:53:36.151022 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:36.150984    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-22qrl"
Oct 30 18:53:36.443282 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:36.443243    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-22qrl"]
Oct 30 18:53:36.445811 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:53:36.445776    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13c9b518_512f_4a88_ae9e_cd8e9fb247cb.slice/crio-8aaa71ab48983fb2236d70e51b577e0d4c7c15f6d8aa0bf25032fcfb75fe678e WatchSource:0}: Error finding container 8aaa71ab48983fb2236d70e51b577e0d4c7c15f6d8aa0bf25032fcfb75fe678e: Status 404 returned error can't find the container with id 8aaa71ab48983fb2236d70e51b577e0d4c7c15f6d8aa0bf25032fcfb75fe678e
Oct 30 18:53:36.448309 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:36.448275    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:53:36.448398 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:36.448358    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:53:36.448441 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:36.448398    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:53:36.612666 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:36.612636    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-22qrl" event={"ID":"13c9b518-512f-4a88-ae9e-cd8e9fb247cb","Type":"ContainerStarted","Data":"b95c5130210fa52372bd5ff3ef7c5602ec5026b642a357cc40f3aeb88fff06be"}
Oct 30 18:53:36.612859 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:36.612726    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-22qrl" event={"ID":"13c9b518-512f-4a88-ae9e-cd8e9fb247cb","Type":"ContainerStarted","Data":"8aaa71ab48983fb2236d70e51b577e0d4c7c15f6d8aa0bf25032fcfb75fe678e"}
Oct 30 18:53:36.640292 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:36.640228    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-22qrl" podStartSLOduration=1.6402085579999999 podStartE2EDuration="1.640208558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:53:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:53:36.633131074 +0000 UTC m=+1039.258801038" watchObservedRunningTime="2026-10-30 18:53:36.640208558 +0000 UTC m=+1039.265878528"
Oct 30 18:53:37.151476 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:37.151442    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-22qrl"
Oct 30 18:53:37.153436 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:37.153408    4525 patch_prober.go:29] interesting pod/e2e-api-a-f37d42-openshift-default-7645c57f7c-22qrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 30 18:53:37.153566 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:37.153469    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-22qrl" podUID="13c9b518-512f-4a88-ae9e-cd8e9fb247cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 30 18:53:38.152057 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:38.152019    4525 patch_prober.go:29] interesting pod/e2e-api-a-f37d42-openshift-default-7645c57f7c-22qrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 30 18:53:38.152484 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:38.152079    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-22qrl" podUID="13c9b518-512f-4a88-ae9e-cd8e9fb247cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 30 18:53:39.151968 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:39.151932    4525 patch_prober.go:29] interesting pod/e2e-api-a-f37d42-openshift-default-7645c57f7c-22qrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 30 18:53:39.152156 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:39.151992    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-22qrl" podUID="13c9b518-512f-4a88-ae9e-cd8e9fb247cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 30 18:53:39.530181 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:39.529848    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-da58b769-openshift-default-69887ddf8f-n79ls"]
Oct 30 18:53:39.534593 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:39.534570    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-da58b769-openshift-default-69887ddf8f-n79ls"
Oct 30 18:53:39.546143 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:39.543481    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-da58b769-openshift-default-dockercfg-27gmd\""
Oct 30 18:53:39.556852 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:39.556820    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-da58b769-openshift-default-69887ddf8f-n79ls"]
Oct 30 18:53:39.826183 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:39.826073    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81739e60-d074-4a42-ade3-a0f68ea9329d-istio-token\") pod \"e2e-mt-da58b769-openshift-default-69887ddf8f-n79ls\" (UID: \"81739e60-d074-4a42-ade3-a0f68ea9329d\") " pod="openshift-ingress/e2e-mt-da58b769-openshift-default-69887ddf8f-n79ls"
Oct 30 18:53:39.831703 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:39.830874    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81739e60-d074-4a42-ade3-a0f68ea9329d-credential-socket\") pod \"e2e-mt-da58b769-openshift-default-69887ddf8f-n79ls\" (UID: \"81739e60-d074-4a42-ade3-a0f68ea9329d\") " pod="openshift-ingress/e2e-mt-da58b769-openshift-default-69887ddf8f-n79ls"
Oct 30 18:53:39.831703 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:39.830923    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81739e60-d074-4a42-ade3-a0f68ea9329d-istio-envoy\") pod \"e2e-mt-da58b769-openshift-default-69887ddf8f-n79ls\" (UID: \"81739e60-d074-4a42-ade3-a0f68ea9329d\") " pod="openshift-ingress/e2e-mt-da58b769-openshift-default-69887ddf8f-n79ls"
Oct 30 18:53:39.831703 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:39.830974    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81739e60-d074-4a42-ade3-a0f68ea9329d-workload-certs\") pod \"e2e-mt-da58b769-openshift-default-69887ddf8f-n79ls\" (UID: \"81739e60-d074-4a42-ade3-a0f68ea9329d\") " pod="openshift-ingress/e2e-mt-da58b769-openshift-default-69887ddf8f-n79ls"
Oct 30 18:53:39.831703 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:39.831016    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81739e60-d074-4a42-ade3-a0f68ea9329d-istio-podinfo\") pod \"e2e-mt-da58b769-openshift-default-69887ddf8f-n79ls\" (UID: \"81739e60-d074-4a42-ade3-a0f68ea9329d\") " pod="openshift-ingress/e2e-mt-da58b769-openshift-default-69887ddf8f-n79ls"
Oct 30 18:53:39.831703 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:39.831087    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81739e60-d074-4a42-ade3-a0f68ea9329d-istiod-ca-cert\") pod \"e2e-mt-da58b769-openshift-default-69887ddf8f-n79ls\" (UID: \"81739e60-d074-4a42-ade3-a0f68ea9329d\") " pod="openshift-ingress/e2e-mt-da58b769-openshift-default-69887ddf8f-n79ls"
Oct 30 18:53:39.831703 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:39.831147    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47ncm\" (UniqueName: \"kubernetes.io/projected/81739e60-d074-4a42-ade3-a0f68ea9329d-kube-api-access-47ncm\") pod \"e2e-mt-da58b769-openshift-default-69887ddf8f-n79ls\" (UID: \"81739e60-d074-4a42-ade3-a0f68ea9329d\") " pod="openshift-ingress/e2e-mt-da58b769-openshift-default-69887ddf8f-n79ls"
Oct 30 18:53:39.831703 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:39.831214    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81739e60-d074-4a42-ade3-a0f68ea9329d-workload-socket\") pod \"e2e-mt-da58b769-openshift-default-69887ddf8f-n79ls\" (UID: \"81739e60-d074-4a42-ade3-a0f68ea9329d\") " pod="openshift-ingress/e2e-mt-da58b769-openshift-default-69887ddf8f-n79ls"
Oct 30 18:53:39.831703 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:39.831248    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81739e60-d074-4a42-ade3-a0f68ea9329d-istio-data\") pod \"e2e-mt-da58b769-openshift-default-69887ddf8f-n79ls\" (UID: \"81739e60-d074-4a42-ade3-a0f68ea9329d\") " pod="openshift-ingress/e2e-mt-da58b769-openshift-default-69887ddf8f-n79ls"
Oct 30 18:53:39.934204 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:39.934085    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81739e60-d074-4a42-ade3-a0f68ea9329d-credential-socket\") pod \"e2e-mt-da58b769-openshift-default-69887ddf8f-n79ls\" (UID: \"81739e60-d074-4a42-ade3-a0f68ea9329d\") " pod="openshift-ingress/e2e-mt-da58b769-openshift-default-69887ddf8f-n79ls"
Oct 30 18:53:39.934204 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:39.934137    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81739e60-d074-4a42-ade3-a0f68ea9329d-istio-envoy\") pod \"e2e-mt-da58b769-openshift-default-69887ddf8f-n79ls\" (UID: \"81739e60-d074-4a42-ade3-a0f68ea9329d\") " pod="openshift-ingress/e2e-mt-da58b769-openshift-default-69887ddf8f-n79ls"
Oct 30 18:53:39.934204 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:39.934182    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81739e60-d074-4a42-ade3-a0f68ea9329d-workload-certs\") pod \"e2e-mt-da58b769-openshift-default-69887ddf8f-n79ls\" (UID: \"81739e60-d074-4a42-ade3-a0f68ea9329d\") " pod="openshift-ingress/e2e-mt-da58b769-openshift-default-69887ddf8f-n79ls"
Oct 30 18:53:39.934509 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:39.934217    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81739e60-d074-4a42-ade3-a0f68ea9329d-istio-podinfo\") pod \"e2e-mt-da58b769-openshift-default-69887ddf8f-n79ls\" (UID: \"81739e60-d074-4a42-ade3-a0f68ea9329d\") " pod="openshift-ingress/e2e-mt-da58b769-openshift-default-69887ddf8f-n79ls"
Oct 30 18:53:39.934509 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:39.934278    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81739e60-d074-4a42-ade3-a0f68ea9329d-istiod-ca-cert\") pod \"e2e-mt-da58b769-openshift-default-69887ddf8f-n79ls\" (UID: \"81739e60-d074-4a42-ade3-a0f68ea9329d\") " pod="openshift-ingress/e2e-mt-da58b769-openshift-default-69887ddf8f-n79ls"
Oct 30 18:53:39.934509 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:39.934303    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47ncm\" (UniqueName: \"kubernetes.io/projected/81739e60-d074-4a42-ade3-a0f68ea9329d-kube-api-access-47ncm\") pod \"e2e-mt-da58b769-openshift-default-69887ddf8f-n79ls\" (UID: \"81739e60-d074-4a42-ade3-a0f68ea9329d\") " pod="openshift-ingress/e2e-mt-da58b769-openshift-default-69887ddf8f-n79ls"
Oct 30 18:53:39.934509 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:39.934353    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81739e60-d074-4a42-ade3-a0f68ea9329d-workload-socket\") pod \"e2e-mt-da58b769-openshift-default-69887ddf8f-n79ls\" (UID: \"81739e60-d074-4a42-ade3-a0f68ea9329d\") " pod="openshift-ingress/e2e-mt-da58b769-openshift-default-69887ddf8f-n79ls"
Oct 30 18:53:39.934509 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:39.934383    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81739e60-d074-4a42-ade3-a0f68ea9329d-istio-data\") pod \"e2e-mt-da58b769-openshift-default-69887ddf8f-n79ls\" (UID: \"81739e60-d074-4a42-ade3-a0f68ea9329d\") " pod="openshift-ingress/e2e-mt-da58b769-openshift-default-69887ddf8f-n79ls"
Oct 30 18:53:39.934509 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:39.934494    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81739e60-d074-4a42-ade3-a0f68ea9329d-istio-token\") pod \"e2e-mt-da58b769-openshift-default-69887ddf8f-n79ls\" (UID: \"81739e60-d074-4a42-ade3-a0f68ea9329d\") " pod="openshift-ingress/e2e-mt-da58b769-openshift-default-69887ddf8f-n79ls"
Oct 30 18:53:39.934840 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:39.934553    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81739e60-d074-4a42-ade3-a0f68ea9329d-credential-socket\") pod \"e2e-mt-da58b769-openshift-default-69887ddf8f-n79ls\" (UID: \"81739e60-d074-4a42-ade3-a0f68ea9329d\") " pod="openshift-ingress/e2e-mt-da58b769-openshift-default-69887ddf8f-n79ls"
Oct 30 18:53:39.935261 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:39.935227    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81739e60-d074-4a42-ade3-a0f68ea9329d-workload-certs\") pod \"e2e-mt-da58b769-openshift-default-69887ddf8f-n79ls\" (UID: \"81739e60-d074-4a42-ade3-a0f68ea9329d\") " pod="openshift-ingress/e2e-mt-da58b769-openshift-default-69887ddf8f-n79ls"
Oct 30 18:53:39.935446 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:39.935254    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81739e60-d074-4a42-ade3-a0f68ea9329d-istiod-ca-cert\") pod \"e2e-mt-da58b769-openshift-default-69887ddf8f-n79ls\" (UID: \"81739e60-d074-4a42-ade3-a0f68ea9329d\") " pod="openshift-ingress/e2e-mt-da58b769-openshift-default-69887ddf8f-n79ls"
Oct 30 18:53:39.935796 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:39.935774    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81739e60-d074-4a42-ade3-a0f68ea9329d-istio-data\") pod \"e2e-mt-da58b769-openshift-default-69887ddf8f-n79ls\" (UID: \"81739e60-d074-4a42-ade3-a0f68ea9329d\") " pod="openshift-ingress/e2e-mt-da58b769-openshift-default-69887ddf8f-n79ls"
Oct 30 18:53:39.936046 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:39.936010    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81739e60-d074-4a42-ade3-a0f68ea9329d-workload-socket\") pod \"e2e-mt-da58b769-openshift-default-69887ddf8f-n79ls\" (UID: \"81739e60-d074-4a42-ade3-a0f68ea9329d\") " pod="openshift-ingress/e2e-mt-da58b769-openshift-default-69887ddf8f-n79ls"
Oct 30 18:53:39.938036 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:39.938000    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81739e60-d074-4a42-ade3-a0f68ea9329d-istio-envoy\") pod \"e2e-mt-da58b769-openshift-default-69887ddf8f-n79ls\" (UID: \"81739e60-d074-4a42-ade3-a0f68ea9329d\") " pod="openshift-ingress/e2e-mt-da58b769-openshift-default-69887ddf8f-n79ls"
Oct 30 18:53:39.938497 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:39.938478    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81739e60-d074-4a42-ade3-a0f68ea9329d-istio-podinfo\") pod \"e2e-mt-da58b769-openshift-default-69887ddf8f-n79ls\" (UID: \"81739e60-d074-4a42-ade3-a0f68ea9329d\") " pod="openshift-ingress/e2e-mt-da58b769-openshift-default-69887ddf8f-n79ls"
Oct 30 18:53:39.943880 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:39.943857    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81739e60-d074-4a42-ade3-a0f68ea9329d-istio-token\") pod \"e2e-mt-da58b769-openshift-default-69887ddf8f-n79ls\" (UID: \"81739e60-d074-4a42-ade3-a0f68ea9329d\") " pod="openshift-ingress/e2e-mt-da58b769-openshift-default-69887ddf8f-n79ls"
Oct 30 18:53:39.943978 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:39.943964    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47ncm\" (UniqueName: \"kubernetes.io/projected/81739e60-d074-4a42-ade3-a0f68ea9329d-kube-api-access-47ncm\") pod \"e2e-mt-da58b769-openshift-default-69887ddf8f-n79ls\" (UID: \"81739e60-d074-4a42-ade3-a0f68ea9329d\") " pod="openshift-ingress/e2e-mt-da58b769-openshift-default-69887ddf8f-n79ls"
Oct 30 18:53:40.151673 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:40.151640    4525 patch_prober.go:29] interesting pod/e2e-api-a-f37d42-openshift-default-7645c57f7c-22qrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 30 18:53:40.151915 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:40.151725    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-22qrl" podUID="13c9b518-512f-4a88-ae9e-cd8e9fb247cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 30 18:53:40.160054 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:40.160028    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-da58b769-openshift-default-69887ddf8f-n79ls"
Oct 30 18:53:40.528457 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:40.528426    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-da58b769-openshift-default-69887ddf8f-n79ls"]
Oct 30 18:53:40.532199 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:40.532167    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:53:40.532312 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:40.532227    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:53:40.532312 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:40.532258    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:53:40.813282 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:40.813179    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-da58b769-openshift-default-69887ddf8f-n79ls" event={"ID":"81739e60-d074-4a42-ade3-a0f68ea9329d","Type":"ContainerStarted","Data":"3eb61f7ea9b79c37b1f93a3a5f2a93a34be89622062ad9a76e12470e5e20c9a3"}
Oct 30 18:53:40.813282 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:40.813245    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-da58b769-openshift-default-69887ddf8f-n79ls" event={"ID":"81739e60-d074-4a42-ade3-a0f68ea9329d","Type":"ContainerStarted","Data":"a9e53e895cc3982b47223434f429e0cc2d970db44cc294a70c3fb71e570a98d0"}
Oct 30 18:53:40.842664 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:40.842563    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-da58b769-openshift-default-69887ddf8f-n79ls" podStartSLOduration=1.842548069 podStartE2EDuration="1.842548069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:53:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:53:40.839588141 +0000 UTC m=+1043.465258106" watchObservedRunningTime="2026-10-30 18:53:40.842548069 +0000 UTC m=+1043.468218032"
Oct 30 18:53:41.152205 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:41.152166    4525 patch_prober.go:29] interesting pod/e2e-api-a-f37d42-openshift-default-7645c57f7c-22qrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 30 18:53:41.154959 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:41.152218    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-22qrl" podUID="13c9b518-512f-4a88-ae9e-cd8e9fb247cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 30 18:53:41.160652 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:41.160633    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-da58b769-openshift-default-69887ddf8f-n79ls"
Oct 30 18:53:41.166207 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:41.166187    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-da58b769-openshift-default-69887ddf8f-n79ls"
Oct 30 18:53:41.909782 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:41.909747    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-da58b769-openshift-default-69887ddf8f-n79ls"
Oct 30 18:53:41.925071 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:41.924649    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-da58b769-openshift-default-69887ddf8f-n79ls"
Oct 30 18:53:42.151830 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:42.151793    4525 patch_prober.go:29] interesting pod/e2e-api-a-f37d42-openshift-default-7645c57f7c-22qrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 30 18:53:42.152008 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:42.151845    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-22qrl" podUID="13c9b518-512f-4a88-ae9e-cd8e9fb247cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 30 18:53:43.156512 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:43.156480    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-22qrl"
Oct 30 18:53:43.157255 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:43.157209    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-22qrl"
Oct 30 18:53:43.158196 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:43.158174    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-22qrl"
Oct 30 18:53:46.919054 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:46.919019    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-87293f-rz82z"]
Oct 30 18:53:46.924149 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:46.924122    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-87293f-rz82z"
Oct 30 18:53:46.927646 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:46.927616    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-87293f-rz82z"]
Oct 30 18:53:46.927891 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:46.927788    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:53:46.928309 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:46.928287    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94hcn\""
Oct 30 18:53:47.045280 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:47.045250    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9edb2112-f66a-467d-8384-7343736ae49b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-87293f-rz82z\" (UID: \"9edb2112-f66a-467d-8384-7343736ae49b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-87293f-rz82z"
Oct 30 18:53:47.045451 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:47.045390    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6ctt\" (UniqueName: \"kubernetes.io/projected/9edb2112-f66a-467d-8384-7343736ae49b-kube-api-access-r6ctt\") pod \"maas-api-revoke-keys-e2e-api-a-87293f-rz82z\" (UID: \"9edb2112-f66a-467d-8384-7343736ae49b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-87293f-rz82z"
Oct 30 18:53:47.045521 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:47.045506    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9edb2112-f66a-467d-8384-7343736ae49b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-87293f-rz82z\" (UID: \"9edb2112-f66a-467d-8384-7343736ae49b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-87293f-rz82z"
Oct 30 18:53:47.146882 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:47.146846    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6ctt\" (UniqueName: \"kubernetes.io/projected/9edb2112-f66a-467d-8384-7343736ae49b-kube-api-access-r6ctt\") pod \"maas-api-revoke-keys-e2e-api-a-87293f-rz82z\" (UID: \"9edb2112-f66a-467d-8384-7343736ae49b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-87293f-rz82z"
Oct 30 18:53:47.147086 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:47.146962    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9edb2112-f66a-467d-8384-7343736ae49b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-87293f-rz82z\" (UID: \"9edb2112-f66a-467d-8384-7343736ae49b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-87293f-rz82z"
Oct 30 18:53:47.147153 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:47.147121    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9edb2112-f66a-467d-8384-7343736ae49b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-87293f-rz82z\" (UID: \"9edb2112-f66a-467d-8384-7343736ae49b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-87293f-rz82z"
Oct 30 18:53:47.147894 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:47.147868    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9edb2112-f66a-467d-8384-7343736ae49b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-87293f-rz82z\" (UID: \"9edb2112-f66a-467d-8384-7343736ae49b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-87293f-rz82z"
Oct 30 18:53:47.150301 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:47.150274    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9edb2112-f66a-467d-8384-7343736ae49b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-87293f-rz82z\" (UID: \"9edb2112-f66a-467d-8384-7343736ae49b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-87293f-rz82z"
Oct 30 18:53:47.159069 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:47.159038    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6ctt\" (UniqueName: \"kubernetes.io/projected/9edb2112-f66a-467d-8384-7343736ae49b-kube-api-access-r6ctt\") pod \"maas-api-revoke-keys-e2e-api-a-87293f-rz82z\" (UID: \"9edb2112-f66a-467d-8384-7343736ae49b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-87293f-rz82z"
Oct 30 18:53:47.244006 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:47.243921    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-87293f-rz82z"
Oct 30 18:53:47.399612 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:53:47.399572    4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d49l" containerName="extractor"
Oct 30 18:53:47.529866 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:53:47.529835    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9edb2112_f66a_467d_8384_7343736ae49b.slice/crio-046433ac2713d2d7b7267029b59871707b151e2e73a91215e4ec5668c6d1c27e WatchSource:0}: Error finding container 046433ac2713d2d7b7267029b59871707b151e2e73a91215e4ec5668c6d1c27e: Status 404 returned error can't find the container with id 046433ac2713d2d7b7267029b59871707b151e2e73a91215e4ec5668c6d1c27e
Oct 30 18:53:47.531329 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:47.531302    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-87293f-rz82z"]
Oct 30 18:53:48.094288 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:48.094151    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-87293f-rz82z" event={"ID":"9edb2112-f66a-467d-8384-7343736ae49b","Type":"ContainerStarted","Data":"026934c3e4e871026d479d332ab691137acc2366bc2df0567ca437b92ba2d49e"}
Oct 30 18:53:48.094664 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:48.094291    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-87293f-rz82z" event={"ID":"9edb2112-f66a-467d-8384-7343736ae49b","Type":"ContainerStarted","Data":"046433ac2713d2d7b7267029b59871707b151e2e73a91215e4ec5668c6d1c27e"}
Oct 30 18:53:48.113561 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:48.113517    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-87293f-rz82z" podStartSLOduration=2.113504705 podStartE2EDuration="2.113504705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:53:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:53:48.111815067 +0000 UTC m=+1050.737485031" watchObservedRunningTime="2026-10-30 18:53:48.113504705 +0000 UTC m=+1050.739174725"
Oct 30 18:53:48.238912 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:48.238875    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f37d42-5dwmd"]
Oct 30 18:53:48.243138 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:48.243120    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f37d42-5dwmd"
Oct 30 18:53:48.250904 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:48.250882    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f37d42-5dwmd"]
Oct 30 18:53:48.364188 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:48.364157    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39ac8767-10a6-449f-884f-7d3b716657a2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f37d42-5dwmd\" (UID: \"39ac8767-10a6-449f-884f-7d3b716657a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f37d42-5dwmd"
Oct 30 18:53:48.364348 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:48.364222    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xt7zw\" (UniqueName: \"kubernetes.io/projected/39ac8767-10a6-449f-884f-7d3b716657a2-kube-api-access-xt7zw\") pod \"maas-api-revoke-keys-e2e-api-a-f37d42-5dwmd\" (UID: \"39ac8767-10a6-449f-884f-7d3b716657a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f37d42-5dwmd"
Oct 30 18:53:48.364423 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:48.364406    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39ac8767-10a6-449f-884f-7d3b716657a2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f37d42-5dwmd\" (UID: \"39ac8767-10a6-449f-884f-7d3b716657a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f37d42-5dwmd"
Oct 30 18:53:48.465552 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:48.465512    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39ac8767-10a6-449f-884f-7d3b716657a2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f37d42-5dwmd\" (UID: \"39ac8767-10a6-449f-884f-7d3b716657a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f37d42-5dwmd"
Oct 30 18:53:48.465754 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:48.465595    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xt7zw\" (UniqueName: \"kubernetes.io/projected/39ac8767-10a6-449f-884f-7d3b716657a2-kube-api-access-xt7zw\") pod \"maas-api-revoke-keys-e2e-api-a-f37d42-5dwmd\" (UID: \"39ac8767-10a6-449f-884f-7d3b716657a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f37d42-5dwmd"
Oct 30 18:53:48.465754 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:48.465728    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39ac8767-10a6-449f-884f-7d3b716657a2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f37d42-5dwmd\" (UID: \"39ac8767-10a6-449f-884f-7d3b716657a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f37d42-5dwmd"
Oct 30 18:53:48.466403 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:48.466370    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39ac8767-10a6-449f-884f-7d3b716657a2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f37d42-5dwmd\" (UID: \"39ac8767-10a6-449f-884f-7d3b716657a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f37d42-5dwmd"
Oct 30 18:53:48.468130 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:48.468110    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39ac8767-10a6-449f-884f-7d3b716657a2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f37d42-5dwmd\" (UID: \"39ac8767-10a6-449f-884f-7d3b716657a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f37d42-5dwmd"
Oct 30 18:53:48.474566 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:48.474548    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xt7zw\" (UniqueName: \"kubernetes.io/projected/39ac8767-10a6-449f-884f-7d3b716657a2-kube-api-access-xt7zw\") pod \"maas-api-revoke-keys-e2e-api-a-f37d42-5dwmd\" (UID: \"39ac8767-10a6-449f-884f-7d3b716657a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f37d42-5dwmd"
Oct 30 18:53:48.562445 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:48.562412    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f37d42-5dwmd"
Oct 30 18:53:48.856209 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:53:48.856185    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39ac8767_10a6_449f_884f_7d3b716657a2.slice/crio-843140a915679eeb2ac81458f08338a7e1fc50511b672258b712eaf66142f707 WatchSource:0}: Error finding container 843140a915679eeb2ac81458f08338a7e1fc50511b672258b712eaf66142f707: Status 404 returned error can't find the container with id 843140a915679eeb2ac81458f08338a7e1fc50511b672258b712eaf66142f707
Oct 30 18:53:48.857233 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:48.857204    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f37d42-5dwmd"]
Oct 30 18:53:49.105760 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:49.105715    4525 generic.go:356] "Generic (PLEG): container finished" podID="9edb2112-f66a-467d-8384-7343736ae49b" containerID="026934c3e4e871026d479d332ab691137acc2366bc2df0567ca437b92ba2d49e" exitCode=0
Oct 30 18:53:49.106268 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:49.105993    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-87293f-rz82z" event={"ID":"9edb2112-f66a-467d-8384-7343736ae49b","Type":"ContainerDied","Data":"026934c3e4e871026d479d332ab691137acc2366bc2df0567ca437b92ba2d49e"}
Oct 30 18:53:49.113862 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:49.113827    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f37d42-5dwmd" event={"ID":"39ac8767-10a6-449f-884f-7d3b716657a2","Type":"ContainerStarted","Data":"843140a915679eeb2ac81458f08338a7e1fc50511b672258b712eaf66142f707"}
Oct 30 18:53:50.124252 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:50.124218    4525 generic.go:356] "Generic (PLEG): container finished" podID="39ac8767-10a6-449f-884f-7d3b716657a2" containerID="89e3979ab1d020d4501e7cd380793ecb40bf4f31ca726a95aa4e5abebf0ae4a9" exitCode=0
Oct 30 18:53:50.124758 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:50.124290    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f37d42-5dwmd" event={"ID":"39ac8767-10a6-449f-884f-7d3b716657a2","Type":"ContainerDied","Data":"89e3979ab1d020d4501e7cd380793ecb40bf4f31ca726a95aa4e5abebf0ae4a9"}
Oct 30 18:53:50.295351 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:50.295322    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-87293f-rz82z"
Oct 30 18:53:50.405036 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:50.404951    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9edb2112-f66a-467d-8384-7343736ae49b-kube-api-access-r6ctt\" (UniqueName: \"kubernetes.io/projected/9edb2112-f66a-467d-8384-7343736ae49b-kube-api-access-r6ctt\") pod \"9edb2112-f66a-467d-8384-7343736ae49b\" (UID: \"9edb2112-f66a-467d-8384-7343736ae49b\") "
Oct 30 18:53:50.405214 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:50.405055    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9edb2112-f66a-467d-8384-7343736ae49b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9edb2112-f66a-467d-8384-7343736ae49b-maas-api-openshift-ca\") pod \"9edb2112-f66a-467d-8384-7343736ae49b\" (UID: \"9edb2112-f66a-467d-8384-7343736ae49b\") "
Oct 30 18:53:50.405214 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:50.405126    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9edb2112-f66a-467d-8384-7343736ae49b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9edb2112-f66a-467d-8384-7343736ae49b-maas-api-xks-ca\") pod \"9edb2112-f66a-467d-8384-7343736ae49b\" (UID: \"9edb2112-f66a-467d-8384-7343736ae49b\") "
Oct 30 18:53:50.405460 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:50.405435    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9edb2112-f66a-467d-8384-7343736ae49b-maas-api-openshift-ca" pod "9edb2112-f66a-467d-8384-7343736ae49b" (UID: "9edb2112-f66a-467d-8384-7343736ae49b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:53:50.405728 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:50.405711    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9edb2112-f66a-467d-8384-7343736ae49b-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:50.407342 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:50.407311    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9edb2112-f66a-467d-8384-7343736ae49b-kube-api-access-r6ctt" pod "9edb2112-f66a-467d-8384-7343736ae49b" (UID: "9edb2112-f66a-467d-8384-7343736ae49b"). InnerVolumeSpecName "kube-api-access-r6ctt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:53:50.407502 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:50.407394    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9edb2112-f66a-467d-8384-7343736ae49b-maas-api-xks-ca" pod "9edb2112-f66a-467d-8384-7343736ae49b" (UID: "9edb2112-f66a-467d-8384-7343736ae49b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:53:50.508762 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:50.507265    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9edb2112-f66a-467d-8384-7343736ae49b-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:50.508762 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:50.507304    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6ctt\" (UniqueName: \"kubernetes.io/projected/9edb2112-f66a-467d-8384-7343736ae49b-kube-api-access-r6ctt\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:51.134371 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:51.134343    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-87293f-rz82z"
Oct 30 18:53:51.134779 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:51.134338    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-87293f-rz82z" event={"ID":"9edb2112-f66a-467d-8384-7343736ae49b","Type":"ContainerDied","Data":"046433ac2713d2d7b7267029b59871707b151e2e73a91215e4ec5668c6d1c27e"}
Oct 30 18:53:51.134779 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:51.134436    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="046433ac2713d2d7b7267029b59871707b151e2e73a91215e4ec5668c6d1c27e"
Oct 30 18:53:51.300958 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:51.300931    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f37d42-5dwmd"
Oct 30 18:53:51.421869 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:51.420902    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39ac8767-10a6-449f-884f-7d3b716657a2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39ac8767-10a6-449f-884f-7d3b716657a2-maas-api-xks-ca\") pod \"39ac8767-10a6-449f-884f-7d3b716657a2\" (UID: \"39ac8767-10a6-449f-884f-7d3b716657a2\") "
Oct 30 18:53:51.421869 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:51.420969    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39ac8767-10a6-449f-884f-7d3b716657a2-kube-api-access-xt7zw\" (UniqueName: \"kubernetes.io/projected/39ac8767-10a6-449f-884f-7d3b716657a2-kube-api-access-xt7zw\") pod \"39ac8767-10a6-449f-884f-7d3b716657a2\" (UID: \"39ac8767-10a6-449f-884f-7d3b716657a2\") "
Oct 30 18:53:51.421869 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:51.421015    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39ac8767-10a6-449f-884f-7d3b716657a2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39ac8767-10a6-449f-884f-7d3b716657a2-maas-api-openshift-ca\") pod \"39ac8767-10a6-449f-884f-7d3b716657a2\" (UID: \"39ac8767-10a6-449f-884f-7d3b716657a2\") "
Oct 30 18:53:51.421869 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:51.421750    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39ac8767-10a6-449f-884f-7d3b716657a2-maas-api-openshift-ca" pod "39ac8767-10a6-449f-884f-7d3b716657a2" (UID: "39ac8767-10a6-449f-884f-7d3b716657a2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:53:51.428233 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:51.428188    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39ac8767-10a6-449f-884f-7d3b716657a2-maas-api-xks-ca" pod "39ac8767-10a6-449f-884f-7d3b716657a2" (UID: "39ac8767-10a6-449f-884f-7d3b716657a2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:53:51.429917 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:51.429896    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39ac8767-10a6-449f-884f-7d3b716657a2-kube-api-access-xt7zw" pod "39ac8767-10a6-449f-884f-7d3b716657a2" (UID: "39ac8767-10a6-449f-884f-7d3b716657a2"). InnerVolumeSpecName "kube-api-access-xt7zw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:53:51.528730 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:51.523017    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39ac8767-10a6-449f-884f-7d3b716657a2-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:51.528730 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:51.523051    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xt7zw\" (UniqueName: \"kubernetes.io/projected/39ac8767-10a6-449f-884f-7d3b716657a2-kube-api-access-xt7zw\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:51.528730 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:51.523065    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39ac8767-10a6-449f-884f-7d3b716657a2-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:52.274611 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:52.274576    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f37d42-5dwmd" event={"ID":"39ac8767-10a6-449f-884f-7d3b716657a2","Type":"ContainerDied","Data":"843140a915679eeb2ac81458f08338a7e1fc50511b672258b712eaf66142f707"}
Oct 30 18:53:52.275208 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:52.275188    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="843140a915679eeb2ac81458f08338a7e1fc50511b672258b712eaf66142f707"
Oct 30 18:53:52.275373 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:53:52.275361    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f37d42-5dwmd"
Oct 30 18:54:03.518808 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:03.518707    4525 generic.go:356] "Generic (PLEG): container finished" podID="97979488-7a76-4f4e-b330-49432c98bbec" containerID="c864ba5c8cf4b56772ab67170cbdd9ed6825d5373cae448336a7a11ba15b8609" exitCode=0
Oct 30 18:54:03.519335 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:03.518843    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7784e5-openshift-default-564776f9f-zm98k" event={"ID":"97979488-7a76-4f4e-b330-49432c98bbec","Type":"ContainerDied","Data":"c864ba5c8cf4b56772ab67170cbdd9ed6825d5373cae448336a7a11ba15b8609"}
Oct 30 18:54:03.519397 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:03.519374    4525 scope.go:122] "RemoveContainer" containerID="c864ba5c8cf4b56772ab67170cbdd9ed6825d5373cae448336a7a11ba15b8609"
Oct 30 18:54:03.520956 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:03.520935    4525 generic.go:356] "Generic (PLEG): container finished" podID="a9ff1283-3fc7-463f-b3b7-fafdd56f2349" containerID="5cfe5306c8da53186d023ff7953bf87c054d9bdea82fcdfad2919e3366f85faf" exitCode=0
Oct 30 18:54:03.521071 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:03.521009    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-066928-openshift-default-55ddf5c76f-pzlmw" event={"ID":"a9ff1283-3fc7-463f-b3b7-fafdd56f2349","Type":"ContainerDied","Data":"5cfe5306c8da53186d023ff7953bf87c054d9bdea82fcdfad2919e3366f85faf"}
Oct 30 18:54:03.521283 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:03.521252    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:54:03.521352 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:03.521339    4525 scope.go:122] "RemoveContainer" containerID="5cfe5306c8da53186d023ff7953bf87c054d9bdea82fcdfad2919e3366f85faf"
Oct 30 18:54:03.521439 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:03.521339    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:54:03.521487 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:03.521473    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:54:03.522801 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:03.522770    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:54:03.522919 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:03.522850    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:54:03.522919 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:03.522892    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:54:04.533985 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:04.533946    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-066928-openshift-default-55ddf5c76f-pzlmw" event={"ID":"a9ff1283-3fc7-463f-b3b7-fafdd56f2349","Type":"ContainerStarted","Data":"0913e11a983a273fe009cde085966709a530584998b47af0f93b9d0f277fd64a"}
Oct 30 18:54:04.537358 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:04.537324    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7784e5-openshift-default-564776f9f-zm98k" event={"ID":"97979488-7a76-4f4e-b330-49432c98bbec","Type":"ContainerStarted","Data":"d600190c7dd6e007e853cdd210e70a8a50c291e874175a4b78c9ee1fc46c1b63"}
Oct 30 18:54:04.760271 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:04.760236    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-066928-openshift-default-55ddf5c76f-pzlmw"
Oct 30 18:54:04.760443 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:04.760342    4525 patch_prober.go:29] interesting pod/e2e-api-b-066928-openshift-default-55ddf5c76f-pzlmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 30 18:54:04.760443 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:04.760376    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-066928-openshift-default-55ddf5c76f-pzlmw" podUID="a9ff1283-3fc7-463f-b3b7-fafdd56f2349" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 30 18:54:05.274354 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:05.274320    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-7784e5-openshift-default-564776f9f-zm98k"
Oct 30 18:54:05.274731 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:05.274573    4525 patch_prober.go:29] interesting pod/e2e-api-b-7784e5-openshift-default-564776f9f-zm98k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Oct 30 18:54:05.274731 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:05.274628    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7784e5-openshift-default-564776f9f-zm98k" podUID="97979488-7a76-4f4e-b330-49432c98bbec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Oct 30 18:54:05.760573 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:05.760122    4525 patch_prober.go:29] interesting pod/e2e-api-b-066928-openshift-default-55ddf5c76f-pzlmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 30 18:54:05.760573 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:05.760182    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-066928-openshift-default-55ddf5c76f-pzlmw" podUID="a9ff1283-3fc7-463f-b3b7-fafdd56f2349" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 30 18:54:06.294714 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:06.284325    4525 patch_prober.go:29] interesting pod/e2e-api-b-7784e5-openshift-default-564776f9f-zm98k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Oct 30 18:54:06.294714 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:06.284392    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7784e5-openshift-default-564776f9f-zm98k" podUID="97979488-7a76-4f4e-b330-49432c98bbec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Oct 30 18:54:06.760438 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:06.760383    4525 patch_prober.go:29] interesting pod/e2e-api-b-066928-openshift-default-55ddf5c76f-pzlmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 30 18:54:06.760597 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:06.760443    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-066928-openshift-default-55ddf5c76f-pzlmw" podUID="a9ff1283-3fc7-463f-b3b7-fafdd56f2349" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 30 18:54:07.274325 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:07.274292    4525 patch_prober.go:29] interesting pod/e2e-api-b-7784e5-openshift-default-564776f9f-zm98k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Oct 30 18:54:07.274486 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:07.274347    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7784e5-openshift-default-564776f9f-zm98k" podUID="97979488-7a76-4f4e-b330-49432c98bbec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Oct 30 18:54:07.762820 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:07.761901    4525 patch_prober.go:29] interesting pod/e2e-api-b-066928-openshift-default-55ddf5c76f-pzlmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 30 18:54:07.762820 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:07.761969    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-066928-openshift-default-55ddf5c76f-pzlmw" podUID="a9ff1283-3fc7-463f-b3b7-fafdd56f2349" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 30 18:54:08.274583 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:08.274548    4525 patch_prober.go:29] interesting pod/e2e-api-b-7784e5-openshift-default-564776f9f-zm98k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Oct 30 18:54:08.274791 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:08.274601    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7784e5-openshift-default-564776f9f-zm98k" podUID="97979488-7a76-4f4e-b330-49432c98bbec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Oct 30 18:54:08.763302 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:08.763266    4525 patch_prober.go:29] interesting pod/e2e-api-b-066928-openshift-default-55ddf5c76f-pzlmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 30 18:54:08.763909 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:08.763878    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-066928-openshift-default-55ddf5c76f-pzlmw" podUID="a9ff1283-3fc7-463f-b3b7-fafdd56f2349" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 30 18:54:09.278484 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:09.278441    4525 patch_prober.go:29] interesting pod/e2e-api-b-7784e5-openshift-default-564776f9f-zm98k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Oct 30 18:54:09.278891 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:09.278849    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7784e5-openshift-default-564776f9f-zm98k" podUID="97979488-7a76-4f4e-b330-49432c98bbec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Oct 30 18:54:09.467304 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:09.467259    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-066928-openshift-default-55ddf5c76f-pzlmw"
Oct 30 18:54:09.468401 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:09.468376    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-7784e5-openshift-default-564776f9f-zm98k"
Oct 30 18:54:09.773442 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:09.773402    4525 patch_prober.go:29] interesting pod/e2e-api-b-066928-openshift-default-55ddf5c76f-pzlmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 30 18:54:09.774503 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:09.774101    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-066928-openshift-default-55ddf5c76f-pzlmw" podUID="a9ff1283-3fc7-463f-b3b7-fafdd56f2349" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 30 18:54:09.993200 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:09.992861    4525 patch_prober.go:29] interesting pod/e2e-api-a-87293f-openshift-default-7c9c9cbc7f-vw6fm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 18:54:09.993200 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:09.992924    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-87293f-openshift-default-7c9c9cbc7f-vw6fm" podUID="cf796179-ce62-433b-87b8-a8bb1405c70b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:54:10.274856 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:10.274325    4525 patch_prober.go:29] interesting pod/e2e-api-b-7784e5-openshift-default-564776f9f-zm98k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Oct 30 18:54:10.274856 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:10.274392    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7784e5-openshift-default-564776f9f-zm98k" podUID="97979488-7a76-4f4e-b330-49432c98bbec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Oct 30 18:54:10.458939 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:10.458902    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-87293f-rz82z"]
Oct 30 18:54:10.459275 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:10.459254    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-87293f-rz82z"]
Oct 30 18:54:11.127193 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:11.127150    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-066928-openshift-default-55ddf5c76f-pzlmw"
Oct 30 18:54:11.131027 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:11.131002    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-066928-openshift-default-55ddf5c76f-pzlmw"
Oct 30 18:54:11.298063 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:11.298004    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-7784e5-openshift-default-564776f9f-zm98k"
Oct 30 18:54:11.319508 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:11.319437    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-7784e5-openshift-default-564776f9f-zm98k"
Oct 30 18:54:11.620745 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:11.618062    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9edb2112-f66a-467d-8384-7343736ae49b" path="/var/lib/kubelet/pods/9edb2112-f66a-467d-8384-7343736ae49b/volumes"
Oct 30 18:54:11.782764 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:11.778126    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f37d42-5dwmd"]
Oct 30 18:54:11.784901 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:11.784876    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f37d42-5dwmd"]
Oct 30 18:54:12.107794 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:12.107746    4525 generic.go:356] "Generic (PLEG): container finished" podID="cf796179-ce62-433b-87b8-a8bb1405c70b" containerID="67aa60478f70c3fa5eb0938ee01dd809a741e22324dd53c360be0b09d59cb8f8" exitCode=0
Oct 30 18:54:12.107982 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:12.107813    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-87293f-openshift-default-7c9c9cbc7f-vw6fm" event={"ID":"cf796179-ce62-433b-87b8-a8bb1405c70b","Type":"ContainerDied","Data":"67aa60478f70c3fa5eb0938ee01dd809a741e22324dd53c360be0b09d59cb8f8"}
Oct 30 18:54:12.108275 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:12.108249    4525 scope.go:122] "RemoveContainer" containerID="67aa60478f70c3fa5eb0938ee01dd809a741e22324dd53c360be0b09d59cb8f8"
Oct 30 18:54:12.110063 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:12.110028    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:54:12.110167 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:12.110139    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:54:12.110207 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:12.110193    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:54:13.143259 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:13.143217    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-87293f-openshift-default-7c9c9cbc7f-vw6fm" event={"ID":"cf796179-ce62-433b-87b8-a8bb1405c70b","Type":"ContainerStarted","Data":"8e6afb6c52487e5111c82f024b3593dd0afabedf8d4610421739b081780ac476"}
Oct 30 18:54:13.197710 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:13.192854    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-87293f-openshift-default-7c9c9cbc7f-vw6fm"
Oct 30 18:54:13.197710 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:13.192993    4525 patch_prober.go:29] interesting pod/e2e-api-a-87293f-openshift-default-7c9c9cbc7f-vw6fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 30 18:54:13.197710 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:13.193037    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-87293f-openshift-default-7c9c9cbc7f-vw6fm" podUID="cf796179-ce62-433b-87b8-a8bb1405c70b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 30 18:54:13.545377 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:13.545303    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39ac8767-10a6-449f-884f-7d3b716657a2" path="/var/lib/kubelet/pods/39ac8767-10a6-449f-884f-7d3b716657a2/volumes"
Oct 30 18:54:14.192078 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:14.192042    4525 patch_prober.go:29] interesting pod/e2e-api-a-87293f-openshift-default-7c9c9cbc7f-vw6fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 30 18:54:14.192738 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:14.192703    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-87293f-openshift-default-7c9c9cbc7f-vw6fm" podUID="cf796179-ce62-433b-87b8-a8bb1405c70b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 30 18:54:15.184856 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:15.184808    4525 patch_prober.go:29] interesting pod/e2e-api-a-87293f-openshift-default-7c9c9cbc7f-vw6fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 30 18:54:15.185161 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:15.185130    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-87293f-openshift-default-7c9c9cbc7f-vw6fm" podUID="cf796179-ce62-433b-87b8-a8bb1405c70b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 30 18:54:16.196705 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:16.190876    4525 patch_prober.go:29] interesting pod/e2e-api-a-87293f-openshift-default-7c9c9cbc7f-vw6fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 30 18:54:16.196705 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:16.190949    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-87293f-openshift-default-7c9c9cbc7f-vw6fm" podUID="cf796179-ce62-433b-87b8-a8bb1405c70b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 30 18:54:17.193446 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:17.192993    4525 patch_prober.go:29] interesting pod/e2e-api-a-87293f-openshift-default-7c9c9cbc7f-vw6fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 30 18:54:17.193446 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:17.193057    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-87293f-openshift-default-7c9c9cbc7f-vw6fm" podUID="cf796179-ce62-433b-87b8-a8bb1405c70b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 30 18:54:17.418535 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:54:17.418496    4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d49l" containerName="extractor"
Oct 30 18:54:17.818526 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:17.818489    4525 scope.go:122] "RemoveContainer" containerID="cc675924e87bef2fafee0a17ec9becf29ee2222b946d72ac6784116c1224accd"
Oct 30 18:54:17.821924 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:17.821223    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70bb645b-wfhgh"]
Oct 30 18:54:17.826234 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:17.822800    4525 cpu_manager.go:414] "containerMap: removing container" podUID="9edb2112-f66a-467d-8384-7343736ae49b" containerName="revoke-keys"
Oct 30 18:54:17.826234 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:17.822826    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9edb2112-f66a-467d-8384-7343736ae49b" containerName="revoke-keys"
Oct 30 18:54:17.826234 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:17.822883    4525 cpu_manager.go:414] "containerMap: removing container" podUID="39ac8767-10a6-449f-884f-7d3b716657a2" containerName="revoke-keys"
Oct 30 18:54:17.826234 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:17.822892    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39ac8767-10a6-449f-884f-7d3b716657a2" containerName="revoke-keys"
Oct 30 18:54:17.826234 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:17.823042    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="9edb2112-f66a-467d-8384-7343736ae49b" containerName="revoke-keys"
Oct 30 18:54:17.826234 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:17.823061    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="39ac8767-10a6-449f-884f-7d3b716657a2" containerName="revoke-keys"
Oct 30 18:54:17.828547 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:17.828521    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70bb645b-wfhgh"
Oct 30 18:54:17.832846 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:17.830386    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70bb645b-wfhgh"]
Oct 30 18:54:17.832846 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:17.832599    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:54:17.832846 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:17.832739    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94hcn\""
Oct 30 18:54:17.844607 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:17.838534    4525 scope.go:122] "RemoveContainer" containerID="d72b48501abc705a2b29750c7a1303f8088da99fc26b73504e2ed20d0613b25e"
Oct 30 18:54:17.862997 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:17.862973    4525 scope.go:122] "RemoveContainer" containerID="2c04f801a091f0141881686961cda69199a8ea9a01154e89c4a194e15920a0a9"
Oct 30 18:54:17.874807 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:17.874785    4525 scope.go:122] "RemoveContainer" containerID="d921bb8aaa58a2522595dffdcf4092ccab2f6a300401eb6d0e698104683da4c3"
Oct 30 18:54:17.929791 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:17.929489    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c983c51-cb90-47f2-8bc2-e2b7806df111-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70bb645b-wfhgh\" (UID: \"0c983c51-cb90-47f2-8bc2-e2b7806df111\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70bb645b-wfhgh"
Oct 30 18:54:17.929791 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:17.929548    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9bfl\" (UniqueName: \"kubernetes.io/projected/0c983c51-cb90-47f2-8bc2-e2b7806df111-kube-api-access-x9bfl\") pod \"maas-api-revoke-sub-70bb645b-wfhgh\" (UID: \"0c983c51-cb90-47f2-8bc2-e2b7806df111\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70bb645b-wfhgh"
Oct 30 18:54:17.930395 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:17.930234    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c983c51-cb90-47f2-8bc2-e2b7806df111-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70bb645b-wfhgh\" (UID: \"0c983c51-cb90-47f2-8bc2-e2b7806df111\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70bb645b-wfhgh"
Oct 30 18:54:18.031571 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:18.031537    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c983c51-cb90-47f2-8bc2-e2b7806df111-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70bb645b-wfhgh\" (UID: \"0c983c51-cb90-47f2-8bc2-e2b7806df111\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70bb645b-wfhgh"
Oct 30 18:54:18.031669 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:18.031653    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c983c51-cb90-47f2-8bc2-e2b7806df111-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70bb645b-wfhgh\" (UID: \"0c983c51-cb90-47f2-8bc2-e2b7806df111\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70bb645b-wfhgh"
Oct 30 18:54:18.031749 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:18.031720    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9bfl\" (UniqueName: \"kubernetes.io/projected/0c983c51-cb90-47f2-8bc2-e2b7806df111-kube-api-access-x9bfl\") pod \"maas-api-revoke-sub-70bb645b-wfhgh\" (UID: \"0c983c51-cb90-47f2-8bc2-e2b7806df111\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70bb645b-wfhgh"
Oct 30 18:54:18.032365 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:18.032333    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c983c51-cb90-47f2-8bc2-e2b7806df111-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70bb645b-wfhgh\" (UID: \"0c983c51-cb90-47f2-8bc2-e2b7806df111\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70bb645b-wfhgh"
Oct 30 18:54:18.035197 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:18.035174    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c983c51-cb90-47f2-8bc2-e2b7806df111-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70bb645b-wfhgh\" (UID: \"0c983c51-cb90-47f2-8bc2-e2b7806df111\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70bb645b-wfhgh"
Oct 30 18:54:18.039870 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:18.039851    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9bfl\" (UniqueName: \"kubernetes.io/projected/0c983c51-cb90-47f2-8bc2-e2b7806df111-kube-api-access-x9bfl\") pod \"maas-api-revoke-sub-70bb645b-wfhgh\" (UID: \"0c983c51-cb90-47f2-8bc2-e2b7806df111\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70bb645b-wfhgh"
Oct 30 18:54:18.129728 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:18.124095    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-87293f-openshift-default-7c9c9cbc7f-vw6fm"]
Oct 30 18:54:18.129728 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:18.124786    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-87293f-openshift-default-7c9c9cbc7f-vw6fm" podUID="cf796179-ce62-433b-87b8-a8bb1405c70b" containerName="istio-proxy" containerID="cri-o://8e6afb6c52487e5111c82f024b3593dd0afabedf8d4610421739b081780ac476" gracePeriod=30
Oct 30 18:54:18.172536 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:18.172286    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70bb645b-wfhgh"
Oct 30 18:54:18.725660 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:18.725509    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7784e5-wl98w"]
Oct 30 18:54:18.732422 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:18.732395    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7784e5-wl98w"]
Oct 30 18:54:18.732583 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:18.732502    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7784e5-wl98w"
Oct 30 18:54:18.761717 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:54:18.761665    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c983c51_cb90_47f2_8bc2_e2b7806df111.slice/crio-94f2cd08c5fdb5ef79a8ba5bdf71965c72f2d72e41e11b71c159903835bc7ec2 WatchSource:0}: Error finding container 94f2cd08c5fdb5ef79a8ba5bdf71965c72f2d72e41e11b71c159903835bc7ec2: Status 404 returned error can't find the container with id 94f2cd08c5fdb5ef79a8ba5bdf71965c72f2d72e41e11b71c159903835bc7ec2
Oct 30 18:54:18.775090 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:18.775055    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70bb645b-wfhgh"]
Oct 30 18:54:18.800537 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:18.800368    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jdfr\" (UniqueName: \"kubernetes.io/projected/6fdcc41d-76be-4f42-9f6e-af06f8169b50-kube-api-access-4jdfr\") pod \"maas-api-revoke-keys-e2e-api-b-7784e5-wl98w\" (UID: \"6fdcc41d-76be-4f42-9f6e-af06f8169b50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7784e5-wl98w"
Oct 30 18:54:18.800761 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:18.800743    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fdcc41d-76be-4f42-9f6e-af06f8169b50-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7784e5-wl98w\" (UID: \"6fdcc41d-76be-4f42-9f6e-af06f8169b50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7784e5-wl98w"
Oct 30 18:54:18.800871 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:18.800860    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fdcc41d-76be-4f42-9f6e-af06f8169b50-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7784e5-wl98w\" (UID: \"6fdcc41d-76be-4f42-9f6e-af06f8169b50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7784e5-wl98w"
Oct 30 18:54:18.902014 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:18.901970    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jdfr\" (UniqueName: \"kubernetes.io/projected/6fdcc41d-76be-4f42-9f6e-af06f8169b50-kube-api-access-4jdfr\") pod \"maas-api-revoke-keys-e2e-api-b-7784e5-wl98w\" (UID: \"6fdcc41d-76be-4f42-9f6e-af06f8169b50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7784e5-wl98w"
Oct 30 18:54:18.902205 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:18.902041    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fdcc41d-76be-4f42-9f6e-af06f8169b50-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7784e5-wl98w\" (UID: \"6fdcc41d-76be-4f42-9f6e-af06f8169b50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7784e5-wl98w"
Oct 30 18:54:18.902205 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:18.902069    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fdcc41d-76be-4f42-9f6e-af06f8169b50-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7784e5-wl98w\" (UID: \"6fdcc41d-76be-4f42-9f6e-af06f8169b50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7784e5-wl98w"
Oct 30 18:54:18.903213 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:18.903185    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fdcc41d-76be-4f42-9f6e-af06f8169b50-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7784e5-wl98w\" (UID: \"6fdcc41d-76be-4f42-9f6e-af06f8169b50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7784e5-wl98w"
Oct 30 18:54:18.977128 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:18.977018    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fdcc41d-76be-4f42-9f6e-af06f8169b50-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7784e5-wl98w\" (UID: \"6fdcc41d-76be-4f42-9f6e-af06f8169b50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7784e5-wl98w"
Oct 30 18:54:18.977296 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:18.977234    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jdfr\" (UniqueName: \"kubernetes.io/projected/6fdcc41d-76be-4f42-9f6e-af06f8169b50-kube-api-access-4jdfr\") pod \"maas-api-revoke-keys-e2e-api-b-7784e5-wl98w\" (UID: \"6fdcc41d-76be-4f42-9f6e-af06f8169b50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7784e5-wl98w"
Oct 30 18:54:19.057748 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:19.052952    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7784e5-wl98w"
Oct 30 18:54:19.388798 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:54:19.388768    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fdcc41d_76be_4f42_9f6e_af06f8169b50.slice/crio-47a363594d09791b74ce6508c91b64b4213fbb03506be6ee388ea3fbee207dc3 WatchSource:0}: Error finding container 47a363594d09791b74ce6508c91b64b4213fbb03506be6ee388ea3fbee207dc3: Status 404 returned error can't find the container with id 47a363594d09791b74ce6508c91b64b4213fbb03506be6ee388ea3fbee207dc3
Oct 30 18:54:19.391522 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:19.391493    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7784e5-wl98w"]
Oct 30 18:54:19.408152 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:19.408111    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7784e5-wl98w" event={"ID":"6fdcc41d-76be-4f42-9f6e-af06f8169b50","Type":"ContainerStarted","Data":"47a363594d09791b74ce6508c91b64b4213fbb03506be6ee388ea3fbee207dc3"}
Oct 30 18:54:19.410265 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:19.410230    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70bb645b-wfhgh" event={"ID":"0c983c51-cb90-47f2-8bc2-e2b7806df111","Type":"ContainerStarted","Data":"cb0d6a8227a3dcc2de5fbfd178e3b0591867b38ac8b591e2e26fe99c27011b24"}
Oct 30 18:54:19.410380 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:19.410291    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70bb645b-wfhgh" event={"ID":"0c983c51-cb90-47f2-8bc2-e2b7806df111","Type":"ContainerStarted","Data":"94f2cd08c5fdb5ef79a8ba5bdf71965c72f2d72e41e11b71c159903835bc7ec2"}
Oct 30 18:54:19.465227 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:19.465147    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70bb645b-wfhgh" podStartSLOduration=2.465123466 podStartE2EDuration="2.465123466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:54:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:54:19.457533897 +0000 UTC m=+1082.083203868" watchObservedRunningTime="2026-10-30 18:54:19.465123466 +0000 UTC m=+1082.090793430"
Oct 30 18:54:19.730494 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:19.730452    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-22qrl"]
Oct 30 18:54:19.731051 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:19.730782    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-22qrl" podUID="13c9b518-512f-4a88-ae9e-cd8e9fb247cb" containerName="istio-proxy" containerID="cri-o://b95c5130210fa52372bd5ff3ef7c5602ec5026b642a357cc40f3aeb88fff06be" gracePeriod=30
Oct 30 18:54:20.264569 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:20.264538    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-066928-8hzcc"]
Oct 30 18:54:20.294564 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:20.294528    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-066928-8hzcc"
Oct 30 18:54:20.294761 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:20.294531    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-066928-8hzcc"]
Oct 30 18:54:20.345724 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:20.345396    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eeccf097-40f0-440a-a90e-3b97186ef05f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-066928-8hzcc\" (UID: \"eeccf097-40f0-440a-a90e-3b97186ef05f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-066928-8hzcc"
Oct 30 18:54:20.345724 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:20.345480    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eeccf097-40f0-440a-a90e-3b97186ef05f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-066928-8hzcc\" (UID: \"eeccf097-40f0-440a-a90e-3b97186ef05f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-066928-8hzcc"
Oct 30 18:54:20.345724 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:20.345553    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5hc9\" (UniqueName: \"kubernetes.io/projected/eeccf097-40f0-440a-a90e-3b97186ef05f-kube-api-access-j5hc9\") pod \"maas-api-revoke-keys-e2e-api-b-066928-8hzcc\" (UID: \"eeccf097-40f0-440a-a90e-3b97186ef05f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-066928-8hzcc"
Oct 30 18:54:20.422257 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:20.421881    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7784e5-wl98w" event={"ID":"6fdcc41d-76be-4f42-9f6e-af06f8169b50","Type":"ContainerStarted","Data":"ce496fee7c64949d9029dae1b4578aeb0dabc73e935b99346ba6d1f8b8ec16e5"}
Oct 30 18:54:20.426202 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:20.425112    4525 generic.go:356] "Generic (PLEG): container finished" podID="0c983c51-cb90-47f2-8bc2-e2b7806df111" containerID="cb0d6a8227a3dcc2de5fbfd178e3b0591867b38ac8b591e2e26fe99c27011b24" exitCode=0
Oct 30 18:54:20.426202 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:20.425235    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70bb645b-wfhgh" event={"ID":"0c983c51-cb90-47f2-8bc2-e2b7806df111","Type":"ContainerDied","Data":"cb0d6a8227a3dcc2de5fbfd178e3b0591867b38ac8b591e2e26fe99c27011b24"}
Oct 30 18:54:20.441403 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:20.441347    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7784e5-wl98w" podStartSLOduration=2.441331125 podStartE2EDuration="2.441331125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:54:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:54:20.438012678 +0000 UTC m=+1083.063682645" watchObservedRunningTime="2026-10-30 18:54:20.441331125 +0000 UTC m=+1083.067001091"
Oct 30 18:54:20.447013 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:20.446986    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5hc9\" (UniqueName: \"kubernetes.io/projected/eeccf097-40f0-440a-a90e-3b97186ef05f-kube-api-access-j5hc9\") pod \"maas-api-revoke-keys-e2e-api-b-066928-8hzcc\" (UID: \"eeccf097-40f0-440a-a90e-3b97186ef05f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-066928-8hzcc"
Oct 30 18:54:20.447218 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:20.447202    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eeccf097-40f0-440a-a90e-3b97186ef05f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-066928-8hzcc\" (UID: \"eeccf097-40f0-440a-a90e-3b97186ef05f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-066928-8hzcc"
Oct 30 18:54:20.447283 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:20.447251    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eeccf097-40f0-440a-a90e-3b97186ef05f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-066928-8hzcc\" (UID: \"eeccf097-40f0-440a-a90e-3b97186ef05f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-066928-8hzcc"
Oct 30 18:54:20.447909 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:20.447846    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eeccf097-40f0-440a-a90e-3b97186ef05f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-066928-8hzcc\" (UID: \"eeccf097-40f0-440a-a90e-3b97186ef05f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-066928-8hzcc"
Oct 30 18:54:20.449647 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:20.449621    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eeccf097-40f0-440a-a90e-3b97186ef05f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-066928-8hzcc\" (UID: \"eeccf097-40f0-440a-a90e-3b97186ef05f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-066928-8hzcc"
Oct 30 18:54:20.458043 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:20.458016    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5hc9\" (UniqueName: \"kubernetes.io/projected/eeccf097-40f0-440a-a90e-3b97186ef05f-kube-api-access-j5hc9\") pod \"maas-api-revoke-keys-e2e-api-b-066928-8hzcc\" (UID: \"eeccf097-40f0-440a-a90e-3b97186ef05f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-066928-8hzcc"
Oct 30 18:54:20.629925 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:20.629889    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-066928-8hzcc"
Oct 30 18:54:20.915401 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:54:20.915368    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeccf097_40f0_440a_a90e_3b97186ef05f.slice/crio-493e33bcfafbe20265762a51bec93222a6b9333d3f8187f2f0fce6d730dc1c32 WatchSource:0}: Error finding container 493e33bcfafbe20265762a51bec93222a6b9333d3f8187f2f0fce6d730dc1c32: Status 404 returned error can't find the container with id 493e33bcfafbe20265762a51bec93222a6b9333d3f8187f2f0fce6d730dc1c32
Oct 30 18:54:20.916736 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:20.916707    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-066928-8hzcc"]
Oct 30 18:54:21.436669 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:21.436561    4525 generic.go:356] "Generic (PLEG): container finished" podID="6fdcc41d-76be-4f42-9f6e-af06f8169b50" containerID="ce496fee7c64949d9029dae1b4578aeb0dabc73e935b99346ba6d1f8b8ec16e5" exitCode=0
Oct 30 18:54:21.452234 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:21.437288    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7784e5-wl98w" event={"ID":"6fdcc41d-76be-4f42-9f6e-af06f8169b50","Type":"ContainerDied","Data":"ce496fee7c64949d9029dae1b4578aeb0dabc73e935b99346ba6d1f8b8ec16e5"}
Oct 30 18:54:21.452234 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:21.439346    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-066928-8hzcc" event={"ID":"eeccf097-40f0-440a-a90e-3b97186ef05f","Type":"ContainerStarted","Data":"a2f75a78c298b0d507653d2da7cc549b9dd7c8b53e00df6576a6926cf4d8d92e"}
Oct 30 18:54:21.452234 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:21.439400    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-066928-8hzcc" event={"ID":"eeccf097-40f0-440a-a90e-3b97186ef05f","Type":"ContainerStarted","Data":"493e33bcfafbe20265762a51bec93222a6b9333d3f8187f2f0fce6d730dc1c32"}
Oct 30 18:54:21.483240 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:21.483186    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-066928-8hzcc" podStartSLOduration=1.483172875 podStartE2EDuration="1.483172875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:54:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:54:21.480940593 +0000 UTC m=+1084.106610557" watchObservedRunningTime="2026-10-30 18:54:21.483172875 +0000 UTC m=+1084.108842839"
Oct 30 18:54:21.686954 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:21.681878    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70bb645b-wfhgh"
Oct 30 18:54:21.774101 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:21.771556    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c983c51-cb90-47f2-8bc2-e2b7806df111-kube-api-access-x9bfl\" (UniqueName: \"kubernetes.io/projected/0c983c51-cb90-47f2-8bc2-e2b7806df111-kube-api-access-x9bfl\") pod \"0c983c51-cb90-47f2-8bc2-e2b7806df111\" (UID: \"0c983c51-cb90-47f2-8bc2-e2b7806df111\") "
Oct 30 18:54:21.774101 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:21.771834    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c983c51-cb90-47f2-8bc2-e2b7806df111-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c983c51-cb90-47f2-8bc2-e2b7806df111-maas-api-xks-ca\") pod \"0c983c51-cb90-47f2-8bc2-e2b7806df111\" (UID: \"0c983c51-cb90-47f2-8bc2-e2b7806df111\") "
Oct 30 18:54:21.774101 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:21.771923    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c983c51-cb90-47f2-8bc2-e2b7806df111-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c983c51-cb90-47f2-8bc2-e2b7806df111-maas-api-openshift-ca\") pod \"0c983c51-cb90-47f2-8bc2-e2b7806df111\" (UID: \"0c983c51-cb90-47f2-8bc2-e2b7806df111\") "
Oct 30 18:54:21.774101 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:21.772967    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c983c51-cb90-47f2-8bc2-e2b7806df111-maas-api-openshift-ca" pod "0c983c51-cb90-47f2-8bc2-e2b7806df111" (UID: "0c983c51-cb90-47f2-8bc2-e2b7806df111"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:54:21.774612 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:21.774583    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c983c51-cb90-47f2-8bc2-e2b7806df111-kube-api-access-x9bfl" pod "0c983c51-cb90-47f2-8bc2-e2b7806df111" (UID: "0c983c51-cb90-47f2-8bc2-e2b7806df111"). InnerVolumeSpecName "kube-api-access-x9bfl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:54:21.777060 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:21.777025    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c983c51-cb90-47f2-8bc2-e2b7806df111-maas-api-xks-ca" pod "0c983c51-cb90-47f2-8bc2-e2b7806df111" (UID: "0c983c51-cb90-47f2-8bc2-e2b7806df111"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:54:21.874374 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:21.874335    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c983c51-cb90-47f2-8bc2-e2b7806df111-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:21.874374 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:21.874368    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c983c51-cb90-47f2-8bc2-e2b7806df111-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:21.874374 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:21.874379    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9bfl\" (UniqueName: \"kubernetes.io/projected/0c983c51-cb90-47f2-8bc2-e2b7806df111-kube-api-access-x9bfl\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:22.452175 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:22.452144    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70bb645b-wfhgh"
Oct 30 18:54:22.453212 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:22.452164    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70bb645b-wfhgh" event={"ID":"0c983c51-cb90-47f2-8bc2-e2b7806df111","Type":"ContainerDied","Data":"94f2cd08c5fdb5ef79a8ba5bdf71965c72f2d72e41e11b71c159903835bc7ec2"}
Oct 30 18:54:22.453417 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:22.453398    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94f2cd08c5fdb5ef79a8ba5bdf71965c72f2d72e41e11b71c159903835bc7ec2"
Oct 30 18:54:22.455505 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:22.455479    4525 generic.go:356] "Generic (PLEG): container finished" podID="81739e60-d074-4a42-ade3-a0f68ea9329d" containerID="3eb61f7ea9b79c37b1f93a3a5f2a93a34be89622062ad9a76e12470e5e20c9a3" exitCode=0
Oct 30 18:54:22.455614 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:22.455524    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-da58b769-openshift-default-69887ddf8f-n79ls" event={"ID":"81739e60-d074-4a42-ade3-a0f68ea9329d","Type":"ContainerDied","Data":"3eb61f7ea9b79c37b1f93a3a5f2a93a34be89622062ad9a76e12470e5e20c9a3"}
Oct 30 18:54:22.455974 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:22.455958    4525 scope.go:122] "RemoveContainer" containerID="3eb61f7ea9b79c37b1f93a3a5f2a93a34be89622062ad9a76e12470e5e20c9a3"
Oct 30 18:54:22.457581 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:22.457555    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:54:22.457692 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:22.457580    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-066928-8hzcc" event={"ID":"eeccf097-40f0-440a-a90e-3b97186ef05f","Type":"ContainerDied","Data":"a2f75a78c298b0d507653d2da7cc549b9dd7c8b53e00df6576a6926cf4d8d92e"}
Oct 30 18:54:22.457692 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:22.457628    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:54:22.457692 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:22.457555    4525 generic.go:356] "Generic (PLEG): container finished" podID="eeccf097-40f0-440a-a90e-3b97186ef05f" containerID="a2f75a78c298b0d507653d2da7cc549b9dd7c8b53e00df6576a6926cf4d8d92e" exitCode=0
Oct 30 18:54:22.457692 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:22.457665    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:54:22.685626 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:22.685601    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7784e5-wl98w"
Oct 30 18:54:22.791828 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:22.791748    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fdcc41d-76be-4f42-9f6e-af06f8169b50-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fdcc41d-76be-4f42-9f6e-af06f8169b50-maas-api-openshift-ca\") pod \"6fdcc41d-76be-4f42-9f6e-af06f8169b50\" (UID: \"6fdcc41d-76be-4f42-9f6e-af06f8169b50\") "
Oct 30 18:54:22.791962 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:22.791921    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6fdcc41d-76be-4f42-9f6e-af06f8169b50-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fdcc41d-76be-4f42-9f6e-af06f8169b50-maas-api-xks-ca\") pod \"6fdcc41d-76be-4f42-9f6e-af06f8169b50\" (UID: \"6fdcc41d-76be-4f42-9f6e-af06f8169b50\") "
Oct 30 18:54:22.792038 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:22.792022    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fdcc41d-76be-4f42-9f6e-af06f8169b50-kube-api-access-4jdfr\" (UniqueName: \"kubernetes.io/projected/6fdcc41d-76be-4f42-9f6e-af06f8169b50-kube-api-access-4jdfr\") pod \"6fdcc41d-76be-4f42-9f6e-af06f8169b50\" (UID: \"6fdcc41d-76be-4f42-9f6e-af06f8169b50\") "
Oct 30 18:54:22.792186 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:22.792159    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fdcc41d-76be-4f42-9f6e-af06f8169b50-maas-api-openshift-ca" pod "6fdcc41d-76be-4f42-9f6e-af06f8169b50" (UID: "6fdcc41d-76be-4f42-9f6e-af06f8169b50"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:54:22.792606 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:22.792582    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fdcc41d-76be-4f42-9f6e-af06f8169b50-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:22.794217 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:22.794187    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fdcc41d-76be-4f42-9f6e-af06f8169b50-kube-api-access-4jdfr" pod "6fdcc41d-76be-4f42-9f6e-af06f8169b50" (UID: "6fdcc41d-76be-4f42-9f6e-af06f8169b50"). InnerVolumeSpecName "kube-api-access-4jdfr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:54:22.794318 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:22.794188    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fdcc41d-76be-4f42-9f6e-af06f8169b50-maas-api-xks-ca" pod "6fdcc41d-76be-4f42-9f6e-af06f8169b50" (UID: "6fdcc41d-76be-4f42-9f6e-af06f8169b50"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:54:22.893890 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:22.893857    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jdfr\" (UniqueName: \"kubernetes.io/projected/6fdcc41d-76be-4f42-9f6e-af06f8169b50-kube-api-access-4jdfr\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:22.893890 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:22.893891    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fdcc41d-76be-4f42-9f6e-af06f8169b50-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:23.467960 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:23.467929    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7784e5-wl98w"
Oct 30 18:54:23.468447 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:23.467953    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7784e5-wl98w" event={"ID":"6fdcc41d-76be-4f42-9f6e-af06f8169b50","Type":"ContainerDied","Data":"47a363594d09791b74ce6508c91b64b4213fbb03506be6ee388ea3fbee207dc3"}
Oct 30 18:54:23.468447 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:23.468008    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="47a363594d09791b74ce6508c91b64b4213fbb03506be6ee388ea3fbee207dc3"
Oct 30 18:54:23.470438 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:23.470415    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-da58b769-openshift-default-69887ddf8f-n79ls" event={"ID":"81739e60-d074-4a42-ade3-a0f68ea9329d","Type":"ContainerStarted","Data":"24cc207c0ffbce5490bd5d432ae75292eb84c13ddee7f6e809a74eb1175fc456"}
Oct 30 18:54:23.705742 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:23.705666    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-066928-8hzcc"
Oct 30 18:54:23.810242 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:23.810146    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eeccf097-40f0-440a-a90e-3b97186ef05f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eeccf097-40f0-440a-a90e-3b97186ef05f-maas-api-openshift-ca\") pod \"eeccf097-40f0-440a-a90e-3b97186ef05f\" (UID: \"eeccf097-40f0-440a-a90e-3b97186ef05f\") "
Oct 30 18:54:23.810426 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:23.810269    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eeccf097-40f0-440a-a90e-3b97186ef05f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eeccf097-40f0-440a-a90e-3b97186ef05f-maas-api-xks-ca\") pod \"eeccf097-40f0-440a-a90e-3b97186ef05f\" (UID: \"eeccf097-40f0-440a-a90e-3b97186ef05f\") "
Oct 30 18:54:23.810426 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:23.810398    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eeccf097-40f0-440a-a90e-3b97186ef05f-kube-api-access-j5hc9\" (UniqueName: \"kubernetes.io/projected/eeccf097-40f0-440a-a90e-3b97186ef05f-kube-api-access-j5hc9\") pod \"eeccf097-40f0-440a-a90e-3b97186ef05f\" (UID: \"eeccf097-40f0-440a-a90e-3b97186ef05f\") "
Oct 30 18:54:23.810541 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:23.810489    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eeccf097-40f0-440a-a90e-3b97186ef05f-maas-api-openshift-ca" pod "eeccf097-40f0-440a-a90e-3b97186ef05f" (UID: "eeccf097-40f0-440a-a90e-3b97186ef05f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:54:23.811111 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:23.811070    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eeccf097-40f0-440a-a90e-3b97186ef05f-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:23.813029 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:23.813002    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eeccf097-40f0-440a-a90e-3b97186ef05f-kube-api-access-j5hc9" pod "eeccf097-40f0-440a-a90e-3b97186ef05f" (UID: "eeccf097-40f0-440a-a90e-3b97186ef05f"). InnerVolumeSpecName "kube-api-access-j5hc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:54:23.813029 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:23.813012    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eeccf097-40f0-440a-a90e-3b97186ef05f-maas-api-xks-ca" pod "eeccf097-40f0-440a-a90e-3b97186ef05f" (UID: "eeccf097-40f0-440a-a90e-3b97186ef05f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:54:23.912219 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:23.912185    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eeccf097-40f0-440a-a90e-3b97186ef05f-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:23.912219 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:23.912217    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5hc9\" (UniqueName: \"kubernetes.io/projected/eeccf097-40f0-440a-a90e-3b97186ef05f-kube-api-access-j5hc9\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:23.951712 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:23.950169    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-87293f-openshift-default-7c9c9cbc7f-vw6fm"
Oct 30 18:54:24.160753 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:24.160725    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-da58b769-openshift-default-69887ddf8f-n79ls"
Oct 30 18:54:24.161893 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:24.161867    4525 patch_prober.go:29] interesting pod/e2e-mt-da58b769-openshift-default-69887ddf8f-n79ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Oct 30 18:54:24.162006 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:24.161915    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-da58b769-openshift-default-69887ddf8f-n79ls" podUID="81739e60-d074-4a42-ade3-a0f68ea9329d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Oct 30 18:54:24.427184 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:24.427158    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-87293f-openshift-default-7c9c9cbc7f-vw6fm"
Oct 30 18:54:24.512723 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:24.512657    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-066928-8hzcc"
Oct 30 18:54:24.512723 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:24.512698    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-066928-8hzcc" event={"ID":"eeccf097-40f0-440a-a90e-3b97186ef05f","Type":"ContainerDied","Data":"493e33bcfafbe20265762a51bec93222a6b9333d3f8187f2f0fce6d730dc1c32"}
Oct 30 18:54:24.513188 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:24.512761    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="493e33bcfafbe20265762a51bec93222a6b9333d3f8187f2f0fce6d730dc1c32"
Oct 30 18:54:24.515182 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:24.515156    4525 generic.go:356] "Generic (PLEG): container finished" podID="cf796179-ce62-433b-87b8-a8bb1405c70b" containerID="8e6afb6c52487e5111c82f024b3593dd0afabedf8d4610421739b081780ac476" exitCode=0
Oct 30 18:54:24.515809 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:24.515785    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-87293f-openshift-default-7c9c9cbc7f-vw6fm"
Oct 30 18:54:24.516042 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:24.515785    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-87293f-openshift-default-7c9c9cbc7f-vw6fm" event={"ID":"cf796179-ce62-433b-87b8-a8bb1405c70b","Type":"ContainerDied","Data":"8e6afb6c52487e5111c82f024b3593dd0afabedf8d4610421739b081780ac476"}
Oct 30 18:54:24.516140 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:24.516114    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-87293f-openshift-default-7c9c9cbc7f-vw6fm" event={"ID":"cf796179-ce62-433b-87b8-a8bb1405c70b","Type":"ContainerDied","Data":"4862e37873a2eb78da592424b01c21b7cb1d7536676de9faf0edbbe14445b5bc"}
Oct 30 18:54:24.516214 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:24.516151    4525 scope.go:122] "RemoveContainer" containerID="8e6afb6c52487e5111c82f024b3593dd0afabedf8d4610421739b081780ac476"
Oct 30 18:54:24.562132 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:24.550334    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf796179-ce62-433b-87b8-a8bb1405c70b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf796179-ce62-433b-87b8-a8bb1405c70b-istio-envoy\") pod \"cf796179-ce62-433b-87b8-a8bb1405c70b\" (UID: \"cf796179-ce62-433b-87b8-a8bb1405c70b\") "
Oct 30 18:54:24.562132 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:24.550410    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cf796179-ce62-433b-87b8-a8bb1405c70b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf796179-ce62-433b-87b8-a8bb1405c70b-istio-podinfo\") pod \"cf796179-ce62-433b-87b8-a8bb1405c70b\" (UID: \"cf796179-ce62-433b-87b8-a8bb1405c70b\") "
Oct 30 18:54:24.562132 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:24.550501    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf796179-ce62-433b-87b8-a8bb1405c70b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf796179-ce62-433b-87b8-a8bb1405c70b-workload-certs\") pod \"cf796179-ce62-433b-87b8-a8bb1405c70b\" (UID: \"cf796179-ce62-433b-87b8-a8bb1405c70b\") "
Oct 30 18:54:24.562132 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:24.550531    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf796179-ce62-433b-87b8-a8bb1405c70b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf796179-ce62-433b-87b8-a8bb1405c70b-credential-socket\") pod \"cf796179-ce62-433b-87b8-a8bb1405c70b\" (UID: \"cf796179-ce62-433b-87b8-a8bb1405c70b\") "
Oct 30 18:54:24.562132 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:24.550567    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf796179-ce62-433b-87b8-a8bb1405c70b-istio-token\" (UniqueName: \"kubernetes.io/projected/cf796179-ce62-433b-87b8-a8bb1405c70b-istio-token\") pod \"cf796179-ce62-433b-87b8-a8bb1405c70b\" (UID: \"cf796179-ce62-433b-87b8-a8bb1405c70b\") "
Oct 30 18:54:24.562132 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:24.550603    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf796179-ce62-433b-87b8-a8bb1405c70b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf796179-ce62-433b-87b8-a8bb1405c70b-istio-data\") pod \"cf796179-ce62-433b-87b8-a8bb1405c70b\" (UID: \"cf796179-ce62-433b-87b8-a8bb1405c70b\") "
Oct 30 18:54:24.562132 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:24.550722    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf796179-ce62-433b-87b8-a8bb1405c70b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf796179-ce62-433b-87b8-a8bb1405c70b-workload-socket\") pod \"cf796179-ce62-433b-87b8-a8bb1405c70b\" (UID: \"cf796179-ce62-433b-87b8-a8bb1405c70b\") "
Oct 30 18:54:24.562132 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:24.550766    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf796179-ce62-433b-87b8-a8bb1405c70b-kube-api-access-vtwft\" (UniqueName: \"kubernetes.io/projected/cf796179-ce62-433b-87b8-a8bb1405c70b-kube-api-access-vtwft\") pod \"cf796179-ce62-433b-87b8-a8bb1405c70b\" (UID: \"cf796179-ce62-433b-87b8-a8bb1405c70b\") "
Oct 30 18:54:24.562132 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:24.550907    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf796179-ce62-433b-87b8-a8bb1405c70b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf796179-ce62-433b-87b8-a8bb1405c70b-istiod-ca-cert\") pod \"cf796179-ce62-433b-87b8-a8bb1405c70b\" (UID: \"cf796179-ce62-433b-87b8-a8bb1405c70b\") "
Oct 30 18:54:24.562132 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:24.552801    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf796179-ce62-433b-87b8-a8bb1405c70b-workload-certs" pod "cf796179-ce62-433b-87b8-a8bb1405c70b" (UID: "cf796179-ce62-433b-87b8-a8bb1405c70b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:54:24.562132 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:24.553207    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf796179-ce62-433b-87b8-a8bb1405c70b-credential-socket" pod "cf796179-ce62-433b-87b8-a8bb1405c70b" (UID: "cf796179-ce62-433b-87b8-a8bb1405c70b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:54:24.562132 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:24.553530    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf796179-ce62-433b-87b8-a8bb1405c70b-istio-data" pod "cf796179-ce62-433b-87b8-a8bb1405c70b" (UID: "cf796179-ce62-433b-87b8-a8bb1405c70b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:54:24.562132 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:24.553965    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf796179-ce62-433b-87b8-a8bb1405c70b-workload-socket" pod "cf796179-ce62-433b-87b8-a8bb1405c70b" (UID: "cf796179-ce62-433b-87b8-a8bb1405c70b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:54:24.563631 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:24.563596    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf796179-ce62-433b-87b8-a8bb1405c70b-istiod-ca-cert" pod "cf796179-ce62-433b-87b8-a8bb1405c70b" (UID: "cf796179-ce62-433b-87b8-a8bb1405c70b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:54:24.592208 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:24.592126    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf796179-ce62-433b-87b8-a8bb1405c70b-istio-envoy" pod "cf796179-ce62-433b-87b8-a8bb1405c70b" (UID: "cf796179-ce62-433b-87b8-a8bb1405c70b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:54:24.592604 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:24.592521    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cf796179-ce62-433b-87b8-a8bb1405c70b-istio-podinfo" pod "cf796179-ce62-433b-87b8-a8bb1405c70b" (UID: "cf796179-ce62-433b-87b8-a8bb1405c70b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 18:54:24.592920 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:24.592877    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf796179-ce62-433b-87b8-a8bb1405c70b-istio-token" pod "cf796179-ce62-433b-87b8-a8bb1405c70b" (UID: "cf796179-ce62-433b-87b8-a8bb1405c70b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:54:24.598706 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:24.595311    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf796179-ce62-433b-87b8-a8bb1405c70b-kube-api-access-vtwft" pod "cf796179-ce62-433b-87b8-a8bb1405c70b" (UID: "cf796179-ce62-433b-87b8-a8bb1405c70b"). InnerVolumeSpecName "kube-api-access-vtwft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:54:24.619038 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:24.619004    4525 scope.go:122] "RemoveContainer" containerID="67aa60478f70c3fa5eb0938ee01dd809a741e22324dd53c360be0b09d59cb8f8"
Oct 30 18:54:24.630927 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:24.630906    4525 scope.go:122] "RemoveContainer" containerID="8e6afb6c52487e5111c82f024b3593dd0afabedf8d4610421739b081780ac476"
Oct 30 18:54:24.631245 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:54:24.631225    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e6afb6c52487e5111c82f024b3593dd0afabedf8d4610421739b081780ac476\": container with ID starting with 8e6afb6c52487e5111c82f024b3593dd0afabedf8d4610421739b081780ac476 not found: ID does not exist" containerID="8e6afb6c52487e5111c82f024b3593dd0afabedf8d4610421739b081780ac476"
Oct 30 18:54:24.631387 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:24.631361    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e6afb6c52487e5111c82f024b3593dd0afabedf8d4610421739b081780ac476"} err="failed to get container status \"8e6afb6c52487e5111c82f024b3593dd0afabedf8d4610421739b081780ac476\": rpc error: code = NotFound desc = could not find container \"8e6afb6c52487e5111c82f024b3593dd0afabedf8d4610421739b081780ac476\": container with ID starting with 8e6afb6c52487e5111c82f024b3593dd0afabedf8d4610421739b081780ac476 not found: ID does not exist"
Oct 30 18:54:24.631490 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:24.631480    4525 scope.go:122] "RemoveContainer" containerID="67aa60478f70c3fa5eb0938ee01dd809a741e22324dd53c360be0b09d59cb8f8"
Oct 30 18:54:24.631940 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:54:24.631920    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67aa60478f70c3fa5eb0938ee01dd809a741e22324dd53c360be0b09d59cb8f8\": container with ID starting with 67aa60478f70c3fa5eb0938ee01dd809a741e22324dd53c360be0b09d59cb8f8 not found: ID does not exist" containerID="67aa60478f70c3fa5eb0938ee01dd809a741e22324dd53c360be0b09d59cb8f8"
Oct 30 18:54:24.632031 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:24.631948    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67aa60478f70c3fa5eb0938ee01dd809a741e22324dd53c360be0b09d59cb8f8"} err="failed to get container status \"67aa60478f70c3fa5eb0938ee01dd809a741e22324dd53c360be0b09d59cb8f8\": rpc error: code = NotFound desc = could not find container \"67aa60478f70c3fa5eb0938ee01dd809a741e22324dd53c360be0b09d59cb8f8\": container with ID starting with 67aa60478f70c3fa5eb0938ee01dd809a741e22324dd53c360be0b09d59cb8f8 not found: ID does not exist"
Oct 30 18:54:24.653346 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:24.653152    4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf796179-ce62-433b-87b8-a8bb1405c70b-istio-envoy\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:24.653346 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:24.653179    4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf796179-ce62-433b-87b8-a8bb1405c70b-istio-podinfo\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:24.653346 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:24.653189    4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf796179-ce62-433b-87b8-a8bb1405c70b-workload-certs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:24.653346 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:24.653197    4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf796179-ce62-433b-87b8-a8bb1405c70b-credential-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:24.653346 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:24.653206    4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf796179-ce62-433b-87b8-a8bb1405c70b-istio-token\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:24.653346 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:24.653215    4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf796179-ce62-433b-87b8-a8bb1405c70b-istio-data\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:24.653346 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:24.653223    4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf796179-ce62-433b-87b8-a8bb1405c70b-workload-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:24.653346 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:24.653232    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtwft\" (UniqueName: \"kubernetes.io/projected/cf796179-ce62-433b-87b8-a8bb1405c70b-kube-api-access-vtwft\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:24.653346 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:24.653241    4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf796179-ce62-433b-87b8-a8bb1405c70b-istiod-ca-cert\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:24.843653 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:24.843611    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-87293f-openshift-default-7c9c9cbc7f-vw6fm"]
Oct 30 18:54:24.849073 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:24.849046    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-87293f-openshift-default-7c9c9cbc7f-vw6fm"]
Oct 30 18:54:25.042717 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:25.042673    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-22qrl"
Oct 30 18:54:25.160776 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:25.160745    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13c9b518-512f-4a88-ae9e-cd8e9fb247cb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13c9b518-512f-4a88-ae9e-cd8e9fb247cb-credential-socket\") pod \"13c9b518-512f-4a88-ae9e-cd8e9fb247cb\" (UID: \"13c9b518-512f-4a88-ae9e-cd8e9fb247cb\") "
Oct 30 18:54:25.160990 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:25.160801    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13c9b518-512f-4a88-ae9e-cd8e9fb247cb-kube-api-access-smh8j\" (UniqueName: \"kubernetes.io/projected/13c9b518-512f-4a88-ae9e-cd8e9fb247cb-kube-api-access-smh8j\") pod \"13c9b518-512f-4a88-ae9e-cd8e9fb247cb\" (UID: \"13c9b518-512f-4a88-ae9e-cd8e9fb247cb\") "
Oct 30 18:54:25.160990 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:25.160853    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/13c9b518-512f-4a88-ae9e-cd8e9fb247cb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13c9b518-512f-4a88-ae9e-cd8e9fb247cb-istio-podinfo\") pod \"13c9b518-512f-4a88-ae9e-cd8e9fb247cb\" (UID: \"13c9b518-512f-4a88-ae9e-cd8e9fb247cb\") "
Oct 30 18:54:25.160990 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:25.160745    4525 patch_prober.go:29] interesting pod/e2e-mt-da58b769-openshift-default-69887ddf8f-n79ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Oct 30 18:54:25.160990 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:25.160955    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-da58b769-openshift-default-69887ddf8f-n79ls" podUID="81739e60-d074-4a42-ade3-a0f68ea9329d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Oct 30 18:54:25.160990 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:25.160889    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13c9b518-512f-4a88-ae9e-cd8e9fb247cb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13c9b518-512f-4a88-ae9e-cd8e9fb247cb-workload-socket\") pod \"13c9b518-512f-4a88-ae9e-cd8e9fb247cb\" (UID: \"13c9b518-512f-4a88-ae9e-cd8e9fb247cb\") "
Oct 30 18:54:25.160990 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:25.160979    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13c9b518-512f-4a88-ae9e-cd8e9fb247cb-credential-socket" pod "13c9b518-512f-4a88-ae9e-cd8e9fb247cb" (UID: "13c9b518-512f-4a88-ae9e-cd8e9fb247cb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:54:25.161335 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:25.161159    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13c9b518-512f-4a88-ae9e-cd8e9fb247cb-workload-socket" pod "13c9b518-512f-4a88-ae9e-cd8e9fb247cb" (UID: "13c9b518-512f-4a88-ae9e-cd8e9fb247cb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:54:25.161335 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:25.161177    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13c9b518-512f-4a88-ae9e-cd8e9fb247cb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13c9b518-512f-4a88-ae9e-cd8e9fb247cb-istio-envoy\") pod \"13c9b518-512f-4a88-ae9e-cd8e9fb247cb\" (UID: \"13c9b518-512f-4a88-ae9e-cd8e9fb247cb\") "
Oct 30 18:54:25.161335 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:25.161232    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13c9b518-512f-4a88-ae9e-cd8e9fb247cb-istio-token\" (UniqueName: \"kubernetes.io/projected/13c9b518-512f-4a88-ae9e-cd8e9fb247cb-istio-token\") pod \"13c9b518-512f-4a88-ae9e-cd8e9fb247cb\" (UID: \"13c9b518-512f-4a88-ae9e-cd8e9fb247cb\") "
Oct 30 18:54:25.161335 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:25.161296    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13c9b518-512f-4a88-ae9e-cd8e9fb247cb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13c9b518-512f-4a88-ae9e-cd8e9fb247cb-istio-data\") pod \"13c9b518-512f-4a88-ae9e-cd8e9fb247cb\" (UID: \"13c9b518-512f-4a88-ae9e-cd8e9fb247cb\") "
Oct 30 18:54:25.161614 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:25.161356    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13c9b518-512f-4a88-ae9e-cd8e9fb247cb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13c9b518-512f-4a88-ae9e-cd8e9fb247cb-workload-certs\") pod \"13c9b518-512f-4a88-ae9e-cd8e9fb247cb\" (UID: \"13c9b518-512f-4a88-ae9e-cd8e9fb247cb\") "
Oct 30 18:54:25.161614 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:25.161410    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13c9b518-512f-4a88-ae9e-cd8e9fb247cb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13c9b518-512f-4a88-ae9e-cd8e9fb247cb-istiod-ca-cert\") pod \"13c9b518-512f-4a88-ae9e-cd8e9fb247cb\" (UID: \"13c9b518-512f-4a88-ae9e-cd8e9fb247cb\") "
Oct 30 18:54:25.161614 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:25.161511    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13c9b518-512f-4a88-ae9e-cd8e9fb247cb-istio-data" pod "13c9b518-512f-4a88-ae9e-cd8e9fb247cb" (UID: "13c9b518-512f-4a88-ae9e-cd8e9fb247cb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:54:25.161614 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:25.161632    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13c9b518-512f-4a88-ae9e-cd8e9fb247cb-workload-certs" pod "13c9b518-512f-4a88-ae9e-cd8e9fb247cb" (UID: "13c9b518-512f-4a88-ae9e-cd8e9fb247cb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:54:25.161983 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:25.161860    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13c9b518-512f-4a88-ae9e-cd8e9fb247cb-istiod-ca-cert" pod "13c9b518-512f-4a88-ae9e-cd8e9fb247cb" (UID: "13c9b518-512f-4a88-ae9e-cd8e9fb247cb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:54:25.162296 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:25.162273    4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13c9b518-512f-4a88-ae9e-cd8e9fb247cb-istio-data\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:25.162381 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:25.162301    4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13c9b518-512f-4a88-ae9e-cd8e9fb247cb-workload-certs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:25.162381 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:25.162318    4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13c9b518-512f-4a88-ae9e-cd8e9fb247cb-istiod-ca-cert\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:25.162381 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:25.162332    4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13c9b518-512f-4a88-ae9e-cd8e9fb247cb-credential-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:25.162381 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:25.162346    4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13c9b518-512f-4a88-ae9e-cd8e9fb247cb-workload-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:25.163174 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:25.163146    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/13c9b518-512f-4a88-ae9e-cd8e9fb247cb-istio-podinfo" pod "13c9b518-512f-4a88-ae9e-cd8e9fb247cb" (UID: "13c9b518-512f-4a88-ae9e-cd8e9fb247cb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 18:54:25.163322 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:25.163304    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13c9b518-512f-4a88-ae9e-cd8e9fb247cb-kube-api-access-smh8j" pod "13c9b518-512f-4a88-ae9e-cd8e9fb247cb" (UID: "13c9b518-512f-4a88-ae9e-cd8e9fb247cb"). InnerVolumeSpecName "kube-api-access-smh8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:54:25.163545 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:25.163527    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13c9b518-512f-4a88-ae9e-cd8e9fb247cb-istio-token" pod "13c9b518-512f-4a88-ae9e-cd8e9fb247cb" (UID: "13c9b518-512f-4a88-ae9e-cd8e9fb247cb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:54:25.163585 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:25.163525    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13c9b518-512f-4a88-ae9e-cd8e9fb247cb-istio-envoy" pod "13c9b518-512f-4a88-ae9e-cd8e9fb247cb" (UID: "13c9b518-512f-4a88-ae9e-cd8e9fb247cb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:54:25.263849 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:25.263812    4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13c9b518-512f-4a88-ae9e-cd8e9fb247cb-istio-envoy\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:25.263849 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:25.263846    4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13c9b518-512f-4a88-ae9e-cd8e9fb247cb-istio-token\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:25.263849 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:25.263856    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smh8j\" (UniqueName: \"kubernetes.io/projected/13c9b518-512f-4a88-ae9e-cd8e9fb247cb-kube-api-access-smh8j\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:25.264090 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:25.263866    4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13c9b518-512f-4a88-ae9e-cd8e9fb247cb-istio-podinfo\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:25.525819 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:25.525778    4525 generic.go:356] "Generic (PLEG): container finished" podID="13c9b518-512f-4a88-ae9e-cd8e9fb247cb" containerID="b95c5130210fa52372bd5ff3ef7c5602ec5026b642a357cc40f3aeb88fff06be" exitCode=0
Oct 30 18:54:25.526278 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:25.525825    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-22qrl"
Oct 30 18:54:25.526278 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:25.525870    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-22qrl" event={"ID":"13c9b518-512f-4a88-ae9e-cd8e9fb247cb","Type":"ContainerDied","Data":"b95c5130210fa52372bd5ff3ef7c5602ec5026b642a357cc40f3aeb88fff06be"}
Oct 30 18:54:25.526278 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:25.525932    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-22qrl" event={"ID":"13c9b518-512f-4a88-ae9e-cd8e9fb247cb","Type":"ContainerDied","Data":"8aaa71ab48983fb2236d70e51b577e0d4c7c15f6d8aa0bf25032fcfb75fe678e"}
Oct 30 18:54:25.526278 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:25.525967    4525 scope.go:122] "RemoveContainer" containerID="b95c5130210fa52372bd5ff3ef7c5602ec5026b642a357cc40f3aeb88fff06be"
Oct 30 18:54:25.537919 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:25.537900    4525 scope.go:122] "RemoveContainer" containerID="b95c5130210fa52372bd5ff3ef7c5602ec5026b642a357cc40f3aeb88fff06be"
Oct 30 18:54:25.538234 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:54:25.538211    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b95c5130210fa52372bd5ff3ef7c5602ec5026b642a357cc40f3aeb88fff06be\": container with ID starting with b95c5130210fa52372bd5ff3ef7c5602ec5026b642a357cc40f3aeb88fff06be not found: ID does not exist" containerID="b95c5130210fa52372bd5ff3ef7c5602ec5026b642a357cc40f3aeb88fff06be"
Oct 30 18:54:25.538280 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:25.538250    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b95c5130210fa52372bd5ff3ef7c5602ec5026b642a357cc40f3aeb88fff06be"} err="failed to get container status \"b95c5130210fa52372bd5ff3ef7c5602ec5026b642a357cc40f3aeb88fff06be\": rpc error: code = NotFound desc = could not find container \"b95c5130210fa52372bd5ff3ef7c5602ec5026b642a357cc40f3aeb88fff06be\": container with ID starting with b95c5130210fa52372bd5ff3ef7c5602ec5026b642a357cc40f3aeb88fff06be not found: ID does not exist"
Oct 30 18:54:25.546055 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:25.546030    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf796179-ce62-433b-87b8-a8bb1405c70b" path="/var/lib/kubelet/pods/cf796179-ce62-433b-87b8-a8bb1405c70b/volumes"
Oct 30 18:54:25.554460 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:25.554426    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-22qrl"]
Oct 30 18:54:25.559525 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:25.559502    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-f37d42-openshift-default-7645c57f7c-22qrl"]
Oct 30 18:54:26.160817 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:26.160785    4525 patch_prober.go:29] interesting pod/e2e-mt-da58b769-openshift-default-69887ddf8f-n79ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Oct 30 18:54:26.160982 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:26.160843    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-da58b769-openshift-default-69887ddf8f-n79ls" podUID="81739e60-d074-4a42-ade3-a0f68ea9329d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Oct 30 18:54:26.906922 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:26.906880    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-da58b769-openshift-default-69887ddf8f-n79ls"
Oct 30 18:54:27.160586 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:27.160501    4525 patch_prober.go:29] interesting pod/e2e-mt-da58b769-openshift-default-69887ddf8f-n79ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Oct 30 18:54:27.160586 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:27.160556    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-da58b769-openshift-default-69887ddf8f-n79ls" podUID="81739e60-d074-4a42-ade3-a0f68ea9329d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Oct 30 18:54:27.546036 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:27.545948    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13c9b518-512f-4a88-ae9e-cd8e9fb247cb" path="/var/lib/kubelet/pods/13c9b518-512f-4a88-ae9e-cd8e9fb247cb/volumes"
Oct 30 18:54:28.161293 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:28.160427    4525 patch_prober.go:29] interesting pod/e2e-mt-da58b769-openshift-default-69887ddf8f-n79ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Oct 30 18:54:28.161293 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:28.160496    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-da58b769-openshift-default-69887ddf8f-n79ls" podUID="81739e60-d074-4a42-ade3-a0f68ea9329d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Oct 30 18:54:29.164642 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:29.164608    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-da58b769-openshift-default-69887ddf8f-n79ls"
Oct 30 18:54:29.165733 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:29.165665    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-da58b769-openshift-default-69887ddf8f-n79ls"
Oct 30 18:54:38.036513 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:38.035824    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7cfca6cc-bhbpv"]
Oct 30 18:54:38.049117 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:38.049059    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7cfca6cc-bhbpv"]
Oct 30 18:54:39.546037 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:39.545999    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35a6c3d1-e874-4203-aa17-d48e3a99e9ee" path="/var/lib/kubelet/pods/35a6c3d1-e874-4203-aa17-d48e3a99e9ee/volumes"
Oct 30 18:54:42.993831 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:42.993804    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-066928-8hzcc"]
Oct 30 18:54:42.998752 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:42.998731    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-066928-8hzcc"]
Oct 30 18:54:43.545952 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:43.545912    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eeccf097-40f0-440a-a90e-3b97186ef05f" path="/var/lib/kubelet/pods/eeccf097-40f0-440a-a90e-3b97186ef05f/volumes"
Oct 30 18:54:44.045758 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:44.044383    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7784e5-wl98w"]
Oct 30 18:54:44.053243 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:44.053203    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7784e5-wl98w"]
Oct 30 18:54:45.604817 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:45.604784    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fdcc41d-76be-4f42-9f6e-af06f8169b50" path="/var/lib/kubelet/pods/6fdcc41d-76be-4f42-9f6e-af06f8169b50/volumes"
Oct 30 18:54:47.398224 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:54:47.398186    4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d49l" containerName="extractor"
Oct 30 18:54:49.028552 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:49.028507    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cce717cd-wxnlh"]
Oct 30 18:54:49.034130 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:49.034098    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cce717cd-wxnlh"]
Oct 30 18:54:49.558327 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:49.558295    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5bf0f22-bbf6-4d98-ae18-51f107bd13d3" path="/var/lib/kubelet/pods/a5bf0f22-bbf6-4d98-ae18-51f107bd13d3/volumes"
Oct 30 18:54:50.221668 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:50.221632    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-066928-openshift-default-55ddf5c76f-pzlmw"]
Oct 30 18:54:50.222190 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:50.221918    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-066928-openshift-default-55ddf5c76f-pzlmw" podUID="a9ff1283-3fc7-463f-b3b7-fafdd56f2349" containerName="istio-proxy" containerID="cri-o://0913e11a983a273fe009cde085966709a530584998b47af0f93b9d0f277fd64a" gracePeriod=30
Oct 30 18:54:51.893519 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:51.893422    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-7784e5-openshift-default-564776f9f-zm98k"]
Oct 30 18:54:51.924708 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:51.910090    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-7784e5-openshift-default-564776f9f-zm98k" podUID="97979488-7a76-4f4e-b330-49432c98bbec" containerName="istio-proxy" containerID="cri-o://d600190c7dd6e007e853cdd210e70a8a50c291e874175a4b78c9ee1fc46c1b63" gracePeriod=30
Oct 30 18:54:52.036648 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:52.036609    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-436f4927-hz94c"]
Oct 30 18:54:52.043483 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:52.043454    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-436f4927-hz94c"]
Oct 30 18:54:53.028219 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:53.028181    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2fcdf464-jkxd9"]
Oct 30 18:54:53.035485 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:53.035458    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2fcdf464-jkxd9"]
Oct 30 18:54:53.548417 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:53.548368    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13733f44-8aaa-4710-aa5f-be68364f5a14" path="/var/lib/kubelet/pods/13733f44-8aaa-4710-aa5f-be68364f5a14/volumes"
Oct 30 18:54:53.548878 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:53.548862    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c09c6117-2835-4013-b4f8-9a4f6ef8064c" path="/var/lib/kubelet/pods/c09c6117-2835-4013-b4f8-9a4f6ef8064c/volumes"
Oct 30 18:54:55.496420 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:55.496360    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-066928-openshift-default-55ddf5c76f-pzlmw"
Oct 30 18:54:55.575261 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:55.575228    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9ff1283-3fc7-463f-b3b7-fafdd56f2349-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9ff1283-3fc7-463f-b3b7-fafdd56f2349-credential-socket\") pod \"a9ff1283-3fc7-463f-b3b7-fafdd56f2349\" (UID: \"a9ff1283-3fc7-463f-b3b7-fafdd56f2349\") "
Oct 30 18:54:55.575475 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:55.575320    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9ff1283-3fc7-463f-b3b7-fafdd56f2349-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9ff1283-3fc7-463f-b3b7-fafdd56f2349-istiod-ca-cert\") pod \"a9ff1283-3fc7-463f-b3b7-fafdd56f2349\" (UID: \"a9ff1283-3fc7-463f-b3b7-fafdd56f2349\") "
Oct 30 18:54:55.575475 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:55.575435    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9ff1283-3fc7-463f-b3b7-fafdd56f2349-kube-api-access-mhpkq\" (UniqueName: \"kubernetes.io/projected/a9ff1283-3fc7-463f-b3b7-fafdd56f2349-kube-api-access-mhpkq\") pod \"a9ff1283-3fc7-463f-b3b7-fafdd56f2349\" (UID: \"a9ff1283-3fc7-463f-b3b7-fafdd56f2349\") "
Oct 30 18:54:55.575567 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:55.575477    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9ff1283-3fc7-463f-b3b7-fafdd56f2349-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9ff1283-3fc7-463f-b3b7-fafdd56f2349-istio-envoy\") pod \"a9ff1283-3fc7-463f-b3b7-fafdd56f2349\" (UID: \"a9ff1283-3fc7-463f-b3b7-fafdd56f2349\") "
Oct 30 18:54:55.575567 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:55.575516    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9ff1283-3fc7-463f-b3b7-fafdd56f2349-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9ff1283-3fc7-463f-b3b7-fafdd56f2349-workload-socket\") pod \"a9ff1283-3fc7-463f-b3b7-fafdd56f2349\" (UID: \"a9ff1283-3fc7-463f-b3b7-fafdd56f2349\") "
Oct 30 18:54:55.575567 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:55.575545    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9ff1283-3fc7-463f-b3b7-fafdd56f2349-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9ff1283-3fc7-463f-b3b7-fafdd56f2349-workload-certs\") pod \"a9ff1283-3fc7-463f-b3b7-fafdd56f2349\" (UID: \"a9ff1283-3fc7-463f-b3b7-fafdd56f2349\") "
Oct 30 18:54:55.575567 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:55.575540    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9ff1283-3fc7-463f-b3b7-fafdd56f2349-credential-socket" pod "a9ff1283-3fc7-463f-b3b7-fafdd56f2349" (UID: "a9ff1283-3fc7-463f-b3b7-fafdd56f2349"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:54:55.575755 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:55.575653    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a9ff1283-3fc7-463f-b3b7-fafdd56f2349-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9ff1283-3fc7-463f-b3b7-fafdd56f2349-istio-podinfo\") pod \"a9ff1283-3fc7-463f-b3b7-fafdd56f2349\" (UID: \"a9ff1283-3fc7-463f-b3b7-fafdd56f2349\") "
Oct 30 18:54:55.575814 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:55.575754    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9ff1283-3fc7-463f-b3b7-fafdd56f2349-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9ff1283-3fc7-463f-b3b7-fafdd56f2349-istio-data\") pod \"a9ff1283-3fc7-463f-b3b7-fafdd56f2349\" (UID: \"a9ff1283-3fc7-463f-b3b7-fafdd56f2349\") "
Oct 30 18:54:55.575814 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:55.575766    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9ff1283-3fc7-463f-b3b7-fafdd56f2349-istiod-ca-cert" pod "a9ff1283-3fc7-463f-b3b7-fafdd56f2349" (UID: "a9ff1283-3fc7-463f-b3b7-fafdd56f2349"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:54:55.575814 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:55.575774    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9ff1283-3fc7-463f-b3b7-fafdd56f2349-workload-socket" pod "a9ff1283-3fc7-463f-b3b7-fafdd56f2349" (UID: "a9ff1283-3fc7-463f-b3b7-fafdd56f2349"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:54:55.575814 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:55.575788    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9ff1283-3fc7-463f-b3b7-fafdd56f2349-istio-token\" (UniqueName: \"kubernetes.io/projected/a9ff1283-3fc7-463f-b3b7-fafdd56f2349-istio-token\") pod \"a9ff1283-3fc7-463f-b3b7-fafdd56f2349\" (UID: \"a9ff1283-3fc7-463f-b3b7-fafdd56f2349\") "
Oct 30 18:54:55.576014 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:55.575934    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9ff1283-3fc7-463f-b3b7-fafdd56f2349-workload-certs" pod "a9ff1283-3fc7-463f-b3b7-fafdd56f2349" (UID: "a9ff1283-3fc7-463f-b3b7-fafdd56f2349"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:54:55.576077 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:55.576055    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9ff1283-3fc7-463f-b3b7-fafdd56f2349-istio-data" pod "a9ff1283-3fc7-463f-b3b7-fafdd56f2349" (UID: "a9ff1283-3fc7-463f-b3b7-fafdd56f2349"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:54:55.576744 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:55.576718    4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9ff1283-3fc7-463f-b3b7-fafdd56f2349-istio-data\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:55.576902 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:55.576748    4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9ff1283-3fc7-463f-b3b7-fafdd56f2349-credential-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:55.576902 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:55.576762    4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9ff1283-3fc7-463f-b3b7-fafdd56f2349-istiod-ca-cert\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:55.576902 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:55.576774    4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9ff1283-3fc7-463f-b3b7-fafdd56f2349-workload-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:55.576902 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:55.576787    4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9ff1283-3fc7-463f-b3b7-fafdd56f2349-workload-certs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:55.578003 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:55.577974    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9ff1283-3fc7-463f-b3b7-fafdd56f2349-kube-api-access-mhpkq" pod "a9ff1283-3fc7-463f-b3b7-fafdd56f2349" (UID: "a9ff1283-3fc7-463f-b3b7-fafdd56f2349"). InnerVolumeSpecName "kube-api-access-mhpkq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:54:55.578003 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:55.577986    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9ff1283-3fc7-463f-b3b7-fafdd56f2349-istio-envoy" pod "a9ff1283-3fc7-463f-b3b7-fafdd56f2349" (UID: "a9ff1283-3fc7-463f-b3b7-fafdd56f2349"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:54:55.578202 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:55.578028    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a9ff1283-3fc7-463f-b3b7-fafdd56f2349-istio-podinfo" pod "a9ff1283-3fc7-463f-b3b7-fafdd56f2349" (UID: "a9ff1283-3fc7-463f-b3b7-fafdd56f2349"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 18:54:55.578202 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:55.578079    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9ff1283-3fc7-463f-b3b7-fafdd56f2349-istio-token" pod "a9ff1283-3fc7-463f-b3b7-fafdd56f2349" (UID: "a9ff1283-3fc7-463f-b3b7-fafdd56f2349"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:54:55.678460 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:55.678430    4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9ff1283-3fc7-463f-b3b7-fafdd56f2349-istio-podinfo\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:55.678460 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:55.678458    4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9ff1283-3fc7-463f-b3b7-fafdd56f2349-istio-token\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:55.678460 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:55.678468    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhpkq\" (UniqueName: \"kubernetes.io/projected/a9ff1283-3fc7-463f-b3b7-fafdd56f2349-kube-api-access-mhpkq\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:55.678746 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:55.678479    4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9ff1283-3fc7-463f-b3b7-fafdd56f2349-istio-envoy\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:55.970464 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:55.970430    4525 generic.go:356] "Generic (PLEG): container finished" podID="a9ff1283-3fc7-463f-b3b7-fafdd56f2349" containerID="0913e11a983a273fe009cde085966709a530584998b47af0f93b9d0f277fd64a" exitCode=0
Oct 30 18:54:55.971056 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:55.970492    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-066928-openshift-default-55ddf5c76f-pzlmw"
Oct 30 18:54:55.971056 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:55.970519    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-066928-openshift-default-55ddf5c76f-pzlmw" event={"ID":"a9ff1283-3fc7-463f-b3b7-fafdd56f2349","Type":"ContainerDied","Data":"0913e11a983a273fe009cde085966709a530584998b47af0f93b9d0f277fd64a"}
Oct 30 18:54:55.971056 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:55.970582    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-066928-openshift-default-55ddf5c76f-pzlmw" event={"ID":"a9ff1283-3fc7-463f-b3b7-fafdd56f2349","Type":"ContainerDied","Data":"797b6ac2e5e6ffd1e50e7e59d4748b9da3cb994aabacb1481136fa4da18a6461"}
Oct 30 18:54:55.971056 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:55.970605    4525 scope.go:122] "RemoveContainer" containerID="0913e11a983a273fe009cde085966709a530584998b47af0f93b9d0f277fd64a"
Oct 30 18:54:55.993288 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:55.993266    4525 scope.go:122] "RemoveContainer" containerID="5cfe5306c8da53186d023ff7953bf87c054d9bdea82fcdfad2919e3366f85faf"
Oct 30 18:54:56.004768 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:56.004740    4525 scope.go:122] "RemoveContainer" containerID="0913e11a983a273fe009cde085966709a530584998b47af0f93b9d0f277fd64a"
Oct 30 18:54:56.005082 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:56.005060    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-066928-openshift-default-55ddf5c76f-pzlmw"]
Oct 30 18:54:56.005222 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:54:56.005079    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0913e11a983a273fe009cde085966709a530584998b47af0f93b9d0f277fd64a\": container with ID starting with 0913e11a983a273fe009cde085966709a530584998b47af0f93b9d0f277fd64a not found: ID does not exist" containerID="0913e11a983a273fe009cde085966709a530584998b47af0f93b9d0f277fd64a"
Oct 30 18:54:56.005222 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:56.005119    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0913e11a983a273fe009cde085966709a530584998b47af0f93b9d0f277fd64a"} err="failed to get container status \"0913e11a983a273fe009cde085966709a530584998b47af0f93b9d0f277fd64a\": rpc error: code = NotFound desc = could not find container \"0913e11a983a273fe009cde085966709a530584998b47af0f93b9d0f277fd64a\": container with ID starting with 0913e11a983a273fe009cde085966709a530584998b47af0f93b9d0f277fd64a not found: ID does not exist"
Oct 30 18:54:56.005222 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:56.005143    4525 scope.go:122] "RemoveContainer" containerID="5cfe5306c8da53186d023ff7953bf87c054d9bdea82fcdfad2919e3366f85faf"
Oct 30 18:54:56.005403 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:54:56.005365    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cfe5306c8da53186d023ff7953bf87c054d9bdea82fcdfad2919e3366f85faf\": container with ID starting with 5cfe5306c8da53186d023ff7953bf87c054d9bdea82fcdfad2919e3366f85faf not found: ID does not exist" containerID="5cfe5306c8da53186d023ff7953bf87c054d9bdea82fcdfad2919e3366f85faf"
Oct 30 18:54:56.005403 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:56.005382    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cfe5306c8da53186d023ff7953bf87c054d9bdea82fcdfad2919e3366f85faf"} err="failed to get container status \"5cfe5306c8da53186d023ff7953bf87c054d9bdea82fcdfad2919e3366f85faf\": rpc error: code = NotFound desc = could not find container \"5cfe5306c8da53186d023ff7953bf87c054d9bdea82fcdfad2919e3366f85faf\": container with ID starting with 5cfe5306c8da53186d023ff7953bf87c054d9bdea82fcdfad2919e3366f85faf not found: ID does not exist"
Oct 30 18:54:56.010460 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:56.010436    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-066928-openshift-default-55ddf5c76f-pzlmw"]
Oct 30 18:54:56.025156 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:56.025130    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60644dac-2k9w7"]
Oct 30 18:54:56.029754 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:56.029735    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60644dac-2k9w7"]
Oct 30 18:54:56.302474 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:56.302391    4525 patch_prober.go:29] interesting pod/e2e-api-b-7784e5-openshift-default-564776f9f-zm98k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Oct 30 18:54:56.302474 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:56.302451    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-7784e5-openshift-default-564776f9f-zm98k" podUID="97979488-7a76-4f4e-b330-49432c98bbec" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:54:56.982430 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:56.982398    4525 generic.go:356] "Generic (PLEG): container finished" podID="97979488-7a76-4f4e-b330-49432c98bbec" containerID="d600190c7dd6e007e853cdd210e70a8a50c291e874175a4b78c9ee1fc46c1b63" exitCode=0
Oct 30 18:54:56.982794 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:56.982444    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7784e5-openshift-default-564776f9f-zm98k" event={"ID":"97979488-7a76-4f4e-b330-49432c98bbec","Type":"ContainerDied","Data":"d600190c7dd6e007e853cdd210e70a8a50c291e874175a4b78c9ee1fc46c1b63"}
Oct 30 18:54:56.982794 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:56.982516    4525 scope.go:122] "RemoveContainer" containerID="c864ba5c8cf4b56772ab67170cbdd9ed6825d5373cae448336a7a11ba15b8609"
Oct 30 18:54:57.546016 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:57.545981    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4adf49fc-7712-4a37-a27e-72500f89d9d0" path="/var/lib/kubelet/pods/4adf49fc-7712-4a37-a27e-72500f89d9d0/volumes"
Oct 30 18:54:57.546621 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:57.546600    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9ff1283-3fc7-463f-b3b7-fafdd56f2349" path="/var/lib/kubelet/pods/a9ff1283-3fc7-463f-b3b7-fafdd56f2349/volumes"
Oct 30 18:54:57.638409 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:57.638381    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-7784e5-openshift-default-564776f9f-zm98k"
Oct 30 18:54:57.709071 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:57.709040    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97979488-7a76-4f4e-b330-49432c98bbec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97979488-7a76-4f4e-b330-49432c98bbec-workload-socket\") pod \"97979488-7a76-4f4e-b330-49432c98bbec\" (UID: \"97979488-7a76-4f4e-b330-49432c98bbec\") "
Oct 30 18:54:57.709266 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:57.709088    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97979488-7a76-4f4e-b330-49432c98bbec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97979488-7a76-4f4e-b330-49432c98bbec-credential-socket\") pod \"97979488-7a76-4f4e-b330-49432c98bbec\" (UID: \"97979488-7a76-4f4e-b330-49432c98bbec\") "
Oct 30 18:54:57.709266 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:57.709155    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/97979488-7a76-4f4e-b330-49432c98bbec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97979488-7a76-4f4e-b330-49432c98bbec-istio-podinfo\") pod \"97979488-7a76-4f4e-b330-49432c98bbec\" (UID: \"97979488-7a76-4f4e-b330-49432c98bbec\") "
Oct 30 18:54:57.709266 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:57.709184    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97979488-7a76-4f4e-b330-49432c98bbec-kube-api-access-dpx46\" (UniqueName: \"kubernetes.io/projected/97979488-7a76-4f4e-b330-49432c98bbec-kube-api-access-dpx46\") pod \"97979488-7a76-4f4e-b330-49432c98bbec\" (UID: \"97979488-7a76-4f4e-b330-49432c98bbec\") "
Oct 30 18:54:57.709453 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:57.709396    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97979488-7a76-4f4e-b330-49432c98bbec-workload-socket" pod "97979488-7a76-4f4e-b330-49432c98bbec" (UID: "97979488-7a76-4f4e-b330-49432c98bbec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:54:57.709636 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:57.709470    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97979488-7a76-4f4e-b330-49432c98bbec-credential-socket" pod "97979488-7a76-4f4e-b330-49432c98bbec" (UID: "97979488-7a76-4f4e-b330-49432c98bbec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:54:57.709787 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:57.709511    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97979488-7a76-4f4e-b330-49432c98bbec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97979488-7a76-4f4e-b330-49432c98bbec-workload-certs\") pod \"97979488-7a76-4f4e-b330-49432c98bbec\" (UID: \"97979488-7a76-4f4e-b330-49432c98bbec\") "
Oct 30 18:54:57.709866 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:57.709816    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97979488-7a76-4f4e-b330-49432c98bbec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97979488-7a76-4f4e-b330-49432c98bbec-istio-envoy\") pod \"97979488-7a76-4f4e-b330-49432c98bbec\" (UID: \"97979488-7a76-4f4e-b330-49432c98bbec\") "
Oct 30 18:54:57.709866 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:57.709669    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97979488-7a76-4f4e-b330-49432c98bbec-workload-certs" pod "97979488-7a76-4f4e-b330-49432c98bbec" (UID: "97979488-7a76-4f4e-b330-49432c98bbec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:54:57.709978 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:57.709853    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97979488-7a76-4f4e-b330-49432c98bbec-istio-token\" (UniqueName: \"kubernetes.io/projected/97979488-7a76-4f4e-b330-49432c98bbec-istio-token\") pod \"97979488-7a76-4f4e-b330-49432c98bbec\" (UID: \"97979488-7a76-4f4e-b330-49432c98bbec\") "
Oct 30 18:54:57.709978 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:57.709932    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97979488-7a76-4f4e-b330-49432c98bbec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97979488-7a76-4f4e-b330-49432c98bbec-istiod-ca-cert\") pod \"97979488-7a76-4f4e-b330-49432c98bbec\" (UID: \"97979488-7a76-4f4e-b330-49432c98bbec\") "
Oct 30 18:54:57.711074 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:57.710203    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97979488-7a76-4f4e-b330-49432c98bbec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97979488-7a76-4f4e-b330-49432c98bbec-istio-data\") pod \"97979488-7a76-4f4e-b330-49432c98bbec\" (UID: \"97979488-7a76-4f4e-b330-49432c98bbec\") "
Oct 30 18:54:57.711074 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:57.710513    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97979488-7a76-4f4e-b330-49432c98bbec-istio-data" pod "97979488-7a76-4f4e-b330-49432c98bbec" (UID: "97979488-7a76-4f4e-b330-49432c98bbec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:54:57.711074 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:57.710550    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97979488-7a76-4f4e-b330-49432c98bbec-istiod-ca-cert" pod "97979488-7a76-4f4e-b330-49432c98bbec" (UID: "97979488-7a76-4f4e-b330-49432c98bbec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:54:57.711074 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:57.710944    4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97979488-7a76-4f4e-b330-49432c98bbec-credential-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:57.711074 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:57.710967    4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97979488-7a76-4f4e-b330-49432c98bbec-workload-certs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:57.711074 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:57.710982    4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97979488-7a76-4f4e-b330-49432c98bbec-istiod-ca-cert\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:57.711074 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:57.710995    4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97979488-7a76-4f4e-b330-49432c98bbec-istio-data\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:57.711074 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:57.711009    4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97979488-7a76-4f4e-b330-49432c98bbec-workload-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:57.712285 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:57.712258    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/97979488-7a76-4f4e-b330-49432c98bbec-istio-podinfo" pod "97979488-7a76-4f4e-b330-49432c98bbec" (UID: "97979488-7a76-4f4e-b330-49432c98bbec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 18:54:57.712890 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:57.712863    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97979488-7a76-4f4e-b330-49432c98bbec-kube-api-access-dpx46" pod "97979488-7a76-4f4e-b330-49432c98bbec" (UID: "97979488-7a76-4f4e-b330-49432c98bbec"). InnerVolumeSpecName "kube-api-access-dpx46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:54:57.713342 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:57.713312    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97979488-7a76-4f4e-b330-49432c98bbec-istio-token" pod "97979488-7a76-4f4e-b330-49432c98bbec" (UID: "97979488-7a76-4f4e-b330-49432c98bbec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:54:57.713435 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:57.713371    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97979488-7a76-4f4e-b330-49432c98bbec-istio-envoy" pod "97979488-7a76-4f4e-b330-49432c98bbec" (UID: "97979488-7a76-4f4e-b330-49432c98bbec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:54:57.816375 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:57.816333    4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97979488-7a76-4f4e-b330-49432c98bbec-istio-podinfo\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:57.816375 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:57.816377    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpx46\" (UniqueName: \"kubernetes.io/projected/97979488-7a76-4f4e-b330-49432c98bbec-kube-api-access-dpx46\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:57.816375 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:57.816396    4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97979488-7a76-4f4e-b330-49432c98bbec-istio-envoy\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:57.816375 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:57.816411    4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97979488-7a76-4f4e-b330-49432c98bbec-istio-token\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:57.992300 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:57.992223    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-7784e5-openshift-default-564776f9f-zm98k"
Oct 30 18:54:57.992300 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:57.992260    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7784e5-openshift-default-564776f9f-zm98k" event={"ID":"97979488-7a76-4f4e-b330-49432c98bbec","Type":"ContainerDied","Data":"fdc1f26257fc1b1eaf24ecd285a6391b55fe11ee51f9060fbb5abf202bafba78"}
Oct 30 18:54:57.992830 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:57.992321    4525 scope.go:122] "RemoveContainer" containerID="d600190c7dd6e007e853cdd210e70a8a50c291e874175a4b78c9ee1fc46c1b63"
Oct 30 18:54:58.018065 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:58.018039    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-7784e5-openshift-default-564776f9f-zm98k"]
Oct 30 18:54:58.022969 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:58.022948    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-7784e5-openshift-default-564776f9f-zm98k"]
Oct 30 18:54:59.556861 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:54:59.556825    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97979488-7a76-4f4e-b330-49432c98bbec" path="/var/lib/kubelet/pods/97979488-7a76-4f4e-b330-49432c98bbec/volumes"
Oct 30 18:55:01.023578 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:01.023542    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-baa9e441-l4dxz"]
Oct 30 18:55:01.027988 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:01.027954    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-baa9e441-l4dxz"]
Oct 30 18:55:01.549753 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:01.549712    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4d5b030-8d02-4b18-9c8e-f47dacc8a72a" path="/var/lib/kubelet/pods/b4d5b030-8d02-4b18-9c8e-f47dacc8a72a/volumes"
Oct 30 18:55:03.605850 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:03.605804    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5468ba3e-2vtp9"]
Oct 30 18:55:03.606512 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:03.606489    4525 cpu_manager.go:414] "containerMap: removing container" podUID="a9ff1283-3fc7-463f-b3b7-fafdd56f2349" containerName="istio-proxy"
Oct 30 18:55:03.606512 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:03.606511    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9ff1283-3fc7-463f-b3b7-fafdd56f2349" containerName="istio-proxy"
Oct 30 18:55:03.606731 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:03.606525    4525 cpu_manager.go:414] "containerMap: removing container" podUID="0c983c51-cb90-47f2-8bc2-e2b7806df111" containerName="revoke-keys"
Oct 30 18:55:03.606731 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:03.606532    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c983c51-cb90-47f2-8bc2-e2b7806df111" containerName="revoke-keys"
Oct 30 18:55:03.606731 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:03.606544    4525 cpu_manager.go:414] "containerMap: removing container" podUID="eeccf097-40f0-440a-a90e-3b97186ef05f" containerName="revoke-keys"
Oct 30 18:55:03.606731 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:03.606551    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eeccf097-40f0-440a-a90e-3b97186ef05f" containerName="revoke-keys"
Oct 30 18:55:03.606731 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:03.606559    4525 cpu_manager.go:414] "containerMap: removing container" podUID="97979488-7a76-4f4e-b330-49432c98bbec" containerName="istio-proxy"
Oct 30 18:55:03.606731 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:03.606567    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97979488-7a76-4f4e-b330-49432c98bbec" containerName="istio-proxy"
Oct 30 18:55:03.606731 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:03.606592    4525 cpu_manager.go:414] "containerMap: removing container" podUID="cf796179-ce62-433b-87b8-a8bb1405c70b" containerName="istio-proxy"
Oct 30 18:55:03.606731 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:03.606598    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf796179-ce62-433b-87b8-a8bb1405c70b" containerName="istio-proxy"
Oct 30 18:55:03.606731 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:03.606637    4525 cpu_manager.go:414] "containerMap: removing container" podUID="13c9b518-512f-4a88-ae9e-cd8e9fb247cb" containerName="istio-proxy"
Oct 30 18:55:03.606731 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:03.606643    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13c9b518-512f-4a88-ae9e-cd8e9fb247cb" containerName="istio-proxy"
Oct 30 18:55:03.606731 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:03.606662    4525 cpu_manager.go:414] "containerMap: removing container" podUID="6fdcc41d-76be-4f42-9f6e-af06f8169b50" containerName="revoke-keys"
Oct 30 18:55:03.606731 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:03.606670    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fdcc41d-76be-4f42-9f6e-af06f8169b50" containerName="revoke-keys"
Oct 30 18:55:03.607314 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:03.606836    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="13c9b518-512f-4a88-ae9e-cd8e9fb247cb" containerName="istio-proxy"
Oct 30 18:55:03.607314 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:03.606852    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf796179-ce62-433b-87b8-a8bb1405c70b" containerName="istio-proxy"
Oct 30 18:55:03.607314 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:03.606863    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fdcc41d-76be-4f42-9f6e-af06f8169b50" containerName="revoke-keys"
Oct 30 18:55:03.607314 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:03.606875    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9ff1283-3fc7-463f-b3b7-fafdd56f2349" containerName="istio-proxy"
Oct 30 18:55:03.607314 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:03.606883    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c983c51-cb90-47f2-8bc2-e2b7806df111" containerName="revoke-keys"
Oct 30 18:55:03.607314 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:03.606892    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="eeccf097-40f0-440a-a90e-3b97186ef05f" containerName="revoke-keys"
Oct 30 18:55:03.607314 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:03.606900    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="97979488-7a76-4f4e-b330-49432c98bbec" containerName="istio-proxy"
Oct 30 18:55:03.607314 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:03.606910    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf796179-ce62-433b-87b8-a8bb1405c70b" containerName="istio-proxy"
Oct 30 18:55:03.607314 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:03.606920    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9ff1283-3fc7-463f-b3b7-fafdd56f2349" containerName="istio-proxy"
Oct 30 18:55:03.607314 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:03.606931    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="97979488-7a76-4f4e-b330-49432c98bbec" containerName="istio-proxy"
Oct 30 18:55:03.623725 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:03.612707    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5468ba3e-2vtp9"]
Oct 30 18:55:03.623725 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:03.612808    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5468ba3e-2vtp9"
Oct 30 18:55:03.623725 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:03.618870    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94hcn\""
Oct 30 18:55:03.623725 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:03.618911    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:55:03.712083 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:03.712045    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9rf6\" (UniqueName: \"kubernetes.io/projected/f698200d-7b66-4e30-9908-d622f2aa620c-kube-api-access-l9rf6\") pod \"maas-api-revoke-sub-5468ba3e-2vtp9\" (UID: \"f698200d-7b66-4e30-9908-d622f2aa620c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5468ba3e-2vtp9"
Oct 30 18:55:03.712250 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:03.712173    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f698200d-7b66-4e30-9908-d622f2aa620c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5468ba3e-2vtp9\" (UID: \"f698200d-7b66-4e30-9908-d622f2aa620c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5468ba3e-2vtp9"
Oct 30 18:55:03.712324 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:03.712311    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f698200d-7b66-4e30-9908-d622f2aa620c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5468ba3e-2vtp9\" (UID: \"f698200d-7b66-4e30-9908-d622f2aa620c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5468ba3e-2vtp9"
Oct 30 18:55:03.818529 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:03.818448    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9rf6\" (UniqueName: \"kubernetes.io/projected/f698200d-7b66-4e30-9908-d622f2aa620c-kube-api-access-l9rf6\") pod \"maas-api-revoke-sub-5468ba3e-2vtp9\" (UID: \"f698200d-7b66-4e30-9908-d622f2aa620c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5468ba3e-2vtp9"
Oct 30 18:55:03.820963 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:03.820932    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f698200d-7b66-4e30-9908-d622f2aa620c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5468ba3e-2vtp9\" (UID: \"f698200d-7b66-4e30-9908-d622f2aa620c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5468ba3e-2vtp9"
Oct 30 18:55:03.825421 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:03.825368    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f698200d-7b66-4e30-9908-d622f2aa620c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5468ba3e-2vtp9\" (UID: \"f698200d-7b66-4e30-9908-d622f2aa620c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5468ba3e-2vtp9"
Oct 30 18:55:03.842518 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:03.842434    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f698200d-7b66-4e30-9908-d622f2aa620c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5468ba3e-2vtp9\" (UID: \"f698200d-7b66-4e30-9908-d622f2aa620c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5468ba3e-2vtp9"
Oct 30 18:55:03.844001 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:03.843968    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f698200d-7b66-4e30-9908-d622f2aa620c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5468ba3e-2vtp9\" (UID: \"f698200d-7b66-4e30-9908-d622f2aa620c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5468ba3e-2vtp9"
Oct 30 18:55:03.848245 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:03.848216    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9rf6\" (UniqueName: \"kubernetes.io/projected/f698200d-7b66-4e30-9908-d622f2aa620c-kube-api-access-l9rf6\") pod \"maas-api-revoke-sub-5468ba3e-2vtp9\" (UID: \"f698200d-7b66-4e30-9908-d622f2aa620c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5468ba3e-2vtp9"
Oct 30 18:55:03.931623 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:03.931541    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5468ba3e-2vtp9"
Oct 30 18:55:04.222321 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:55:04.222291    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf698200d_7b66_4e30_9908_d622f2aa620c.slice/crio-6d7fb4a47ecd8bd9f695ece603edbb41e3de1a0fdc439a48df3469d6dd865ebd WatchSource:0}: Error finding container 6d7fb4a47ecd8bd9f695ece603edbb41e3de1a0fdc439a48df3469d6dd865ebd: Status 404 returned error can't find the container with id 6d7fb4a47ecd8bd9f695ece603edbb41e3de1a0fdc439a48df3469d6dd865ebd
Oct 30 18:55:04.223522 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:04.223500    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5468ba3e-2vtp9"]
Oct 30 18:55:05.057579 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:05.057546    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5468ba3e-2vtp9" event={"ID":"f698200d-7b66-4e30-9908-d622f2aa620c","Type":"ContainerStarted","Data":"0c58e4acbdb96c73029cfdc6fbba135f28079ebdbdc1ef94568faaa3a70eb860"}
Oct 30 18:55:05.058126 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:05.057609    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5468ba3e-2vtp9" event={"ID":"f698200d-7b66-4e30-9908-d622f2aa620c","Type":"ContainerStarted","Data":"6d7fb4a47ecd8bd9f695ece603edbb41e3de1a0fdc439a48df3469d6dd865ebd"}
Oct 30 18:55:05.075477 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:05.075430    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5468ba3e-2vtp9" podStartSLOduration=2.075416028 podStartE2EDuration="2.075416028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:55:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:55:05.074157552 +0000 UTC m=+1127.699827516" watchObservedRunningTime="2026-10-30 18:55:05.075416028 +0000 UTC m=+1127.701085991"
Oct 30 18:55:06.069659 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:06.069621    4525 generic.go:356] "Generic (PLEG): container finished" podID="f698200d-7b66-4e30-9908-d622f2aa620c" containerID="0c58e4acbdb96c73029cfdc6fbba135f28079ebdbdc1ef94568faaa3a70eb860" exitCode=0
Oct 30 18:55:06.070118 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:06.069706    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5468ba3e-2vtp9" event={"ID":"f698200d-7b66-4e30-9908-d622f2aa620c","Type":"ContainerDied","Data":"0c58e4acbdb96c73029cfdc6fbba135f28079ebdbdc1ef94568faaa3a70eb860"}
Oct 30 18:55:07.262723 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:07.262675    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5468ba3e-2vtp9"
Oct 30 18:55:07.399960 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:07.399923    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f698200d-7b66-4e30-9908-d622f2aa620c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f698200d-7b66-4e30-9908-d622f2aa620c-maas-api-xks-ca\") pod \"f698200d-7b66-4e30-9908-d622f2aa620c\" (UID: \"f698200d-7b66-4e30-9908-d622f2aa620c\") "
Oct 30 18:55:07.400155 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:07.400085    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f698200d-7b66-4e30-9908-d622f2aa620c-kube-api-access-l9rf6\" (UniqueName: \"kubernetes.io/projected/f698200d-7b66-4e30-9908-d622f2aa620c-kube-api-access-l9rf6\") pod \"f698200d-7b66-4e30-9908-d622f2aa620c\" (UID: \"f698200d-7b66-4e30-9908-d622f2aa620c\") "
Oct 30 18:55:07.400200 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:07.400182    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f698200d-7b66-4e30-9908-d622f2aa620c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f698200d-7b66-4e30-9908-d622f2aa620c-maas-api-openshift-ca\") pod \"f698200d-7b66-4e30-9908-d622f2aa620c\" (UID: \"f698200d-7b66-4e30-9908-d622f2aa620c\") "
Oct 30 18:55:07.400588 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:07.400558    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f698200d-7b66-4e30-9908-d622f2aa620c-maas-api-openshift-ca" pod "f698200d-7b66-4e30-9908-d622f2aa620c" (UID: "f698200d-7b66-4e30-9908-d622f2aa620c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:55:07.400865 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:07.400819    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f698200d-7b66-4e30-9908-d622f2aa620c-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:07.402374 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:07.402345    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f698200d-7b66-4e30-9908-d622f2aa620c-kube-api-access-l9rf6" pod "f698200d-7b66-4e30-9908-d622f2aa620c" (UID: "f698200d-7b66-4e30-9908-d622f2aa620c"). InnerVolumeSpecName "kube-api-access-l9rf6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:55:07.402374 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:07.402347    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f698200d-7b66-4e30-9908-d622f2aa620c-maas-api-xks-ca" pod "f698200d-7b66-4e30-9908-d622f2aa620c" (UID: "f698200d-7b66-4e30-9908-d622f2aa620c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:55:07.502481 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:07.502444    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f698200d-7b66-4e30-9908-d622f2aa620c-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:07.502481 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:07.502479    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9rf6\" (UniqueName: \"kubernetes.io/projected/f698200d-7b66-4e30-9908-d622f2aa620c-kube-api-access-l9rf6\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:08.138704 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:08.137775    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5468ba3e-2vtp9" event={"ID":"f698200d-7b66-4e30-9908-d622f2aa620c","Type":"ContainerDied","Data":"6d7fb4a47ecd8bd9f695ece603edbb41e3de1a0fdc439a48df3469d6dd865ebd"}
Oct 30 18:55:08.138704 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:08.137850    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d7fb4a47ecd8bd9f695ece603edbb41e3de1a0fdc439a48df3469d6dd865ebd"
Oct 30 18:55:08.138704 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:08.137922    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5468ba3e-2vtp9"
Oct 30 18:55:09.031372 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:09.031335    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb66e9c4-9kb7p"]
Oct 30 18:55:09.036336 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:09.036304    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb66e9c4-9kb7p"]
Oct 30 18:55:09.545600 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:09.545561    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d6c76d7-185c-49e5-8357-fbef4138d925" path="/var/lib/kubelet/pods/4d6c76d7-185c-49e5-8357-fbef4138d925/volumes"
Oct 30 18:55:17.398872 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:55:17.398825    4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d49l" containerName="extractor"
Oct 30 18:55:18.963555 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:18.963521    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-9982ad-openshift-default-6f478df445-gr62t"]
Oct 30 18:55:18.964410 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:18.964388    4525 cpu_manager.go:414] "containerMap: removing container" podUID="a9ff1283-3fc7-463f-b3b7-fafdd56f2349" containerName="istio-proxy"
Oct 30 18:55:18.964464 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:18.964415    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9ff1283-3fc7-463f-b3b7-fafdd56f2349" containerName="istio-proxy"
Oct 30 18:55:18.964464 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:18.964436    4525 cpu_manager.go:414] "containerMap: removing container" podUID="97979488-7a76-4f4e-b330-49432c98bbec" containerName="istio-proxy"
Oct 30 18:55:18.964464 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:18.964444    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97979488-7a76-4f4e-b330-49432c98bbec" containerName="istio-proxy"
Oct 30 18:55:18.965673 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:18.964472    4525 cpu_manager.go:414] "containerMap: removing container" podUID="cf796179-ce62-433b-87b8-a8bb1405c70b" containerName="istio-proxy"
Oct 30 18:55:18.965673 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:18.964480    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf796179-ce62-433b-87b8-a8bb1405c70b" containerName="istio-proxy"
Oct 30 18:55:18.965673 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:18.964501    4525 cpu_manager.go:414] "containerMap: removing container" podUID="f698200d-7b66-4e30-9908-d622f2aa620c" containerName="revoke-keys"
Oct 30 18:55:18.965673 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:18.964508    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f698200d-7b66-4e30-9908-d622f2aa620c" containerName="revoke-keys"
Oct 30 18:55:18.965673 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:18.964660    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="f698200d-7b66-4e30-9908-d622f2aa620c" containerName="revoke-keys"
Oct 30 18:55:18.976587 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:18.976386    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-9982ad-openshift-default-6f478df445-gr62t"]
Oct 30 18:55:18.976841 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:18.976549    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9982ad-openshift-default-6f478df445-gr62t"
Oct 30 18:55:18.986463 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:18.986443    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-9982ad-openshift-default-dockercfg-fn5t6\""
Oct 30 18:55:19.036778 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:19.036742    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10240f17-6wtp6"]
Oct 30 18:55:19.049014 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:19.048944    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10240f17-6wtp6"]
Oct 30 18:55:19.122704 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:19.115525    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed65886b-3abc-4b57-9206-8e0603411200-istio-podinfo\") pod \"e2e-api-b-9982ad-openshift-default-6f478df445-gr62t\" (UID: \"ed65886b-3abc-4b57-9206-8e0603411200\") " pod="openshift-ingress/e2e-api-b-9982ad-openshift-default-6f478df445-gr62t"
Oct 30 18:55:19.122704 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:19.115735    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed65886b-3abc-4b57-9206-8e0603411200-istio-envoy\") pod \"e2e-api-b-9982ad-openshift-default-6f478df445-gr62t\" (UID: \"ed65886b-3abc-4b57-9206-8e0603411200\") " pod="openshift-ingress/e2e-api-b-9982ad-openshift-default-6f478df445-gr62t"
Oct 30 18:55:19.122704 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:19.115782    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed65886b-3abc-4b57-9206-8e0603411200-istiod-ca-cert\") pod \"e2e-api-b-9982ad-openshift-default-6f478df445-gr62t\" (UID: \"ed65886b-3abc-4b57-9206-8e0603411200\") " pod="openshift-ingress/e2e-api-b-9982ad-openshift-default-6f478df445-gr62t"
Oct 30 18:55:19.122704 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:19.115818    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed65886b-3abc-4b57-9206-8e0603411200-istio-data\") pod \"e2e-api-b-9982ad-openshift-default-6f478df445-gr62t\" (UID: \"ed65886b-3abc-4b57-9206-8e0603411200\") " pod="openshift-ingress/e2e-api-b-9982ad-openshift-default-6f478df445-gr62t"
Oct 30 18:55:19.122704 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:19.115922    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd5fl\" (UniqueName: \"kubernetes.io/projected/ed65886b-3abc-4b57-9206-8e0603411200-kube-api-access-vd5fl\") pod \"e2e-api-b-9982ad-openshift-default-6f478df445-gr62t\" (UID: \"ed65886b-3abc-4b57-9206-8e0603411200\") " pod="openshift-ingress/e2e-api-b-9982ad-openshift-default-6f478df445-gr62t"
Oct 30 18:55:19.122704 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:19.115995    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed65886b-3abc-4b57-9206-8e0603411200-workload-certs\") pod \"e2e-api-b-9982ad-openshift-default-6f478df445-gr62t\" (UID: \"ed65886b-3abc-4b57-9206-8e0603411200\") " pod="openshift-ingress/e2e-api-b-9982ad-openshift-default-6f478df445-gr62t"
Oct 30 18:55:19.122704 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:19.116054    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed65886b-3abc-4b57-9206-8e0603411200-workload-socket\") pod \"e2e-api-b-9982ad-openshift-default-6f478df445-gr62t\" (UID: \"ed65886b-3abc-4b57-9206-8e0603411200\") " pod="openshift-ingress/e2e-api-b-9982ad-openshift-default-6f478df445-gr62t"
Oct 30 18:55:19.122704 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:19.116090    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed65886b-3abc-4b57-9206-8e0603411200-istio-token\") pod \"e2e-api-b-9982ad-openshift-default-6f478df445-gr62t\" (UID: \"ed65886b-3abc-4b57-9206-8e0603411200\") " pod="openshift-ingress/e2e-api-b-9982ad-openshift-default-6f478df445-gr62t"
Oct 30 18:55:19.122704 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:19.116134    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed65886b-3abc-4b57-9206-8e0603411200-credential-socket\") pod \"e2e-api-b-9982ad-openshift-default-6f478df445-gr62t\" (UID: \"ed65886b-3abc-4b57-9206-8e0603411200\") " pod="openshift-ingress/e2e-api-b-9982ad-openshift-default-6f478df445-gr62t"
Oct 30 18:55:19.220766 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:19.220508    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed65886b-3abc-4b57-9206-8e0603411200-workload-socket\") pod \"e2e-api-b-9982ad-openshift-default-6f478df445-gr62t\" (UID: \"ed65886b-3abc-4b57-9206-8e0603411200\") " pod="openshift-ingress/e2e-api-b-9982ad-openshift-default-6f478df445-gr62t"
Oct 30 18:55:19.220766 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:19.220575    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed65886b-3abc-4b57-9206-8e0603411200-istio-token\") pod \"e2e-api-b-9982ad-openshift-default-6f478df445-gr62t\" (UID: \"ed65886b-3abc-4b57-9206-8e0603411200\") " pod="openshift-ingress/e2e-api-b-9982ad-openshift-default-6f478df445-gr62t"
Oct 30 18:55:19.220766 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:19.220620    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed65886b-3abc-4b57-9206-8e0603411200-credential-socket\") pod \"e2e-api-b-9982ad-openshift-default-6f478df445-gr62t\" (UID: \"ed65886b-3abc-4b57-9206-8e0603411200\") " pod="openshift-ingress/e2e-api-b-9982ad-openshift-default-6f478df445-gr62t"
Oct 30 18:55:19.220766 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:19.220656    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed65886b-3abc-4b57-9206-8e0603411200-istio-podinfo\") pod \"e2e-api-b-9982ad-openshift-default-6f478df445-gr62t\" (UID: \"ed65886b-3abc-4b57-9206-8e0603411200\") " pod="openshift-ingress/e2e-api-b-9982ad-openshift-default-6f478df445-gr62t"
Oct 30 18:55:19.221122 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:19.220802    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed65886b-3abc-4b57-9206-8e0603411200-istio-envoy\") pod \"e2e-api-b-9982ad-openshift-default-6f478df445-gr62t\" (UID: \"ed65886b-3abc-4b57-9206-8e0603411200\") " pod="openshift-ingress/e2e-api-b-9982ad-openshift-default-6f478df445-gr62t"
Oct 30 18:55:19.221122 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:19.220839    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed65886b-3abc-4b57-9206-8e0603411200-istiod-ca-cert\") pod \"e2e-api-b-9982ad-openshift-default-6f478df445-gr62t\" (UID: \"ed65886b-3abc-4b57-9206-8e0603411200\") " pod="openshift-ingress/e2e-api-b-9982ad-openshift-default-6f478df445-gr62t"
Oct 30 18:55:19.221122 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:19.220879    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed65886b-3abc-4b57-9206-8e0603411200-istio-data\") pod \"e2e-api-b-9982ad-openshift-default-6f478df445-gr62t\" (UID: \"ed65886b-3abc-4b57-9206-8e0603411200\") " pod="openshift-ingress/e2e-api-b-9982ad-openshift-default-6f478df445-gr62t"
Oct 30 18:55:19.221122 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:19.220988    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd5fl\" (UniqueName: \"kubernetes.io/projected/ed65886b-3abc-4b57-9206-8e0603411200-kube-api-access-vd5fl\") pod \"e2e-api-b-9982ad-openshift-default-6f478df445-gr62t\" (UID: \"ed65886b-3abc-4b57-9206-8e0603411200\") " pod="openshift-ingress/e2e-api-b-9982ad-openshift-default-6f478df445-gr62t"
Oct 30 18:55:19.221122 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:19.221058    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed65886b-3abc-4b57-9206-8e0603411200-workload-certs\") pod \"e2e-api-b-9982ad-openshift-default-6f478df445-gr62t\" (UID: \"ed65886b-3abc-4b57-9206-8e0603411200\") " pod="openshift-ingress/e2e-api-b-9982ad-openshift-default-6f478df445-gr62t"
Oct 30 18:55:19.223722 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:19.221528    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed65886b-3abc-4b57-9206-8e0603411200-workload-certs\") pod \"e2e-api-b-9982ad-openshift-default-6f478df445-gr62t\" (UID: \"ed65886b-3abc-4b57-9206-8e0603411200\") " pod="openshift-ingress/e2e-api-b-9982ad-openshift-default-6f478df445-gr62t"
Oct 30 18:55:19.223722 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:19.221795    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed65886b-3abc-4b57-9206-8e0603411200-workload-socket\") pod \"e2e-api-b-9982ad-openshift-default-6f478df445-gr62t\" (UID: \"ed65886b-3abc-4b57-9206-8e0603411200\") " pod="openshift-ingress/e2e-api-b-9982ad-openshift-default-6f478df445-gr62t"
Oct 30 18:55:19.223722 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:19.222454    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed65886b-3abc-4b57-9206-8e0603411200-credential-socket\") pod \"e2e-api-b-9982ad-openshift-default-6f478df445-gr62t\" (UID: \"ed65886b-3abc-4b57-9206-8e0603411200\") " pod="openshift-ingress/e2e-api-b-9982ad-openshift-default-6f478df445-gr62t"
Oct 30 18:55:19.223722 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:19.223075    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed65886b-3abc-4b57-9206-8e0603411200-istio-data\") pod \"e2e-api-b-9982ad-openshift-default-6f478df445-gr62t\" (UID: \"ed65886b-3abc-4b57-9206-8e0603411200\") " pod="openshift-ingress/e2e-api-b-9982ad-openshift-default-6f478df445-gr62t"
Oct 30 18:55:19.226711 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:19.224229    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed65886b-3abc-4b57-9206-8e0603411200-istiod-ca-cert\") pod \"e2e-api-b-9982ad-openshift-default-6f478df445-gr62t\" (UID: \"ed65886b-3abc-4b57-9206-8e0603411200\") " pod="openshift-ingress/e2e-api-b-9982ad-openshift-default-6f478df445-gr62t"
Oct 30 18:55:19.254529 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:19.254497    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed65886b-3abc-4b57-9206-8e0603411200-istio-podinfo\") pod \"e2e-api-b-9982ad-openshift-default-6f478df445-gr62t\" (UID: \"ed65886b-3abc-4b57-9206-8e0603411200\") " pod="openshift-ingress/e2e-api-b-9982ad-openshift-default-6f478df445-gr62t"
Oct 30 18:55:19.256975 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:19.256949    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed65886b-3abc-4b57-9206-8e0603411200-istio-envoy\") pod \"e2e-api-b-9982ad-openshift-default-6f478df445-gr62t\" (UID: \"ed65886b-3abc-4b57-9206-8e0603411200\") " pod="openshift-ingress/e2e-api-b-9982ad-openshift-default-6f478df445-gr62t"
Oct 30 18:55:19.258660 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:19.258023    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed65886b-3abc-4b57-9206-8e0603411200-istio-token\") pod \"e2e-api-b-9982ad-openshift-default-6f478df445-gr62t\" (UID: \"ed65886b-3abc-4b57-9206-8e0603411200\") " pod="openshift-ingress/e2e-api-b-9982ad-openshift-default-6f478df445-gr62t"
Oct 30 18:55:19.260918 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:19.260896    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd5fl\" (UniqueName: \"kubernetes.io/projected/ed65886b-3abc-4b57-9206-8e0603411200-kube-api-access-vd5fl\") pod \"e2e-api-b-9982ad-openshift-default-6f478df445-gr62t\" (UID: \"ed65886b-3abc-4b57-9206-8e0603411200\") " pod="openshift-ingress/e2e-api-b-9982ad-openshift-default-6f478df445-gr62t"
Oct 30 18:55:19.301964 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:19.301535    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9982ad-openshift-default-6f478df445-gr62t"
Oct 30 18:55:19.584613 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:19.584539    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0df7654-ec74-41ac-adfc-bacec681ff36" path="/var/lib/kubelet/pods/d0df7654-ec74-41ac-adfc-bacec681ff36/volumes"
Oct 30 18:55:19.603840 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:55:19.603804    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded65886b_3abc_4b57_9206_8e0603411200.slice/crio-49658b08c9f13adb96358104bd4b7ddbd8f59d9918922281b2836fb4199e4a2e WatchSource:0}: Error finding container 49658b08c9f13adb96358104bd4b7ddbd8f59d9918922281b2836fb4199e4a2e: Status 404 returned error can't find the container with id 49658b08c9f13adb96358104bd4b7ddbd8f59d9918922281b2836fb4199e4a2e
Oct 30 18:55:19.605472 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:19.605450    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-9982ad-openshift-default-6f478df445-gr62t"]
Oct 30 18:55:19.611737 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:19.608473    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:55:19.611737 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:19.608558    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:55:19.611737 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:19.608590    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:55:20.266566 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:20.266534    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9982ad-openshift-default-6f478df445-gr62t" event={"ID":"ed65886b-3abc-4b57-9206-8e0603411200","Type":"ContainerStarted","Data":"68154ca9a57e6d106cc15aab5cacacf31c1c64634f8c4eba89fa726c0adcb905"}
Oct 30 18:55:20.266989 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:20.266584    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9982ad-openshift-default-6f478df445-gr62t" event={"ID":"ed65886b-3abc-4b57-9206-8e0603411200","Type":"ContainerStarted","Data":"49658b08c9f13adb96358104bd4b7ddbd8f59d9918922281b2836fb4199e4a2e"}
Oct 30 18:55:20.288861 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:20.288809    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-9982ad-openshift-default-6f478df445-gr62t" podStartSLOduration=2.288795141 podStartE2EDuration="2.288795141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:55:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:55:20.286663096 +0000 UTC m=+1142.912333060" watchObservedRunningTime="2026-10-30 18:55:20.288795141 +0000 UTC m=+1142.914465105"
Oct 30 18:55:20.302031 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:20.302006    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-9982ad-openshift-default-6f478df445-gr62t"
Oct 30 18:55:20.307699 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:20.307658    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-9982ad-openshift-default-6f478df445-gr62t"
Oct 30 18:55:21.275135 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:21.275107    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-9982ad-openshift-default-6f478df445-gr62t"
Oct 30 18:55:21.276044 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:21.276024    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-9982ad-openshift-default-6f478df445-gr62t"
Oct 30 18:55:26.705058 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:26.705021    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-da58b769-q44hh"]
Oct 30 18:55:26.709574 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:26.709555    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-da58b769-q44hh"
Oct 30 18:55:26.713818 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:26.713791    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:55:26.714152 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:26.714123    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94hcn\""
Oct 30 18:55:26.721485 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:26.721458    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-da58b769-q44hh"]
Oct 30 18:55:26.736118 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:26.736089    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66m6m\" (UniqueName: \"kubernetes.io/projected/e131d18d-909e-44be-bcc8-b42960212936-kube-api-access-66m6m\") pod \"maas-api-revoke-keys-e2e-mt-da58b769-q44hh\" (UID: \"e131d18d-909e-44be-bcc8-b42960212936\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-da58b769-q44hh"
Oct 30 18:55:26.736283 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:26.736265    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e131d18d-909e-44be-bcc8-b42960212936-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-da58b769-q44hh\" (UID: \"e131d18d-909e-44be-bcc8-b42960212936\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-da58b769-q44hh"
Oct 30 18:55:26.736432 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:26.736406    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e131d18d-909e-44be-bcc8-b42960212936-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-da58b769-q44hh\" (UID: \"e131d18d-909e-44be-bcc8-b42960212936\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-da58b769-q44hh"
Oct 30 18:55:26.839105 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:26.839063    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e131d18d-909e-44be-bcc8-b42960212936-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-da58b769-q44hh\" (UID: \"e131d18d-909e-44be-bcc8-b42960212936\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-da58b769-q44hh"
Oct 30 18:55:26.839344 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:26.839186    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e131d18d-909e-44be-bcc8-b42960212936-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-da58b769-q44hh\" (UID: \"e131d18d-909e-44be-bcc8-b42960212936\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-da58b769-q44hh"
Oct 30 18:55:26.839344 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:26.839311    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66m6m\" (UniqueName: \"kubernetes.io/projected/e131d18d-909e-44be-bcc8-b42960212936-kube-api-access-66m6m\") pod \"maas-api-revoke-keys-e2e-mt-da58b769-q44hh\" (UID: \"e131d18d-909e-44be-bcc8-b42960212936\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-da58b769-q44hh"
Oct 30 18:55:26.840174 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:26.840146    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e131d18d-909e-44be-bcc8-b42960212936-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-da58b769-q44hh\" (UID: \"e131d18d-909e-44be-bcc8-b42960212936\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-da58b769-q44hh"
Oct 30 18:55:26.841923 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:26.841897    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e131d18d-909e-44be-bcc8-b42960212936-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-da58b769-q44hh\" (UID: \"e131d18d-909e-44be-bcc8-b42960212936\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-da58b769-q44hh"
Oct 30 18:55:26.847639 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:26.847615    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66m6m\" (UniqueName: \"kubernetes.io/projected/e131d18d-909e-44be-bcc8-b42960212936-kube-api-access-66m6m\") pod \"maas-api-revoke-keys-e2e-mt-da58b769-q44hh\" (UID: \"e131d18d-909e-44be-bcc8-b42960212936\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-da58b769-q44hh"
Oct 30 18:55:27.048002 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:27.047902    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-da58b769-q44hh"
Oct 30 18:55:27.341858 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:27.341823    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-da58b769-q44hh"]
Oct 30 18:55:27.342317 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:55:27.342295    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode131d18d_909e_44be_bcc8_b42960212936.slice/crio-a30d1fd75bbf5d6ec01dac8ff1a0353236cd951ed9bcc03919f5f67d0499c868 WatchSource:0}: Error finding container a30d1fd75bbf5d6ec01dac8ff1a0353236cd951ed9bcc03919f5f67d0499c868: Status 404 returned error can't find the container with id a30d1fd75bbf5d6ec01dac8ff1a0353236cd951ed9bcc03919f5f67d0499c868
Oct 30 18:55:28.361714 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:28.359648    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-da58b769-q44hh" event={"ID":"e131d18d-909e-44be-bcc8-b42960212936","Type":"ContainerStarted","Data":"5ea4ffe143407064eef7cfa43bdd3d88f589ab13edae31e3eb6a7c3bb2d2846b"}
Oct 30 18:55:28.361714 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:28.359739    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-da58b769-q44hh" event={"ID":"e131d18d-909e-44be-bcc8-b42960212936","Type":"ContainerStarted","Data":"a30d1fd75bbf5d6ec01dac8ff1a0353236cd951ed9bcc03919f5f67d0499c868"}
Oct 30 18:55:28.377291 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:28.377231    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-da58b769-q44hh" podStartSLOduration=2.377212733 podStartE2EDuration="2.377212733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:55:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:55:28.375979692 +0000 UTC m=+1151.001649658" watchObservedRunningTime="2026-10-30 18:55:28.377212733 +0000 UTC m=+1151.002882699"
Oct 30 18:55:29.369107 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:29.369076    4525 generic.go:356] "Generic (PLEG): container finished" podID="e131d18d-909e-44be-bcc8-b42960212936" containerID="5ea4ffe143407064eef7cfa43bdd3d88f589ab13edae31e3eb6a7c3bb2d2846b" exitCode=0
Oct 30 18:55:29.369532 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:29.369157    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-da58b769-q44hh" event={"ID":"e131d18d-909e-44be-bcc8-b42960212936","Type":"ContainerDied","Data":"5ea4ffe143407064eef7cfa43bdd3d88f589ab13edae31e3eb6a7c3bb2d2846b"}
Oct 30 18:55:30.531087 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:30.531063    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-da58b769-q44hh"
Oct 30 18:55:30.614296 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:30.613559    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e131d18d-909e-44be-bcc8-b42960212936-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e131d18d-909e-44be-bcc8-b42960212936-maas-api-xks-ca\") pod \"e131d18d-909e-44be-bcc8-b42960212936\" (UID: \"e131d18d-909e-44be-bcc8-b42960212936\") "
Oct 30 18:55:30.614296 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:30.613787    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e131d18d-909e-44be-bcc8-b42960212936-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e131d18d-909e-44be-bcc8-b42960212936-maas-api-openshift-ca\") pod \"e131d18d-909e-44be-bcc8-b42960212936\" (UID: \"e131d18d-909e-44be-bcc8-b42960212936\") "
Oct 30 18:55:30.614296 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:30.613906    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e131d18d-909e-44be-bcc8-b42960212936-kube-api-access-66m6m\" (UniqueName: \"kubernetes.io/projected/e131d18d-909e-44be-bcc8-b42960212936-kube-api-access-66m6m\") pod \"e131d18d-909e-44be-bcc8-b42960212936\" (UID: \"e131d18d-909e-44be-bcc8-b42960212936\") "
Oct 30 18:55:30.622702 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:30.622654    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e131d18d-909e-44be-bcc8-b42960212936-maas-api-openshift-ca" pod "e131d18d-909e-44be-bcc8-b42960212936" (UID: "e131d18d-909e-44be-bcc8-b42960212936"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:55:30.630546 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:30.630428    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e131d18d-909e-44be-bcc8-b42960212936-maas-api-xks-ca" pod "e131d18d-909e-44be-bcc8-b42960212936" (UID: "e131d18d-909e-44be-bcc8-b42960212936"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:55:30.636151 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:30.636120    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e131d18d-909e-44be-bcc8-b42960212936-kube-api-access-66m6m" pod "e131d18d-909e-44be-bcc8-b42960212936" (UID: "e131d18d-909e-44be-bcc8-b42960212936"). InnerVolumeSpecName "kube-api-access-66m6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:55:30.717383 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:30.717346    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e131d18d-909e-44be-bcc8-b42960212936-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:30.717383 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:30.717383    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e131d18d-909e-44be-bcc8-b42960212936-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:30.717582 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:30.717397    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66m6m\" (UniqueName: \"kubernetes.io/projected/e131d18d-909e-44be-bcc8-b42960212936-kube-api-access-66m6m\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:31.388857 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:31.388825    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-da58b769-q44hh"
Oct 30 18:55:31.388857 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:31.388848    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-da58b769-q44hh" event={"ID":"e131d18d-909e-44be-bcc8-b42960212936","Type":"ContainerDied","Data":"a30d1fd75bbf5d6ec01dac8ff1a0353236cd951ed9bcc03919f5f67d0499c868"}
Oct 30 18:55:31.389120 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:31.388921    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a30d1fd75bbf5d6ec01dac8ff1a0353236cd951ed9bcc03919f5f67d0499c868"
Oct 30 18:55:32.918468 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:32.918327    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9"]
Oct 30 18:55:32.919420 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:32.919395    4525 cpu_manager.go:414] "containerMap: removing container" podUID="e131d18d-909e-44be-bcc8-b42960212936" containerName="revoke-keys"
Oct 30 18:55:32.919420 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:32.919422    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e131d18d-909e-44be-bcc8-b42960212936" containerName="revoke-keys"
Oct 30 18:55:32.919627 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:32.919612    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="e131d18d-909e-44be-bcc8-b42960212936" containerName="revoke-keys"
Oct 30 18:55:32.924419 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:32.924396    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9"
Oct 30 18:55:32.931105 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:32.931080    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9"]
Oct 30 18:55:32.931908 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:32.931887    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-93574a-openshift-default-dockercfg-d46c7\""
Oct 30 18:55:33.062705 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:33.057544    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1-workload-socket\") pod \"e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9\" (UID: \"5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1\") " pod="openshift-ingress/e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9"
Oct 30 18:55:33.062705 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:33.057631    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1-istio-podinfo\") pod \"e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9\" (UID: \"5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1\") " pod="openshift-ingress/e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9"
Oct 30 18:55:33.062705 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:33.057671    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1-istio-token\") pod \"e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9\" (UID: \"5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1\") " pod="openshift-ingress/e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9"
Oct 30 18:55:33.062705 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:33.057790    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1-credential-socket\") pod \"e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9\" (UID: \"5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1\") " pod="openshift-ingress/e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9"
Oct 30 18:55:33.062705 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:33.057817    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1-istiod-ca-cert\") pod \"e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9\" (UID: \"5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1\") " pod="openshift-ingress/e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9"
Oct 30 18:55:33.062705 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:33.057847    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1-istio-envoy\") pod \"e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9\" (UID: \"5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1\") " pod="openshift-ingress/e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9"
Oct 30 18:55:33.062705 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:33.057879    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5kxs\" (UniqueName: \"kubernetes.io/projected/5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1-kube-api-access-h5kxs\") pod \"e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9\" (UID: \"5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1\") " pod="openshift-ingress/e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9"
Oct 30 18:55:33.062705 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:33.057909    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1-istio-data\") pod \"e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9\" (UID: \"5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1\") " pod="openshift-ingress/e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9"
Oct 30 18:55:33.062705 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:33.058067    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1-workload-certs\") pod \"e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9\" (UID: \"5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1\") " pod="openshift-ingress/e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9"
Oct 30 18:55:33.173753 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:33.168874    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1-istio-podinfo\") pod \"e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9\" (UID: \"5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1\") " pod="openshift-ingress/e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9"
Oct 30 18:55:33.173753 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:33.168982    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1-istio-token\") pod \"e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9\" (UID: \"5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1\") " pod="openshift-ingress/e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9"
Oct 30 18:55:33.173753 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:33.169075    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1-credential-socket\") pod \"e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9\" (UID: \"5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1\") " pod="openshift-ingress/e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9"
Oct 30 18:55:33.173753 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:33.169103    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1-istiod-ca-cert\") pod \"e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9\" (UID: \"5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1\") " pod="openshift-ingress/e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9"
Oct 30 18:55:33.173753 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:33.169148    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1-istio-envoy\") pod \"e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9\" (UID: \"5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1\") " pod="openshift-ingress/e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9"
Oct 30 18:55:33.173753 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:33.169184    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5kxs\" (UniqueName: \"kubernetes.io/projected/5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1-kube-api-access-h5kxs\") pod \"e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9\" (UID: \"5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1\") " pod="openshift-ingress/e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9"
Oct 30 18:55:33.173753 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:33.169217    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1-istio-data\") pod \"e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9\" (UID: \"5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1\") " pod="openshift-ingress/e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9"
Oct 30 18:55:33.173753 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:33.169367    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1-workload-certs\") pod \"e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9\" (UID: \"5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1\") " pod="openshift-ingress/e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9"
Oct 30 18:55:33.173753 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:33.169467    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1-workload-socket\") pod \"e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9\" (UID: \"5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1\") " pod="openshift-ingress/e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9"
Oct 30 18:55:33.173753 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:33.170010    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1-workload-socket\") pod \"e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9\" (UID: \"5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1\") " pod="openshift-ingress/e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9"
Oct 30 18:55:33.173753 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:33.171316    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1-istio-data\") pod \"e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9\" (UID: \"5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1\") " pod="openshift-ingress/e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9"
Oct 30 18:55:33.173753 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:33.171770    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1-workload-certs\") pod \"e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9\" (UID: \"5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1\") " pod="openshift-ingress/e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9"
Oct 30 18:55:33.173753 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:33.172424    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1-istiod-ca-cert\") pod \"e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9\" (UID: \"5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1\") " pod="openshift-ingress/e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9"
Oct 30 18:55:33.173753 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:33.172661    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1-credential-socket\") pod \"e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9\" (UID: \"5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1\") " pod="openshift-ingress/e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9"
Oct 30 18:55:33.175956 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:33.175930    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1-istio-envoy\") pod \"e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9\" (UID: \"5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1\") " pod="openshift-ingress/e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9"
Oct 30 18:55:33.192197 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:33.187730    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5kxs\" (UniqueName: \"kubernetes.io/projected/5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1-kube-api-access-h5kxs\") pod \"e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9\" (UID: \"5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1\") " pod="openshift-ingress/e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9"
Oct 30 18:55:33.193453 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:33.193425    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1-istio-token\") pod \"e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9\" (UID: \"5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1\") " pod="openshift-ingress/e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9"
Oct 30 18:55:33.200750 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:33.197158    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1-istio-podinfo\") pod \"e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9\" (UID: \"5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1\") " pod="openshift-ingress/e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9"
Oct 30 18:55:33.245261 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:33.245222    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9"
Oct 30 18:55:33.553750 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:55:33.553717    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c7894e8_ae3d_4cf3_b7db_5597ec2b36a1.slice/crio-82a4a3cfc26a03af234e7b38d7e1735e0502e3f5b365bf8be4007c5ea4d67932 WatchSource:0}: Error finding container 82a4a3cfc26a03af234e7b38d7e1735e0502e3f5b365bf8be4007c5ea4d67932: Status 404 returned error can't find the container with id 82a4a3cfc26a03af234e7b38d7e1735e0502e3f5b365bf8be4007c5ea4d67932
Oct 30 18:55:33.555864 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:33.555848    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9"]
Oct 30 18:55:33.556779 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:33.556750    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:55:33.556834 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:33.556812    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:55:33.556869 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:33.556840    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:55:34.450136 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:34.450091    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9" event={"ID":"5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1","Type":"ContainerStarted","Data":"c18a57044ec8e94244aacce3bb29a64cfdd8474ebd9781aa20b51a70a84298c6"}
Oct 30 18:55:34.450862 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:34.450344    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9" event={"ID":"5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1","Type":"ContainerStarted","Data":"82a4a3cfc26a03af234e7b38d7e1735e0502e3f5b365bf8be4007c5ea4d67932"}
Oct 30 18:55:34.459693 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:34.459626    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9" podStartSLOduration=2.459606803 podStartE2EDuration="2.459606803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:55:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:55:34.459182331 +0000 UTC m=+1157.084852298" watchObservedRunningTime="2026-10-30 18:55:34.459606803 +0000 UTC m=+1157.085276796"
Oct 30 18:55:35.246350 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:35.246318    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9"
Oct 30 18:55:35.251718 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:35.251658    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9"
Oct 30 18:55:35.458939 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:35.458901    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9"
Oct 30 18:55:35.459752 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:35.459727    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9"
Oct 30 18:55:37.361880 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:37.361851    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-635398a3-h52fw"]
Oct 30 18:55:37.366209 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:37.366186    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-635398a3-h52fw"
Oct 30 18:55:37.371077 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:37.371056    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:55:37.371490 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:37.371466    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94hcn\""
Oct 30 18:55:37.377608 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:37.377242    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-635398a3-h52fw"]
Oct 30 18:55:37.458885 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:37.458851    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac4b3887-d485-4fdc-848d-b9dff0adf0b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-635398a3-h52fw\" (UID: \"ac4b3887-d485-4fdc-848d-b9dff0adf0b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-635398a3-h52fw"
Oct 30 18:55:37.459065 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:37.458932    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9sgd\" (UniqueName: \"kubernetes.io/projected/ac4b3887-d485-4fdc-848d-b9dff0adf0b0-kube-api-access-z9sgd\") pod \"maas-api-revoke-sub-635398a3-h52fw\" (UID: \"ac4b3887-d485-4fdc-848d-b9dff0adf0b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-635398a3-h52fw"
Oct 30 18:55:37.459136 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:37.459063    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac4b3887-d485-4fdc-848d-b9dff0adf0b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-635398a3-h52fw\" (UID: \"ac4b3887-d485-4fdc-848d-b9dff0adf0b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-635398a3-h52fw"
Oct 30 18:55:37.582711 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:37.580722    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac4b3887-d485-4fdc-848d-b9dff0adf0b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-635398a3-h52fw\" (UID: \"ac4b3887-d485-4fdc-848d-b9dff0adf0b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-635398a3-h52fw"
Oct 30 18:55:37.582711 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:37.580825    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9sgd\" (UniqueName: \"kubernetes.io/projected/ac4b3887-d485-4fdc-848d-b9dff0adf0b0-kube-api-access-z9sgd\") pod \"maas-api-revoke-sub-635398a3-h52fw\" (UID: \"ac4b3887-d485-4fdc-848d-b9dff0adf0b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-635398a3-h52fw"
Oct 30 18:55:37.639364 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:37.580900    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac4b3887-d485-4fdc-848d-b9dff0adf0b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-635398a3-h52fw\" (UID: \"ac4b3887-d485-4fdc-848d-b9dff0adf0b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-635398a3-h52fw"
Oct 30 18:55:37.639802 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:37.606819    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac4b3887-d485-4fdc-848d-b9dff0adf0b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-635398a3-h52fw\" (UID: \"ac4b3887-d485-4fdc-848d-b9dff0adf0b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-635398a3-h52fw"
Oct 30 18:55:37.681857 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:37.681827    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac4b3887-d485-4fdc-848d-b9dff0adf0b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-635398a3-h52fw\" (UID: \"ac4b3887-d485-4fdc-848d-b9dff0adf0b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-635398a3-h52fw"
Oct 30 18:55:37.686605 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:37.686582    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9sgd\" (UniqueName: \"kubernetes.io/projected/ac4b3887-d485-4fdc-848d-b9dff0adf0b0-kube-api-access-z9sgd\") pod \"maas-api-revoke-sub-635398a3-h52fw\" (UID: \"ac4b3887-d485-4fdc-848d-b9dff0adf0b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-635398a3-h52fw"
Oct 30 18:55:37.693953 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:37.693923    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-635398a3-h52fw"
Oct 30 18:55:38.021607 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:55:38.021566    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac4b3887_d485_4fdc_848d_b9dff0adf0b0.slice/crio-b75316e0971079f045a3cecd3f703948021385b516bdaae59c02b2894e0f757e WatchSource:0}: Error finding container b75316e0971079f045a3cecd3f703948021385b516bdaae59c02b2894e0f757e: Status 404 returned error can't find the container with id b75316e0971079f045a3cecd3f703948021385b516bdaae59c02b2894e0f757e
Oct 30 18:55:38.022114 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:38.022089    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-635398a3-h52fw"]
Oct 30 18:55:38.624526 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:38.624493    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-635398a3-h52fw" event={"ID":"ac4b3887-d485-4fdc-848d-b9dff0adf0b0","Type":"ContainerStarted","Data":"9e0f03f6326cc853b605222094b1d29ab448b78977eb409a393a6d19402c9c55"}
Oct 30 18:55:38.624963 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:38.624546    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-635398a3-h52fw" event={"ID":"ac4b3887-d485-4fdc-848d-b9dff0adf0b0","Type":"ContainerStarted","Data":"b75316e0971079f045a3cecd3f703948021385b516bdaae59c02b2894e0f757e"}
Oct 30 18:55:38.644519 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:38.644434    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-635398a3-h52fw" podStartSLOduration=1.644419991 podStartE2EDuration="1.644419991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:55:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:55:38.640401867 +0000 UTC m=+1161.266071830" watchObservedRunningTime="2026-10-30 18:55:38.644419991 +0000 UTC m=+1161.270089955"
Oct 30 18:55:38.862532 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:38.862495    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx"]
Oct 30 18:55:38.872236 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:38.872208    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx"
Oct 30 18:55:38.875723 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:38.875698    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx"]
Oct 30 18:55:38.896169 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:38.896095    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-040d9c-openshift-default-dockercfg-qqj9q\""
Oct 30 18:55:38.983219 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:38.983188    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d4927ad-ead6-477f-8105-4cdd3daadea5-istiod-ca-cert\") pod \"e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx\" (UID: \"2d4927ad-ead6-477f-8105-4cdd3daadea5\") " pod="openshift-ingress/e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx"
Oct 30 18:55:38.983370 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:38.983226    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d4927ad-ead6-477f-8105-4cdd3daadea5-istio-token\") pod \"e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx\" (UID: \"2d4927ad-ead6-477f-8105-4cdd3daadea5\") " pod="openshift-ingress/e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx"
Oct 30 18:55:38.983370 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:38.983357    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d4927ad-ead6-477f-8105-4cdd3daadea5-istio-envoy\") pod \"e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx\" (UID: \"2d4927ad-ead6-477f-8105-4cdd3daadea5\") " pod="openshift-ingress/e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx"
Oct 30 18:55:38.983457 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:38.983401    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d4927ad-ead6-477f-8105-4cdd3daadea5-istio-podinfo\") pod \"e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx\" (UID: \"2d4927ad-ead6-477f-8105-4cdd3daadea5\") " pod="openshift-ingress/e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx"
Oct 30 18:55:38.983548 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:38.983535    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d4927ad-ead6-477f-8105-4cdd3daadea5-workload-socket\") pod \"e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx\" (UID: \"2d4927ad-ead6-477f-8105-4cdd3daadea5\") " pod="openshift-ingress/e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx"
Oct 30 18:55:38.983589 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:38.983555    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d4927ad-ead6-477f-8105-4cdd3daadea5-workload-certs\") pod \"e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx\" (UID: \"2d4927ad-ead6-477f-8105-4cdd3daadea5\") " pod="openshift-ingress/e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx"
Oct 30 18:55:38.983704 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:38.983665    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsmjp\" (UniqueName: \"kubernetes.io/projected/2d4927ad-ead6-477f-8105-4cdd3daadea5-kube-api-access-tsmjp\") pod \"e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx\" (UID: \"2d4927ad-ead6-477f-8105-4cdd3daadea5\") " pod="openshift-ingress/e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx"
Oct 30 18:55:38.983818 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:38.983738    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d4927ad-ead6-477f-8105-4cdd3daadea5-istio-data\") pod \"e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx\" (UID: \"2d4927ad-ead6-477f-8105-4cdd3daadea5\") " pod="openshift-ingress/e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx"
Oct 30 18:55:38.983818 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:38.983782    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d4927ad-ead6-477f-8105-4cdd3daadea5-credential-socket\") pod \"e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx\" (UID: \"2d4927ad-ead6-477f-8105-4cdd3daadea5\") " pod="openshift-ingress/e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx"
Oct 30 18:55:39.085474 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:39.085433    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d4927ad-ead6-477f-8105-4cdd3daadea5-istio-token\") pod \"e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx\" (UID: \"2d4927ad-ead6-477f-8105-4cdd3daadea5\") " pod="openshift-ingress/e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx"
Oct 30 18:55:39.085670 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:39.085536    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d4927ad-ead6-477f-8105-4cdd3daadea5-istio-envoy\") pod \"e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx\" (UID: \"2d4927ad-ead6-477f-8105-4cdd3daadea5\") " pod="openshift-ingress/e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx"
Oct 30 18:55:39.085670 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:39.085574    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d4927ad-ead6-477f-8105-4cdd3daadea5-istio-podinfo\") pod \"e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx\" (UID: \"2d4927ad-ead6-477f-8105-4cdd3daadea5\") " pod="openshift-ingress/e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx"
Oct 30 18:55:39.085670 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:39.085660    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d4927ad-ead6-477f-8105-4cdd3daadea5-workload-socket\") pod \"e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx\" (UID: \"2d4927ad-ead6-477f-8105-4cdd3daadea5\") " pod="openshift-ingress/e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx"
Oct 30 18:55:39.085901 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:39.085725    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d4927ad-ead6-477f-8105-4cdd3daadea5-workload-certs\") pod \"e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx\" (UID: \"2d4927ad-ead6-477f-8105-4cdd3daadea5\") " pod="openshift-ingress/e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx"
Oct 30 18:55:39.085901 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:39.085794    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsmjp\" (UniqueName: \"kubernetes.io/projected/2d4927ad-ead6-477f-8105-4cdd3daadea5-kube-api-access-tsmjp\") pod \"e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx\" (UID: \"2d4927ad-ead6-477f-8105-4cdd3daadea5\") " pod="openshift-ingress/e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx"
Oct 30 18:55:39.085901 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:39.085825    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d4927ad-ead6-477f-8105-4cdd3daadea5-istio-data\") pod \"e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx\" (UID: \"2d4927ad-ead6-477f-8105-4cdd3daadea5\") " pod="openshift-ingress/e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx"
Oct 30 18:55:39.085901 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:39.085878    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d4927ad-ead6-477f-8105-4cdd3daadea5-credential-socket\") pod \"e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx\" (UID: \"2d4927ad-ead6-477f-8105-4cdd3daadea5\") " pod="openshift-ingress/e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx"
Oct 30 18:55:39.086058 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:39.085963    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d4927ad-ead6-477f-8105-4cdd3daadea5-istiod-ca-cert\") pod \"e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx\" (UID: \"2d4927ad-ead6-477f-8105-4cdd3daadea5\") " pod="openshift-ingress/e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx"
Oct 30 18:55:39.086207 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:39.086181    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d4927ad-ead6-477f-8105-4cdd3daadea5-workload-certs\") pod \"e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx\" (UID: \"2d4927ad-ead6-477f-8105-4cdd3daadea5\") " pod="openshift-ingress/e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx"
Oct 30 18:55:39.086283 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:39.086261    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d4927ad-ead6-477f-8105-4cdd3daadea5-istio-data\") pod \"e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx\" (UID: \"2d4927ad-ead6-477f-8105-4cdd3daadea5\") " pod="openshift-ingress/e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx"
Oct 30 18:55:39.086343 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:39.086297    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d4927ad-ead6-477f-8105-4cdd3daadea5-credential-socket\") pod \"e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx\" (UID: \"2d4927ad-ead6-477f-8105-4cdd3daadea5\") " pod="openshift-ingress/e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx"
Oct 30 18:55:39.086343 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:39.086325    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d4927ad-ead6-477f-8105-4cdd3daadea5-workload-socket\") pod \"e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx\" (UID: \"2d4927ad-ead6-477f-8105-4cdd3daadea5\") " pod="openshift-ingress/e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx"
Oct 30 18:55:39.086713 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:39.086661    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d4927ad-ead6-477f-8105-4cdd3daadea5-istiod-ca-cert\") pod \"e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx\" (UID: \"2d4927ad-ead6-477f-8105-4cdd3daadea5\") " pod="openshift-ingress/e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx"
Oct 30 18:55:39.088186 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:39.088164    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d4927ad-ead6-477f-8105-4cdd3daadea5-istio-envoy\") pod \"e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx\" (UID: \"2d4927ad-ead6-477f-8105-4cdd3daadea5\") " pod="openshift-ingress/e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx"
Oct 30 18:55:39.088475 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:39.088460    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d4927ad-ead6-477f-8105-4cdd3daadea5-istio-podinfo\") pod \"e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx\" (UID: \"2d4927ad-ead6-477f-8105-4cdd3daadea5\") " pod="openshift-ingress/e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx"
Oct 30 18:55:39.095209 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:39.095179    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsmjp\" (UniqueName: \"kubernetes.io/projected/2d4927ad-ead6-477f-8105-4cdd3daadea5-kube-api-access-tsmjp\") pod \"e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx\" (UID: \"2d4927ad-ead6-477f-8105-4cdd3daadea5\") " pod="openshift-ingress/e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx"
Oct 30 18:55:39.095809 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:39.095764    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d4927ad-ead6-477f-8105-4cdd3daadea5-istio-token\") pod \"e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx\" (UID: \"2d4927ad-ead6-477f-8105-4cdd3daadea5\") " pod="openshift-ingress/e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx"
Oct 30 18:55:39.213845 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:39.204112    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx"
Oct 30 18:55:39.605171 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:55:39.605140    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d4927ad_ead6_477f_8105_4cdd3daadea5.slice/crio-d540e119762d2f4a3c87a7147d05c6e0a6acad805352038d834f68a9d5e5045d WatchSource:0}: Error finding container d540e119762d2f4a3c87a7147d05c6e0a6acad805352038d834f68a9d5e5045d: Status 404 returned error can't find the container with id d540e119762d2f4a3c87a7147d05c6e0a6acad805352038d834f68a9d5e5045d
Oct 30 18:55:39.605908 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:39.605882    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx"]
Oct 30 18:55:39.608206 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:39.608176    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:55:39.608338 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:39.608240    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:55:39.608338 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:39.608275    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:55:39.635137 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:39.635108    4525 generic.go:356] "Generic (PLEG): container finished" podID="ac4b3887-d485-4fdc-848d-b9dff0adf0b0" containerID="9e0f03f6326cc853b605222094b1d29ab448b78977eb409a393a6d19402c9c55" exitCode=0
Oct 30 18:55:39.635527 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:39.635192    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-635398a3-h52fw" event={"ID":"ac4b3887-d485-4fdc-848d-b9dff0adf0b0","Type":"ContainerDied","Data":"9e0f03f6326cc853b605222094b1d29ab448b78977eb409a393a6d19402c9c55"}
Oct 30 18:55:39.636648 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:39.636626    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx" event={"ID":"2d4927ad-ead6-477f-8105-4cdd3daadea5","Type":"ContainerStarted","Data":"d540e119762d2f4a3c87a7147d05c6e0a6acad805352038d834f68a9d5e5045d"}
Oct 30 18:55:40.647428 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:40.647390    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx" event={"ID":"2d4927ad-ead6-477f-8105-4cdd3daadea5","Type":"ContainerStarted","Data":"7024b7c83690f48ca73015a8dcfe2ab56c7816c4e0b933f9c0cff271502866ac"}
Oct 30 18:55:40.683110 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:40.683026    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx" podStartSLOduration=2.6830044060000002 podStartE2EDuration="2.683004406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:55:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:55:40.667111524 +0000 UTC m=+1163.292781492" watchObservedRunningTime="2026-10-30 18:55:40.683004406 +0000 UTC m=+1163.308674433"
Oct 30 18:55:40.852642 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:40.852603    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-635398a3-h52fw"
Oct 30 18:55:40.893273 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:40.893239    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac4b3887-d485-4fdc-848d-b9dff0adf0b0-kube-api-access-z9sgd\" (UniqueName: \"kubernetes.io/projected/ac4b3887-d485-4fdc-848d-b9dff0adf0b0-kube-api-access-z9sgd\") pod \"ac4b3887-d485-4fdc-848d-b9dff0adf0b0\" (UID: \"ac4b3887-d485-4fdc-848d-b9dff0adf0b0\") "
Oct 30 18:55:40.893465 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:40.893301    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac4b3887-d485-4fdc-848d-b9dff0adf0b0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac4b3887-d485-4fdc-848d-b9dff0adf0b0-maas-api-xks-ca\") pod \"ac4b3887-d485-4fdc-848d-b9dff0adf0b0\" (UID: \"ac4b3887-d485-4fdc-848d-b9dff0adf0b0\") "
Oct 30 18:55:40.893465 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:40.893456    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac4b3887-d485-4fdc-848d-b9dff0adf0b0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac4b3887-d485-4fdc-848d-b9dff0adf0b0-maas-api-openshift-ca\") pod \"ac4b3887-d485-4fdc-848d-b9dff0adf0b0\" (UID: \"ac4b3887-d485-4fdc-848d-b9dff0adf0b0\") "
Oct 30 18:55:40.894352 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:40.893885    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac4b3887-d485-4fdc-848d-b9dff0adf0b0-maas-api-openshift-ca" pod "ac4b3887-d485-4fdc-848d-b9dff0adf0b0" (UID: "ac4b3887-d485-4fdc-848d-b9dff0adf0b0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:55:40.894352 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:40.894316    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac4b3887-d485-4fdc-848d-b9dff0adf0b0-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:40.895903 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:40.895867    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac4b3887-d485-4fdc-848d-b9dff0adf0b0-maas-api-xks-ca" pod "ac4b3887-d485-4fdc-848d-b9dff0adf0b0" (UID: "ac4b3887-d485-4fdc-848d-b9dff0adf0b0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:55:40.896421 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:40.896358    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac4b3887-d485-4fdc-848d-b9dff0adf0b0-kube-api-access-z9sgd" pod "ac4b3887-d485-4fdc-848d-b9dff0adf0b0" (UID: "ac4b3887-d485-4fdc-848d-b9dff0adf0b0"). InnerVolumeSpecName "kube-api-access-z9sgd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:55:40.995877 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:40.995836    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9sgd\" (UniqueName: \"kubernetes.io/projected/ac4b3887-d485-4fdc-848d-b9dff0adf0b0-kube-api-access-z9sgd\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:40.995877 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:40.995869    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac4b3887-d485-4fdc-848d-b9dff0adf0b0-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:41.209296 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:41.209207    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx"
Oct 30 18:55:41.215082 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:41.215055    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx"
Oct 30 18:55:41.658179 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:41.658151    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-635398a3-h52fw"
Oct 30 18:55:41.658600 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:41.658182    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-635398a3-h52fw" event={"ID":"ac4b3887-d485-4fdc-848d-b9dff0adf0b0","Type":"ContainerDied","Data":"b75316e0971079f045a3cecd3f703948021385b516bdaae59c02b2894e0f757e"}
Oct 30 18:55:41.658600 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:41.658236    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b75316e0971079f045a3cecd3f703948021385b516bdaae59c02b2894e0f757e"
Oct 30 18:55:41.658600 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:41.658541    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx"
Oct 30 18:55:41.659498 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:41.659473    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx"
Oct 30 18:55:47.398343 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:55:47.398305    4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d49l" containerName="extractor"
Oct 30 18:55:49.007970 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:49.007937    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dc0f87-fdbdk"]
Oct 30 18:55:49.008549 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:49.008535    4525 cpu_manager.go:414] "containerMap: removing container" podUID="ac4b3887-d485-4fdc-848d-b9dff0adf0b0" containerName="revoke-keys"
Oct 30 18:55:49.008619 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:49.008550    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac4b3887-d485-4fdc-848d-b9dff0adf0b0" containerName="revoke-keys"
Oct 30 18:55:49.008658 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:49.008648    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac4b3887-d485-4fdc-848d-b9dff0adf0b0" containerName="revoke-keys"
Oct 30 18:55:49.012533 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:49.012513    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dc0f87-fdbdk"
Oct 30 18:55:49.015974 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:49.015949    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:55:49.016876 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:49.016855    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94hcn\""
Oct 30 18:55:49.019104 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:49.019082    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dc0f87-fdbdk"]
Oct 30 18:55:49.169532 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:49.169500    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4c5cf74-22ff-47fc-a7d5-3f3eddd3b239-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-dc0f87-fdbdk\" (UID: \"e4c5cf74-22ff-47fc-a7d5-3f3eddd3b239\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dc0f87-fdbdk"
Oct 30 18:55:49.169725 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:49.169564    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4c5cf74-22ff-47fc-a7d5-3f3eddd3b239-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-dc0f87-fdbdk\" (UID: \"e4c5cf74-22ff-47fc-a7d5-3f3eddd3b239\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dc0f87-fdbdk"
Oct 30 18:55:49.169859 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:49.169838    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnd9s\" (UniqueName: \"kubernetes.io/projected/e4c5cf74-22ff-47fc-a7d5-3f3eddd3b239-kube-api-access-bnd9s\") pod \"maas-api-revoke-keys-e2e-api-a-dc0f87-fdbdk\" (UID: \"e4c5cf74-22ff-47fc-a7d5-3f3eddd3b239\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dc0f87-fdbdk"
Oct 30 18:55:49.271256 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:49.271155    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnd9s\" (UniqueName: \"kubernetes.io/projected/e4c5cf74-22ff-47fc-a7d5-3f3eddd3b239-kube-api-access-bnd9s\") pod \"maas-api-revoke-keys-e2e-api-a-dc0f87-fdbdk\" (UID: \"e4c5cf74-22ff-47fc-a7d5-3f3eddd3b239\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dc0f87-fdbdk"
Oct 30 18:55:49.271256 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:49.271239    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4c5cf74-22ff-47fc-a7d5-3f3eddd3b239-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-dc0f87-fdbdk\" (UID: \"e4c5cf74-22ff-47fc-a7d5-3f3eddd3b239\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dc0f87-fdbdk"
Oct 30 18:55:49.271488 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:49.271288    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4c5cf74-22ff-47fc-a7d5-3f3eddd3b239-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-dc0f87-fdbdk\" (UID: \"e4c5cf74-22ff-47fc-a7d5-3f3eddd3b239\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dc0f87-fdbdk"
Oct 30 18:55:49.272035 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:49.272010    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4c5cf74-22ff-47fc-a7d5-3f3eddd3b239-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-dc0f87-fdbdk\" (UID: \"e4c5cf74-22ff-47fc-a7d5-3f3eddd3b239\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dc0f87-fdbdk"
Oct 30 18:55:49.273868 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:49.273841    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4c5cf74-22ff-47fc-a7d5-3f3eddd3b239-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-dc0f87-fdbdk\" (UID: \"e4c5cf74-22ff-47fc-a7d5-3f3eddd3b239\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dc0f87-fdbdk"
Oct 30 18:55:49.279144 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:49.279123    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnd9s\" (UniqueName: \"kubernetes.io/projected/e4c5cf74-22ff-47fc-a7d5-3f3eddd3b239-kube-api-access-bnd9s\") pod \"maas-api-revoke-keys-e2e-api-a-dc0f87-fdbdk\" (UID: \"e4c5cf74-22ff-47fc-a7d5-3f3eddd3b239\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dc0f87-fdbdk"
Oct 30 18:55:49.331746 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:49.331721    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dc0f87-fdbdk"
Oct 30 18:55:49.618361 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:55:49.618333    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4c5cf74_22ff_47fc_a7d5_3f3eddd3b239.slice/crio-cdc380602029fd9ee96f90d4c8b1340289bd91ef8b64b072deeb5f7ed3737c80 WatchSource:0}: Error finding container cdc380602029fd9ee96f90d4c8b1340289bd91ef8b64b072deeb5f7ed3737c80: Status 404 returned error can't find the container with id cdc380602029fd9ee96f90d4c8b1340289bd91ef8b64b072deeb5f7ed3737c80
Oct 30 18:55:49.620554 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:49.620531    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dc0f87-fdbdk"]
Oct 30 18:55:49.791707 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:49.789089    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dc0f87-fdbdk" event={"ID":"e4c5cf74-22ff-47fc-a7d5-3f3eddd3b239","Type":"ContainerStarted","Data":"cdc380602029fd9ee96f90d4c8b1340289bd91ef8b64b072deeb5f7ed3737c80"}
Oct 30 18:55:50.895581 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:50.895551    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dc0f87-fdbdk" event={"ID":"e4c5cf74-22ff-47fc-a7d5-3f3eddd3b239","Type":"ContainerStarted","Data":"77b8f76751f2ecfb673475d5c389695275b75f72ce3bdefc8f6a1e30a6c6d871"}
Oct 30 18:55:51.911545 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:51.911501    4525 generic.go:356] "Generic (PLEG): container finished" podID="e4c5cf74-22ff-47fc-a7d5-3f3eddd3b239" containerID="77b8f76751f2ecfb673475d5c389695275b75f72ce3bdefc8f6a1e30a6c6d871" exitCode=0
Oct 30 18:55:51.912058 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:51.911550    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dc0f87-fdbdk" event={"ID":"e4c5cf74-22ff-47fc-a7d5-3f3eddd3b239","Type":"ContainerDied","Data":"77b8f76751f2ecfb673475d5c389695275b75f72ce3bdefc8f6a1e30a6c6d871"}
Oct 30 18:55:52.167166 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:52.167104    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dc0f87-fdbdk"
Oct 30 18:55:52.210725 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:52.210670    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4c5cf74-22ff-47fc-a7d5-3f3eddd3b239-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4c5cf74-22ff-47fc-a7d5-3f3eddd3b239-maas-api-xks-ca\") pod \"e4c5cf74-22ff-47fc-a7d5-3f3eddd3b239\" (UID: \"e4c5cf74-22ff-47fc-a7d5-3f3eddd3b239\") "
Oct 30 18:55:52.211081 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:52.211061    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4c5cf74-22ff-47fc-a7d5-3f3eddd3b239-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4c5cf74-22ff-47fc-a7d5-3f3eddd3b239-maas-api-openshift-ca\") pod \"e4c5cf74-22ff-47fc-a7d5-3f3eddd3b239\" (UID: \"e4c5cf74-22ff-47fc-a7d5-3f3eddd3b239\") "
Oct 30 18:55:52.211247 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:52.211230    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4c5cf74-22ff-47fc-a7d5-3f3eddd3b239-kube-api-access-bnd9s\" (UniqueName: \"kubernetes.io/projected/e4c5cf74-22ff-47fc-a7d5-3f3eddd3b239-kube-api-access-bnd9s\") pod \"e4c5cf74-22ff-47fc-a7d5-3f3eddd3b239\" (UID: \"e4c5cf74-22ff-47fc-a7d5-3f3eddd3b239\") "
Oct 30 18:55:52.214094 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:52.214037    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4c5cf74-22ff-47fc-a7d5-3f3eddd3b239-maas-api-openshift-ca" pod "e4c5cf74-22ff-47fc-a7d5-3f3eddd3b239" (UID: "e4c5cf74-22ff-47fc-a7d5-3f3eddd3b239"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:55:52.215269 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:52.215242    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4c5cf74-22ff-47fc-a7d5-3f3eddd3b239-kube-api-access-bnd9s" pod "e4c5cf74-22ff-47fc-a7d5-3f3eddd3b239" (UID: "e4c5cf74-22ff-47fc-a7d5-3f3eddd3b239"). InnerVolumeSpecName "kube-api-access-bnd9s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:55:52.227436 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:52.227405    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4c5cf74-22ff-47fc-a7d5-3f3eddd3b239-maas-api-xks-ca" pod "e4c5cf74-22ff-47fc-a7d5-3f3eddd3b239" (UID: "e4c5cf74-22ff-47fc-a7d5-3f3eddd3b239"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:55:52.313627 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:52.313592    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4c5cf74-22ff-47fc-a7d5-3f3eddd3b239-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:52.313627 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:52.313624    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4c5cf74-22ff-47fc-a7d5-3f3eddd3b239-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:52.313627 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:52.313634    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnd9s\" (UniqueName: \"kubernetes.io/projected/e4c5cf74-22ff-47fc-a7d5-3f3eddd3b239-kube-api-access-bnd9s\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:52.921882 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:52.921847    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dc0f87-fdbdk" event={"ID":"e4c5cf74-22ff-47fc-a7d5-3f3eddd3b239","Type":"ContainerDied","Data":"cdc380602029fd9ee96f90d4c8b1340289bd91ef8b64b072deeb5f7ed3737c80"}
Oct 30 18:55:52.921882 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:52.921866    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dc0f87-fdbdk"
Oct 30 18:55:52.922325 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:52.921897    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cdc380602029fd9ee96f90d4c8b1340289bd91ef8b64b072deeb5f7ed3737c80"
Oct 30 18:55:55.227706 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:55.227649    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-da58b769-q44hh"]
Oct 30 18:55:55.238788 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:55.238138    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-da58b769-q44hh"]
Oct 30 18:55:55.545302 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:55.545221    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e131d18d-909e-44be-bcc8-b42960212936" path="/var/lib/kubelet/pods/e131d18d-909e-44be-bcc8-b42960212936/volumes"
Oct 30 18:55:59.003751 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:58.998509    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-8f1c37-openshift-default-7cd9c57fdf-kwhps"]
Oct 30 18:55:59.003751 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:58.999394    4525 cpu_manager.go:414] "containerMap: removing container" podUID="e4c5cf74-22ff-47fc-a7d5-3f3eddd3b239" containerName="revoke-keys"
Oct 30 18:55:59.003751 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:58.999415    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4c5cf74-22ff-47fc-a7d5-3f3eddd3b239" containerName="revoke-keys"
Oct 30 18:55:59.003751 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:58.999614    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4c5cf74-22ff-47fc-a7d5-3f3eddd3b239" containerName="revoke-keys"
Oct 30 18:55:59.015774 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:59.015591    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-8f1c37-openshift-default-7cd9c57fdf-kwhps"
Oct 30 18:55:59.026628 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:59.026594    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-8f1c37-openshift-default-7cd9c57fdf-kwhps"]
Oct 30 18:55:59.029426 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:59.029400    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-8f1c37-openshift-default-dockercfg-x49xv\""
Oct 30 18:55:59.118548 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:59.118513    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjhrn\" (UniqueName: \"kubernetes.io/projected/9c3b7950-7451-4f37-8119-f9574c3045f3-kube-api-access-qjhrn\") pod \"e2e-api-a-8f1c37-openshift-default-7cd9c57fdf-kwhps\" (UID: \"9c3b7950-7451-4f37-8119-f9574c3045f3\") " pod="openshift-ingress/e2e-api-a-8f1c37-openshift-default-7cd9c57fdf-kwhps"
Oct 30 18:55:59.118725 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:59.118582    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c3b7950-7451-4f37-8119-f9574c3045f3-workload-socket\") pod \"e2e-api-a-8f1c37-openshift-default-7cd9c57fdf-kwhps\" (UID: \"9c3b7950-7451-4f37-8119-f9574c3045f3\") " pod="openshift-ingress/e2e-api-a-8f1c37-openshift-default-7cd9c57fdf-kwhps"
Oct 30 18:55:59.118786 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:59.118755    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c3b7950-7451-4f37-8119-f9574c3045f3-istio-data\") pod \"e2e-api-a-8f1c37-openshift-default-7cd9c57fdf-kwhps\" (UID: \"9c3b7950-7451-4f37-8119-f9574c3045f3\") " pod="openshift-ingress/e2e-api-a-8f1c37-openshift-default-7cd9c57fdf-kwhps"
Oct 30 18:55:59.118936 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:59.118916    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c3b7950-7451-4f37-8119-f9574c3045f3-istiod-ca-cert\") pod \"e2e-api-a-8f1c37-openshift-default-7cd9c57fdf-kwhps\" (UID: \"9c3b7950-7451-4f37-8119-f9574c3045f3\") " pod="openshift-ingress/e2e-api-a-8f1c37-openshift-default-7cd9c57fdf-kwhps"
Oct 30 18:55:59.119042 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:59.119023    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c3b7950-7451-4f37-8119-f9574c3045f3-istio-envoy\") pod \"e2e-api-a-8f1c37-openshift-default-7cd9c57fdf-kwhps\" (UID: \"9c3b7950-7451-4f37-8119-f9574c3045f3\") " pod="openshift-ingress/e2e-api-a-8f1c37-openshift-default-7cd9c57fdf-kwhps"
Oct 30 18:55:59.119116 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:59.119100    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c3b7950-7451-4f37-8119-f9574c3045f3-istio-token\") pod \"e2e-api-a-8f1c37-openshift-default-7cd9c57fdf-kwhps\" (UID: \"9c3b7950-7451-4f37-8119-f9574c3045f3\") " pod="openshift-ingress/e2e-api-a-8f1c37-openshift-default-7cd9c57fdf-kwhps"
Oct 30 18:55:59.119227 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:59.119212    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c3b7950-7451-4f37-8119-f9574c3045f3-credential-socket\") pod \"e2e-api-a-8f1c37-openshift-default-7cd9c57fdf-kwhps\" (UID: \"9c3b7950-7451-4f37-8119-f9574c3045f3\") " pod="openshift-ingress/e2e-api-a-8f1c37-openshift-default-7cd9c57fdf-kwhps"
Oct 30 18:55:59.119280 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:59.119254    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c3b7950-7451-4f37-8119-f9574c3045f3-istio-podinfo\") pod \"e2e-api-a-8f1c37-openshift-default-7cd9c57fdf-kwhps\" (UID: \"9c3b7950-7451-4f37-8119-f9574c3045f3\") " pod="openshift-ingress/e2e-api-a-8f1c37-openshift-default-7cd9c57fdf-kwhps"
Oct 30 18:55:59.119370 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:59.119313    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c3b7950-7451-4f37-8119-f9574c3045f3-workload-certs\") pod \"e2e-api-a-8f1c37-openshift-default-7cd9c57fdf-kwhps\" (UID: \"9c3b7950-7451-4f37-8119-f9574c3045f3\") " pod="openshift-ingress/e2e-api-a-8f1c37-openshift-default-7cd9c57fdf-kwhps"
Oct 30 18:55:59.225469 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:59.225430    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c3b7950-7451-4f37-8119-f9574c3045f3-istiod-ca-cert\") pod \"e2e-api-a-8f1c37-openshift-default-7cd9c57fdf-kwhps\" (UID: \"9c3b7950-7451-4f37-8119-f9574c3045f3\") " pod="openshift-ingress/e2e-api-a-8f1c37-openshift-default-7cd9c57fdf-kwhps"
Oct 30 18:55:59.225669 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:59.225517    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c3b7950-7451-4f37-8119-f9574c3045f3-istio-envoy\") pod \"e2e-api-a-8f1c37-openshift-default-7cd9c57fdf-kwhps\" (UID: \"9c3b7950-7451-4f37-8119-f9574c3045f3\") " pod="openshift-ingress/e2e-api-a-8f1c37-openshift-default-7cd9c57fdf-kwhps"
Oct 30 18:55:59.225669 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:59.225578    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c3b7950-7451-4f37-8119-f9574c3045f3-istio-token\") pod \"e2e-api-a-8f1c37-openshift-default-7cd9c57fdf-kwhps\" (UID: \"9c3b7950-7451-4f37-8119-f9574c3045f3\") " pod="openshift-ingress/e2e-api-a-8f1c37-openshift-default-7cd9c57fdf-kwhps"
Oct 30 18:55:59.225669 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:59.225640    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c3b7950-7451-4f37-8119-f9574c3045f3-credential-socket\") pod \"e2e-api-a-8f1c37-openshift-default-7cd9c57fdf-kwhps\" (UID: \"9c3b7950-7451-4f37-8119-f9574c3045f3\") " pod="openshift-ingress/e2e-api-a-8f1c37-openshift-default-7cd9c57fdf-kwhps"
Oct 30 18:55:59.225906 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:59.225706    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c3b7950-7451-4f37-8119-f9574c3045f3-istio-podinfo\") pod \"e2e-api-a-8f1c37-openshift-default-7cd9c57fdf-kwhps\" (UID: \"9c3b7950-7451-4f37-8119-f9574c3045f3\") " pod="openshift-ingress/e2e-api-a-8f1c37-openshift-default-7cd9c57fdf-kwhps"
Oct 30 18:55:59.225906 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:59.225735    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c3b7950-7451-4f37-8119-f9574c3045f3-workload-certs\") pod \"e2e-api-a-8f1c37-openshift-default-7cd9c57fdf-kwhps\" (UID: \"9c3b7950-7451-4f37-8119-f9574c3045f3\") " pod="openshift-ingress/e2e-api-a-8f1c37-openshift-default-7cd9c57fdf-kwhps"
Oct 30 18:55:59.225906 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:59.225796    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjhrn\" (UniqueName: \"kubernetes.io/projected/9c3b7950-7451-4f37-8119-f9574c3045f3-kube-api-access-qjhrn\") pod \"e2e-api-a-8f1c37-openshift-default-7cd9c57fdf-kwhps\" (UID: \"9c3b7950-7451-4f37-8119-f9574c3045f3\") " pod="openshift-ingress/e2e-api-a-8f1c37-openshift-default-7cd9c57fdf-kwhps"
Oct 30 18:55:59.225906 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:59.225858    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c3b7950-7451-4f37-8119-f9574c3045f3-workload-socket\") pod \"e2e-api-a-8f1c37-openshift-default-7cd9c57fdf-kwhps\" (UID: \"9c3b7950-7451-4f37-8119-f9574c3045f3\") " pod="openshift-ingress/e2e-api-a-8f1c37-openshift-default-7cd9c57fdf-kwhps"
Oct 30 18:55:59.226117 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:59.225937    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c3b7950-7451-4f37-8119-f9574c3045f3-istio-data\") pod \"e2e-api-a-8f1c37-openshift-default-7cd9c57fdf-kwhps\" (UID: \"9c3b7950-7451-4f37-8119-f9574c3045f3\") " pod="openshift-ingress/e2e-api-a-8f1c37-openshift-default-7cd9c57fdf-kwhps"
Oct 30 18:55:59.226463 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:59.226275    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c3b7950-7451-4f37-8119-f9574c3045f3-istiod-ca-cert\") pod \"e2e-api-a-8f1c37-openshift-default-7cd9c57fdf-kwhps\" (UID: \"9c3b7950-7451-4f37-8119-f9574c3045f3\") " pod="openshift-ingress/e2e-api-a-8f1c37-openshift-default-7cd9c57fdf-kwhps"
Oct 30 18:55:59.226463 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:59.226328    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c3b7950-7451-4f37-8119-f9574c3045f3-istio-data\") pod \"e2e-api-a-8f1c37-openshift-default-7cd9c57fdf-kwhps\" (UID: \"9c3b7950-7451-4f37-8119-f9574c3045f3\") " pod="openshift-ingress/e2e-api-a-8f1c37-openshift-default-7cd9c57fdf-kwhps"
Oct 30 18:55:59.226600 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:59.226547    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c3b7950-7451-4f37-8119-f9574c3045f3-workload-certs\") pod \"e2e-api-a-8f1c37-openshift-default-7cd9c57fdf-kwhps\" (UID: \"9c3b7950-7451-4f37-8119-f9574c3045f3\") " pod="openshift-ingress/e2e-api-a-8f1c37-openshift-default-7cd9c57fdf-kwhps"
Oct 30 18:55:59.227031 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:59.227012    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c3b7950-7451-4f37-8119-f9574c3045f3-workload-socket\") pod \"e2e-api-a-8f1c37-openshift-default-7cd9c57fdf-kwhps\" (UID: \"9c3b7950-7451-4f37-8119-f9574c3045f3\") " pod="openshift-ingress/e2e-api-a-8f1c37-openshift-default-7cd9c57fdf-kwhps"
Oct 30 18:55:59.227228 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:59.227202    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c3b7950-7451-4f37-8119-f9574c3045f3-credential-socket\") pod \"e2e-api-a-8f1c37-openshift-default-7cd9c57fdf-kwhps\" (UID: \"9c3b7950-7451-4f37-8119-f9574c3045f3\") " pod="openshift-ingress/e2e-api-a-8f1c37-openshift-default-7cd9c57fdf-kwhps"
Oct 30 18:55:59.228572 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:59.228551    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c3b7950-7451-4f37-8119-f9574c3045f3-istio-envoy\") pod \"e2e-api-a-8f1c37-openshift-default-7cd9c57fdf-kwhps\" (UID: \"9c3b7950-7451-4f37-8119-f9574c3045f3\") " pod="openshift-ingress/e2e-api-a-8f1c37-openshift-default-7cd9c57fdf-kwhps"
Oct 30 18:55:59.229047 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:59.229028    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c3b7950-7451-4f37-8119-f9574c3045f3-istio-podinfo\") pod \"e2e-api-a-8f1c37-openshift-default-7cd9c57fdf-kwhps\" (UID: \"9c3b7950-7451-4f37-8119-f9574c3045f3\") " pod="openshift-ingress/e2e-api-a-8f1c37-openshift-default-7cd9c57fdf-kwhps"
Oct 30 18:55:59.236926 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:59.236902    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c3b7950-7451-4f37-8119-f9574c3045f3-istio-token\") pod \"e2e-api-a-8f1c37-openshift-default-7cd9c57fdf-kwhps\" (UID: \"9c3b7950-7451-4f37-8119-f9574c3045f3\") " pod="openshift-ingress/e2e-api-a-8f1c37-openshift-default-7cd9c57fdf-kwhps"
Oct 30 18:55:59.237915 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:59.237892    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjhrn\" (UniqueName: \"kubernetes.io/projected/9c3b7950-7451-4f37-8119-f9574c3045f3-kube-api-access-qjhrn\") pod \"e2e-api-a-8f1c37-openshift-default-7cd9c57fdf-kwhps\" (UID: \"9c3b7950-7451-4f37-8119-f9574c3045f3\") " pod="openshift-ingress/e2e-api-a-8f1c37-openshift-default-7cd9c57fdf-kwhps"
Oct 30 18:55:59.349781 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:59.349700    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-8f1c37-openshift-default-7cd9c57fdf-kwhps"
Oct 30 18:55:59.784025 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:59.783990    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-8f1c37-openshift-default-7cd9c57fdf-kwhps"]
Oct 30 18:55:59.789311 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:59.788808    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:55:59.789311 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:59.788891    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:55:59.789311 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:55:59.788937    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:56:00.179916 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:00.179875    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8f1c37-openshift-default-7cd9c57fdf-kwhps" event={"ID":"9c3b7950-7451-4f37-8119-f9574c3045f3","Type":"ContainerStarted","Data":"18a288522f8406d4217f3b2d58d5f6c7858fac117df2b308e65e0868deb47272"}
Oct 30 18:56:00.180325 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:00.179943    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8f1c37-openshift-default-7cd9c57fdf-kwhps" event={"ID":"9c3b7950-7451-4f37-8119-f9574c3045f3","Type":"ContainerStarted","Data":"6456efc3c3b473993b9bd1ad703bd38538b2ff81507ce95a4bf405fe988c0480"}
Oct 30 18:56:00.201804 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:00.201751    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-8f1c37-openshift-default-7cd9c57fdf-kwhps" podStartSLOduration=2.201738034 podStartE2EDuration="2.201738034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:55:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:56:00.19991516 +0000 UTC m=+1182.825585159" watchObservedRunningTime="2026-10-30 18:56:00.201738034 +0000 UTC m=+1182.827407998"
Oct 30 18:56:00.353995 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:00.353950    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-8f1c37-openshift-default-7cd9c57fdf-kwhps"
Oct 30 18:56:00.368210 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:00.367428    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-8f1c37-openshift-default-7cd9c57fdf-kwhps"
Oct 30 18:56:01.191290 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:01.191259    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-8f1c37-openshift-default-7cd9c57fdf-kwhps"
Oct 30 18:56:01.192264 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:01.192238    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-8f1c37-openshift-default-7cd9c57fdf-kwhps"
Oct 30 18:56:02.201106 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:02.201069    4525 generic.go:356] "Generic (PLEG): container finished" podID="ed65886b-3abc-4b57-9206-8e0603411200" containerID="68154ca9a57e6d106cc15aab5cacacf31c1c64634f8c4eba89fa726c0adcb905" exitCode=0
Oct 30 18:56:02.201653 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:02.201163    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9982ad-openshift-default-6f478df445-gr62t" event={"ID":"ed65886b-3abc-4b57-9206-8e0603411200","Type":"ContainerDied","Data":"68154ca9a57e6d106cc15aab5cacacf31c1c64634f8c4eba89fa726c0adcb905"}
Oct 30 18:56:02.201818 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:02.201802    4525 scope.go:122] "RemoveContainer" containerID="68154ca9a57e6d106cc15aab5cacacf31c1c64634f8c4eba89fa726c0adcb905"
Oct 30 18:56:02.203286 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:02.203252    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:56:02.203369 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:02.203322    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:56:02.203369 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:02.203355    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:56:02.945251 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:02.945218    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-93574a-84684456c9-k4pt5"]
Oct 30 18:56:02.950223 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:02.950197    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-93574a-84684456c9-k4pt5"
Oct 30 18:56:02.955589 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:02.955568    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-93574a-ccbc886cb-kq9wb"]
Oct 30 18:56:02.960046 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:02.960029    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-93574a-ccbc886cb-kq9wb"
Oct 30 18:56:02.964655 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:02.964632    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-93574a\""
Oct 30 18:56:02.968031 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:02.967773    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-93574a-dockercfg-tjcqw\""
Oct 30 18:56:02.972366 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:02.972345    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-93574a-84684456c9-k4pt5"]
Oct 30 18:56:03.012970 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:03.012932    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-93574a-ccbc886cb-kq9wb"]
Oct 30 18:56:03.020251 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:03.020203    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nllf6\" (UniqueName: \"kubernetes.io/projected/33de14c3-409a-4c51-b4c4-74a849da5b63-kube-api-access-nllf6\") pod \"payload-processing-e2e-ipp-a-93574a-ccbc886cb-kq9wb\" (UID: \"33de14c3-409a-4c51-b4c4-74a849da5b63\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-93574a-ccbc886cb-kq9wb"
Oct 30 18:56:03.020596 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:03.020568    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ltfr\" (UniqueName: \"kubernetes.io/projected/4728df97-385f-4c6d-ac79-be8509ddc7ad-kube-api-access-8ltfr\") pod \"payload-pre-processing-e2e-ipp-a-93574a-84684456c9-k4pt5\" (UID: \"4728df97-385f-4c6d-ac79-be8509ddc7ad\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-93574a-84684456c9-k4pt5"
Oct 30 18:56:03.020944 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:03.020911    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4728df97-385f-4c6d-ac79-be8509ddc7ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-93574a-84684456c9-k4pt5\" (UID: \"4728df97-385f-4c6d-ac79-be8509ddc7ad\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-93574a-84684456c9-k4pt5"
Oct 30 18:56:03.021181 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:03.021165    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33de14c3-409a-4c51-b4c4-74a849da5b63-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-93574a-ccbc886cb-kq9wb\" (UID: \"33de14c3-409a-4c51-b4c4-74a849da5b63\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-93574a-ccbc886cb-kq9wb"
Oct 30 18:56:03.122137 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:03.122101    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4728df97-385f-4c6d-ac79-be8509ddc7ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-93574a-84684456c9-k4pt5\" (UID: \"4728df97-385f-4c6d-ac79-be8509ddc7ad\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-93574a-84684456c9-k4pt5"
Oct 30 18:56:03.122325 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:03.122214    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33de14c3-409a-4c51-b4c4-74a849da5b63-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-93574a-ccbc886cb-kq9wb\" (UID: \"33de14c3-409a-4c51-b4c4-74a849da5b63\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-93574a-ccbc886cb-kq9wb"
Oct 30 18:56:03.122325 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:03.122278    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nllf6\" (UniqueName: \"kubernetes.io/projected/33de14c3-409a-4c51-b4c4-74a849da5b63-kube-api-access-nllf6\") pod \"payload-processing-e2e-ipp-a-93574a-ccbc886cb-kq9wb\" (UID: \"33de14c3-409a-4c51-b4c4-74a849da5b63\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-93574a-ccbc886cb-kq9wb"
Oct 30 18:56:03.122447 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:03.122359    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ltfr\" (UniqueName: \"kubernetes.io/projected/4728df97-385f-4c6d-ac79-be8509ddc7ad-kube-api-access-8ltfr\") pod \"payload-pre-processing-e2e-ipp-a-93574a-84684456c9-k4pt5\" (UID: \"4728df97-385f-4c6d-ac79-be8509ddc7ad\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-93574a-84684456c9-k4pt5"
Oct 30 18:56:03.122884 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:03.122861    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4728df97-385f-4c6d-ac79-be8509ddc7ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-93574a-84684456c9-k4pt5\" (UID: \"4728df97-385f-4c6d-ac79-be8509ddc7ad\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-93574a-84684456c9-k4pt5"
Oct 30 18:56:03.123133 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:03.123114    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33de14c3-409a-4c51-b4c4-74a849da5b63-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-93574a-ccbc886cb-kq9wb\" (UID: \"33de14c3-409a-4c51-b4c4-74a849da5b63\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-93574a-ccbc886cb-kq9wb"
Oct 30 18:56:03.131148 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:03.131123    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nllf6\" (UniqueName: \"kubernetes.io/projected/33de14c3-409a-4c51-b4c4-74a849da5b63-kube-api-access-nllf6\") pod \"payload-processing-e2e-ipp-a-93574a-ccbc886cb-kq9wb\" (UID: \"33de14c3-409a-4c51-b4c4-74a849da5b63\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-93574a-ccbc886cb-kq9wb"
Oct 30 18:56:03.131244 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:03.131190    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ltfr\" (UniqueName: \"kubernetes.io/projected/4728df97-385f-4c6d-ac79-be8509ddc7ad-kube-api-access-8ltfr\") pod \"payload-pre-processing-e2e-ipp-a-93574a-84684456c9-k4pt5\" (UID: \"4728df97-385f-4c6d-ac79-be8509ddc7ad\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-93574a-84684456c9-k4pt5"
Oct 30 18:56:03.213648 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:03.213553    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9982ad-openshift-default-6f478df445-gr62t" event={"ID":"ed65886b-3abc-4b57-9206-8e0603411200","Type":"ContainerStarted","Data":"cbd730893f8345ba95b708f10b7c44fa10f0d9ca38bdc0a39e14e24a9626f3ae"}
Oct 30 18:56:03.266733 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:03.266667    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-93574a-84684456c9-k4pt5"
Oct 30 18:56:03.306609 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:03.302422    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-9982ad-openshift-default-6f478df445-gr62t"
Oct 30 18:56:03.306890 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:03.306735    4525 patch_prober.go:29] interesting pod/e2e-api-b-9982ad-openshift-default-6f478df445-gr62t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Oct 30 18:56:03.306890 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:03.306799    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9982ad-openshift-default-6f478df445-gr62t" podUID="ed65886b-3abc-4b57-9206-8e0603411200" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Oct 30 18:56:03.310887 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:03.310862    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-93574a-ccbc886cb-kq9wb"
Oct 30 18:56:03.956952 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:03.956920    4525 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 18:56:03.961120 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:03.961095    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-93574a-84684456c9-k4pt5"]
Oct 30 18:56:04.024892 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:04.024862    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-93574a-ccbc886cb-kq9wb"]
Oct 30 18:56:04.025077 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:56:04.025055    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33de14c3_409a_4c51_b4c4_74a849da5b63.slice/crio-1132e3e932c6b8b8ab1bac44df60a578ceeea9b51b33529932d66b76207aedff WatchSource:0}: Error finding container 1132e3e932c6b8b8ab1bac44df60a578ceeea9b51b33529932d66b76207aedff: Status 404 returned error can't find the container with id 1132e3e932c6b8b8ab1bac44df60a578ceeea9b51b33529932d66b76207aedff
Oct 30 18:56:04.271599 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:04.271238    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-93574a-84684456c9-k4pt5" event={"ID":"4728df97-385f-4c6d-ac79-be8509ddc7ad","Type":"ContainerStarted","Data":"f8e395e1a9d274817c96a583f055bb64ad5629c7ee862156d613dce84e31e4e6"}
Oct 30 18:56:04.296704 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:04.286223    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-93574a-ccbc886cb-kq9wb" event={"ID":"33de14c3-409a-4c51-b4c4-74a849da5b63","Type":"ContainerStarted","Data":"1132e3e932c6b8b8ab1bac44df60a578ceeea9b51b33529932d66b76207aedff"}
Oct 30 18:56:04.333704 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:04.330187    4525 patch_prober.go:29] interesting pod/e2e-api-b-9982ad-openshift-default-6f478df445-gr62t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Oct 30 18:56:04.333704 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:04.330259    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9982ad-openshift-default-6f478df445-gr62t" podUID="ed65886b-3abc-4b57-9206-8e0603411200" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Oct 30 18:56:05.302959 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:05.302499    4525 patch_prober.go:29] interesting pod/e2e-api-b-9982ad-openshift-default-6f478df445-gr62t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Oct 30 18:56:05.302959 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:05.302560    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9982ad-openshift-default-6f478df445-gr62t" podUID="ed65886b-3abc-4b57-9206-8e0603411200" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Oct 30 18:56:06.305727 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:06.303241    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-9982ad-openshift-default-6f478df445-gr62t"
Oct 30 18:56:06.305727 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:06.304313    4525 patch_prober.go:29] interesting pod/e2e-api-b-9982ad-openshift-default-6f478df445-gr62t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Oct 30 18:56:06.305727 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:06.304357    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9982ad-openshift-default-6f478df445-gr62t" podUID="ed65886b-3abc-4b57-9206-8e0603411200" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Oct 30 18:56:07.315716 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:07.311079    4525 patch_prober.go:29] interesting pod/e2e-api-b-9982ad-openshift-default-6f478df445-gr62t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Oct 30 18:56:07.315716 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:07.311143    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9982ad-openshift-default-6f478df445-gr62t" podUID="ed65886b-3abc-4b57-9206-8e0603411200" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Oct 30 18:56:08.302706 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:08.302126    4525 patch_prober.go:29] interesting pod/e2e-api-b-9982ad-openshift-default-6f478df445-gr62t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Oct 30 18:56:08.302706 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:08.302206    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9982ad-openshift-default-6f478df445-gr62t" podUID="ed65886b-3abc-4b57-9206-8e0603411200" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Oct 30 18:56:08.455870 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:08.455828    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-93574a-84684456c9-k4pt5" event={"ID":"4728df97-385f-4c6d-ac79-be8509ddc7ad","Type":"ContainerStarted","Data":"8fb3ffc670470a1aa1dd3fd56a850d37206d5bab833a613db9ac647eee05a61a"}
Oct 30 18:56:08.456325 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:08.455935    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-93574a-84684456c9-k4pt5"
Oct 30 18:56:08.457732 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:08.457700    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-93574a-ccbc886cb-kq9wb" event={"ID":"33de14c3-409a-4c51-b4c4-74a849da5b63","Type":"ContainerStarted","Data":"15adbe12d785bb996e9c7462d61ec8dac5e227b576d82792b93cb56d717c8806"}
Oct 30 18:56:08.457885 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:08.457804    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-93574a-ccbc886cb-kq9wb"
Oct 30 18:56:08.474993 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:08.474928    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-93574a-84684456c9-k4pt5" podStartSLOduration=3.165600597 podStartE2EDuration="6.474910033s" totalImagesPullingTime="3.309309436s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:56:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:56:08.472762427 +0000 UTC m=+1191.098432392" watchObservedRunningTime="2026-10-30 18:56:08.474910033 +0000 UTC m=+1191.100579999"
Oct 30 18:56:08.491921 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:08.491874    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-93574a-ccbc886cb-kq9wb" podStartSLOduration=3.12382652 podStartE2EDuration="6.491858902s" totalImagesPullingTime="3.368032382s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:56:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:56:08.488551862 +0000 UTC m=+1191.114221828" watchObservedRunningTime="2026-10-30 18:56:08.491858902 +0000 UTC m=+1191.117528866"
Oct 30 18:56:09.307246 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:09.307214    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-9982ad-openshift-default-6f478df445-gr62t"
Oct 30 18:56:09.308537 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:09.308520    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-9982ad-openshift-default-6f478df445-gr62t"
Oct 30 18:56:17.548706 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:56:17.529995    4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d49l" containerName="extractor"
Oct 30 18:56:18.130706 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:18.130163    4525 scope.go:122] "RemoveContainer" containerID="53cc1281188c36711e03d4f331e49fdd13802543589c56bfb9acdde49deea1cb"
Oct 30 18:56:18.144092 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:18.144068    4525 scope.go:122] "RemoveContainer" containerID="aca975120a4d12ccf001b26d9fa2f5e1690a4cb5bacfb89cd5e0b52d29da8c0b"
Oct 30 18:56:18.157421 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:18.157399    4525 scope.go:122] "RemoveContainer" containerID="830230dce6c9c86f7b462b16d3a9eadfd73bfa01210f5150a4e441a1c9590659"
Oct 30 18:56:18.169288 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:18.169264    4525 scope.go:122] "RemoveContainer" containerID="0b2741fe94f37d0a61ff2ceadc18faaecf784f00a6fe34b161c9f97aaac991e6"
Oct 30 18:56:18.230726 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:18.230579    4525 scope.go:122] "RemoveContainer" containerID="bf8247c0c864f9d6bca294d99482dd416e6aac8f2de4e58f4390d7810b4db031"
Oct 30 18:56:18.245414 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:18.245392    4525 scope.go:122] "RemoveContainer" containerID="a5f90c0a08b4da14c5207644a429b2991383f0cf426241b35803a8db4b92cf70"
Oct 30 18:56:18.255862 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:18.255839    4525 scope.go:122] "RemoveContainer" containerID="3b66cdb5e76d679a9c954b03d867ba1c4710007325158dd3682dfda2372e1242"
Oct 30 18:56:18.265239 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:18.265218    4525 scope.go:122] "RemoveContainer" containerID="f83260bf5fd59e11d67dc243506405532a3e0cf85e908bc89c24db286fda49fd"
Oct 30 18:56:19.042424 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:19.042115    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51fe5e9b-pskxl"]
Oct 30 18:56:19.047719 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:19.047690    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b473ede5-lndt8"]
Oct 30 18:56:19.052550 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:19.052531    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51fe5e9b-pskxl"]
Oct 30 18:56:19.056826 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:19.056809    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b473ede5-lndt8"]
Oct 30 18:56:19.493558 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:19.483328    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-93574a-84684456c9-k4pt5"
Oct 30 18:56:19.514708 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:19.494818    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-93574a-ccbc886cb-kq9wb"
Oct 30 18:56:19.564537 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:19.564508    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="412a1669-045f-4356-a0b4-773b3988ee04" path="/var/lib/kubelet/pods/412a1669-045f-4356-a0b4-773b3988ee04/volumes"
Oct 30 18:56:19.565237 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:19.565218    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3d06734-31e4-4da4-9618-f8aadfad076a" path="/var/lib/kubelet/pods/e3d06734-31e4-4da4-9618-f8aadfad076a/volumes"
Oct 30 18:56:24.225447 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:24.223300    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td"]
Oct 30 18:56:24.274417 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:24.274375    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td"]
Oct 30 18:56:24.274605 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:24.274511    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td"
Oct 30 18:56:24.281271 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:24.281239    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-24fceb-openshift-default-dockercfg-v4p8f\""
Oct 30 18:56:24.392921 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:24.392880    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bce8221a-1318-4f64-908c-d927050e1ac1-workload-certs\") pod \"e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td\" (UID: \"bce8221a-1318-4f64-908c-d927050e1ac1\") " pod="openshift-ingress/e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td"
Oct 30 18:56:24.393088 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:24.392944    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bce8221a-1318-4f64-908c-d927050e1ac1-istio-envoy\") pod \"e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td\" (UID: \"bce8221a-1318-4f64-908c-d927050e1ac1\") " pod="openshift-ingress/e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td"
Oct 30 18:56:24.393131 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:24.393102    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f24h9\" (UniqueName: \"kubernetes.io/projected/bce8221a-1318-4f64-908c-d927050e1ac1-kube-api-access-f24h9\") pod \"e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td\" (UID: \"bce8221a-1318-4f64-908c-d927050e1ac1\") " pod="openshift-ingress/e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td"
Oct 30 18:56:24.393293 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:24.393274    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bce8221a-1318-4f64-908c-d927050e1ac1-workload-socket\") pod \"e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td\" (UID: \"bce8221a-1318-4f64-908c-d927050e1ac1\") " pod="openshift-ingress/e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td"
Oct 30 18:56:24.393342 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:24.393326    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bce8221a-1318-4f64-908c-d927050e1ac1-credential-socket\") pod \"e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td\" (UID: \"bce8221a-1318-4f64-908c-d927050e1ac1\") " pod="openshift-ingress/e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td"
Oct 30 18:56:24.393443 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:24.393425    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bce8221a-1318-4f64-908c-d927050e1ac1-istio-token\") pod \"e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td\" (UID: \"bce8221a-1318-4f64-908c-d927050e1ac1\") " pod="openshift-ingress/e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td"
Oct 30 18:56:24.393529 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:24.393510    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bce8221a-1318-4f64-908c-d927050e1ac1-istio-data\") pod \"e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td\" (UID: \"bce8221a-1318-4f64-908c-d927050e1ac1\") " pod="openshift-ingress/e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td"
Oct 30 18:56:24.393592 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:24.393577    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bce8221a-1318-4f64-908c-d927050e1ac1-istio-podinfo\") pod \"e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td\" (UID: \"bce8221a-1318-4f64-908c-d927050e1ac1\") " pod="openshift-ingress/e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td"
Oct 30 18:56:24.393708 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:24.393665    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bce8221a-1318-4f64-908c-d927050e1ac1-istiod-ca-cert\") pod \"e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td\" (UID: \"bce8221a-1318-4f64-908c-d927050e1ac1\") " pod="openshift-ingress/e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td"
Oct 30 18:56:24.492596 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:24.492499    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dc0f87-fdbdk"]
Oct 30 18:56:24.494834 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:24.494801    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bce8221a-1318-4f64-908c-d927050e1ac1-istio-envoy\") pod \"e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td\" (UID: \"bce8221a-1318-4f64-908c-d927050e1ac1\") " pod="openshift-ingress/e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td"
Oct 30 18:56:24.495004 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:24.494883    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f24h9\" (UniqueName: \"kubernetes.io/projected/bce8221a-1318-4f64-908c-d927050e1ac1-kube-api-access-f24h9\") pod \"e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td\" (UID: \"bce8221a-1318-4f64-908c-d927050e1ac1\") " pod="openshift-ingress/e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td"
Oct 30 18:56:24.495086 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:24.495002    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bce8221a-1318-4f64-908c-d927050e1ac1-workload-socket\") pod \"e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td\" (UID: \"bce8221a-1318-4f64-908c-d927050e1ac1\") " pod="openshift-ingress/e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td"
Oct 30 18:56:24.495086 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:24.495047    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bce8221a-1318-4f64-908c-d927050e1ac1-credential-socket\") pod \"e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td\" (UID: \"bce8221a-1318-4f64-908c-d927050e1ac1\") " pod="openshift-ingress/e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td"
Oct 30 18:56:24.495234 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:24.495108    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bce8221a-1318-4f64-908c-d927050e1ac1-istio-token\") pod \"e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td\" (UID: \"bce8221a-1318-4f64-908c-d927050e1ac1\") " pod="openshift-ingress/e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td"
Oct 30 18:56:24.495234 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:24.495151    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bce8221a-1318-4f64-908c-d927050e1ac1-istio-data\") pod \"e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td\" (UID: \"bce8221a-1318-4f64-908c-d927050e1ac1\") " pod="openshift-ingress/e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td"
Oct 30 18:56:24.495234 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:24.495209    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bce8221a-1318-4f64-908c-d927050e1ac1-istio-podinfo\") pod \"e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td\" (UID: \"bce8221a-1318-4f64-908c-d927050e1ac1\") " pod="openshift-ingress/e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td"
Oct 30 18:56:24.495409 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:24.495252    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bce8221a-1318-4f64-908c-d927050e1ac1-istiod-ca-cert\") pod \"e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td\" (UID: \"bce8221a-1318-4f64-908c-d927050e1ac1\") " pod="openshift-ingress/e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td"
Oct 30 18:56:24.495409 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:24.495345    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bce8221a-1318-4f64-908c-d927050e1ac1-workload-certs\") pod \"e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td\" (UID: \"bce8221a-1318-4f64-908c-d927050e1ac1\") " pod="openshift-ingress/e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td"
Oct 30 18:56:24.497444 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:24.495840    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bce8221a-1318-4f64-908c-d927050e1ac1-workload-certs\") pod \"e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td\" (UID: \"bce8221a-1318-4f64-908c-d927050e1ac1\") " pod="openshift-ingress/e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td"
Oct 30 18:56:24.497444 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:24.496206    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bce8221a-1318-4f64-908c-d927050e1ac1-istio-data\") pod \"e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td\" (UID: \"bce8221a-1318-4f64-908c-d927050e1ac1\") " pod="openshift-ingress/e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td"
Oct 30 18:56:24.497444 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:24.496409    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bce8221a-1318-4f64-908c-d927050e1ac1-workload-socket\") pod \"e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td\" (UID: \"bce8221a-1318-4f64-908c-d927050e1ac1\") " pod="openshift-ingress/e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td"
Oct 30 18:56:24.497444 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:24.496661    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bce8221a-1318-4f64-908c-d927050e1ac1-credential-socket\") pod \"e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td\" (UID: \"bce8221a-1318-4f64-908c-d927050e1ac1\") " pod="openshift-ingress/e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td"
Oct 30 18:56:24.497444 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:24.497304    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bce8221a-1318-4f64-908c-d927050e1ac1-istiod-ca-cert\") pod \"e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td\" (UID: \"bce8221a-1318-4f64-908c-d927050e1ac1\") " pod="openshift-ingress/e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td"
Oct 30 18:56:24.499220 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:24.499193    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bce8221a-1318-4f64-908c-d927050e1ac1-istio-podinfo\") pod \"e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td\" (UID: \"bce8221a-1318-4f64-908c-d927050e1ac1\") " pod="openshift-ingress/e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td"
Oct 30 18:56:24.518023 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:24.501311    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dc0f87-fdbdk"]
Oct 30 18:56:24.537894 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:24.534741    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bce8221a-1318-4f64-908c-d927050e1ac1-istio-envoy\") pod \"e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td\" (UID: \"bce8221a-1318-4f64-908c-d927050e1ac1\") " pod="openshift-ingress/e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td"
Oct 30 18:56:24.569649 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:24.550176    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f24h9\" (UniqueName: \"kubernetes.io/projected/bce8221a-1318-4f64-908c-d927050e1ac1-kube-api-access-f24h9\") pod \"e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td\" (UID: \"bce8221a-1318-4f64-908c-d927050e1ac1\") " pod="openshift-ingress/e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td"
Oct 30 18:56:24.598859 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:24.598791    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bce8221a-1318-4f64-908c-d927050e1ac1-istio-token\") pod \"e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td\" (UID: \"bce8221a-1318-4f64-908c-d927050e1ac1\") " pod="openshift-ingress/e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td"
Oct 30 18:56:24.662701 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:24.658010    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td"
Oct 30 18:56:25.164435 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:56:25.164396    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbce8221a_1318_4f64_908c_d927050e1ac1.slice/crio-28a17a675a4c4b80505ab34df5db5b9b2a5d0d17301ad5cec2306d03367869d0 WatchSource:0}: Error finding container 28a17a675a4c4b80505ab34df5db5b9b2a5d0d17301ad5cec2306d03367869d0: Status 404 returned error can't find the container with id 28a17a675a4c4b80505ab34df5db5b9b2a5d0d17301ad5cec2306d03367869d0
Oct 30 18:56:25.167420 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:25.167394    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td"]
Oct 30 18:56:25.167526 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:25.167437    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:56:25.167526 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:25.167504    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:56:25.167612 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:25.167536    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:56:25.549547 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:25.549463    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4c5cf74-22ff-47fc-a7d5-3f3eddd3b239" path="/var/lib/kubelet/pods/e4c5cf74-22ff-47fc-a7d5-3f3eddd3b239/volumes"
Oct 30 18:56:26.027598 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:26.027563    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td" event={"ID":"bce8221a-1318-4f64-908c-d927050e1ac1","Type":"ContainerStarted","Data":"8c925841a89466566a44959880d802bb2800584d1ab4b27d6b60ceb240d95868"}
Oct 30 18:56:26.027840 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:26.027626    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td" event={"ID":"bce8221a-1318-4f64-908c-d927050e1ac1","Type":"ContainerStarted","Data":"28a17a675a4c4b80505ab34df5db5b9b2a5d0d17301ad5cec2306d03367869d0"}
Oct 30 18:56:26.060501 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:26.060445    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td" podStartSLOduration=2.060430068 podStartE2EDuration="2.060430068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:56:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:56:26.058568826 +0000 UTC m=+1208.684238790" watchObservedRunningTime="2026-10-30 18:56:26.060430068 +0000 UTC m=+1208.686100032"
Oct 30 18:56:26.663706 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:26.658782    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td"
Oct 30 18:56:26.679188 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:26.675278    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td"
Oct 30 18:56:27.036197 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:27.036125    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td"
Oct 30 18:56:27.037082 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:27.037061    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td"
Oct 30 18:56:27.702063 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:27.699912    4525 patch_prober.go:29] interesting pod/e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 18:56:27.702063 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:27.700006    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx" podUID="2d4927ad-ead6-477f-8105-4cdd3daadea5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:56:28.045746 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:28.045718    4525 generic.go:356] "Generic (PLEG): container finished" podID="5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1" containerID="c18a57044ec8e94244aacce3bb29a64cfdd8474ebd9781aa20b51a70a84298c6" exitCode=0
Oct 30 18:56:28.045877 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:28.045793    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9" event={"ID":"5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1","Type":"ContainerDied","Data":"c18a57044ec8e94244aacce3bb29a64cfdd8474ebd9781aa20b51a70a84298c6"}
Oct 30 18:56:28.046308 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:28.046289    4525 scope.go:122] "RemoveContainer" containerID="c18a57044ec8e94244aacce3bb29a64cfdd8474ebd9781aa20b51a70a84298c6"
Oct 30 18:56:28.047622 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:28.047604    4525 generic.go:356] "Generic (PLEG): container finished" podID="2d4927ad-ead6-477f-8105-4cdd3daadea5" containerID="7024b7c83690f48ca73015a8dcfe2ab56c7816c4e0b933f9c0cff271502866ac" exitCode=0
Oct 30 18:56:28.047734 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:28.047691    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx" event={"ID":"2d4927ad-ead6-477f-8105-4cdd3daadea5","Type":"ContainerDied","Data":"7024b7c83690f48ca73015a8dcfe2ab56c7816c4e0b933f9c0cff271502866ac"}
Oct 30 18:56:28.048039 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:28.048015    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:56:28.048080 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:28.048063    4525 scope.go:122] "RemoveContainer" containerID="7024b7c83690f48ca73015a8dcfe2ab56c7816c4e0b933f9c0cff271502866ac"
Oct 30 18:56:28.048114 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:28.048077    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:56:28.048149 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:28.048124    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:56:28.049563 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:28.049532    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:56:28.049654 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:28.049635    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:56:28.049735 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:28.049718    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:56:28.777024 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:28.773394    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-12a6e1-openshift-default-5d85cdb88c-85478"]
Oct 30 18:56:28.785217 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:28.785181    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-12a6e1-openshift-default-5d85cdb88c-85478"
Oct 30 18:56:28.813153 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:28.787580    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-12a6e1-openshift-default-5d85cdb88c-85478"]
Oct 30 18:56:28.813153 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:28.796105    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-12a6e1-openshift-default-dockercfg-rrvcr\""
Oct 30 18:56:28.838372 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:28.838336    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245-workload-certs\") pod \"e2e-api-b-12a6e1-openshift-default-5d85cdb88c-85478\" (UID: \"cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245\") " pod="openshift-ingress/e2e-api-b-12a6e1-openshift-default-5d85cdb88c-85478"
Oct 30 18:56:28.838522 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:28.838404    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245-istio-podinfo\") pod \"e2e-api-b-12a6e1-openshift-default-5d85cdb88c-85478\" (UID: \"cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245\") " pod="openshift-ingress/e2e-api-b-12a6e1-openshift-default-5d85cdb88c-85478"
Oct 30 18:56:28.838658 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:28.838640    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245-workload-socket\") pod \"e2e-api-b-12a6e1-openshift-default-5d85cdb88c-85478\" (UID: \"cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245\") " pod="openshift-ingress/e2e-api-b-12a6e1-openshift-default-5d85cdb88c-85478"
Oct 30 18:56:28.838750 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:28.838671    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmbct\" (UniqueName: \"kubernetes.io/projected/cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245-kube-api-access-vmbct\") pod \"e2e-api-b-12a6e1-openshift-default-5d85cdb88c-85478\" (UID: \"cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245\") " pod="openshift-ingress/e2e-api-b-12a6e1-openshift-default-5d85cdb88c-85478"
Oct 30 18:56:28.838808 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:28.838783    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245-istio-token\") pod \"e2e-api-b-12a6e1-openshift-default-5d85cdb88c-85478\" (UID: \"cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245\") " pod="openshift-ingress/e2e-api-b-12a6e1-openshift-default-5d85cdb88c-85478"
Oct 30 18:56:28.838873 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:28.838829    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245-istio-envoy\") pod \"e2e-api-b-12a6e1-openshift-default-5d85cdb88c-85478\" (UID: \"cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245\") " pod="openshift-ingress/e2e-api-b-12a6e1-openshift-default-5d85cdb88c-85478"
Oct 30 18:56:28.838925 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:28.838873    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245-istio-data\") pod \"e2e-api-b-12a6e1-openshift-default-5d85cdb88c-85478\" (UID: \"cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245\") " pod="openshift-ingress/e2e-api-b-12a6e1-openshift-default-5d85cdb88c-85478"
Oct 30 18:56:28.838925 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:28.838919    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245-credential-socket\") pod \"e2e-api-b-12a6e1-openshift-default-5d85cdb88c-85478\" (UID: \"cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245\") " pod="openshift-ingress/e2e-api-b-12a6e1-openshift-default-5d85cdb88c-85478"
Oct 30 18:56:28.839034 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:28.838955    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245-istiod-ca-cert\") pod \"e2e-api-b-12a6e1-openshift-default-5d85cdb88c-85478\" (UID: \"cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245\") " pod="openshift-ingress/e2e-api-b-12a6e1-openshift-default-5d85cdb88c-85478"
Oct 30 18:56:28.945770 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:28.940644    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245-workload-socket\") pod \"e2e-api-b-12a6e1-openshift-default-5d85cdb88c-85478\" (UID: \"cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245\") " pod="openshift-ingress/e2e-api-b-12a6e1-openshift-default-5d85cdb88c-85478"
Oct 30 18:56:28.945770 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:28.940714    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmbct\" (UniqueName: \"kubernetes.io/projected/cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245-kube-api-access-vmbct\") pod \"e2e-api-b-12a6e1-openshift-default-5d85cdb88c-85478\" (UID: \"cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245\") " pod="openshift-ingress/e2e-api-b-12a6e1-openshift-default-5d85cdb88c-85478"
Oct 30 18:56:28.945770 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:28.940810    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245-istio-token\") pod \"e2e-api-b-12a6e1-openshift-default-5d85cdb88c-85478\" (UID: \"cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245\") " pod="openshift-ingress/e2e-api-b-12a6e1-openshift-default-5d85cdb88c-85478"
Oct 30 18:56:28.945770 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:28.940848    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245-istio-envoy\") pod \"e2e-api-b-12a6e1-openshift-default-5d85cdb88c-85478\" (UID: \"cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245\") " pod="openshift-ingress/e2e-api-b-12a6e1-openshift-default-5d85cdb88c-85478"
Oct 30 18:56:28.945770 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:28.940887    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245-istio-data\") pod \"e2e-api-b-12a6e1-openshift-default-5d85cdb88c-85478\" (UID: \"cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245\") " pod="openshift-ingress/e2e-api-b-12a6e1-openshift-default-5d85cdb88c-85478"
Oct 30 18:56:28.945770 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:28.940928    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245-credential-socket\") pod \"e2e-api-b-12a6e1-openshift-default-5d85cdb88c-85478\" (UID: \"cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245\") " pod="openshift-ingress/e2e-api-b-12a6e1-openshift-default-5d85cdb88c-85478"
Oct 30 18:56:28.945770 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:28.940957    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245-istiod-ca-cert\") pod \"e2e-api-b-12a6e1-openshift-default-5d85cdb88c-85478\" (UID: \"cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245\") " pod="openshift-ingress/e2e-api-b-12a6e1-openshift-default-5d85cdb88c-85478"
Oct 30 18:56:28.945770 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:28.941055    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245-workload-certs\") pod \"e2e-api-b-12a6e1-openshift-default-5d85cdb88c-85478\" (UID: \"cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245\") " pod="openshift-ingress/e2e-api-b-12a6e1-openshift-default-5d85cdb88c-85478"
Oct 30 18:56:28.945770 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:28.941111    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245-istio-podinfo\") pod \"e2e-api-b-12a6e1-openshift-default-5d85cdb88c-85478\" (UID: \"cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245\") " pod="openshift-ingress/e2e-api-b-12a6e1-openshift-default-5d85cdb88c-85478"
Oct 30 18:56:28.945770 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:28.942603    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245-workload-socket\") pod \"e2e-api-b-12a6e1-openshift-default-5d85cdb88c-85478\" (UID: \"cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245\") " pod="openshift-ingress/e2e-api-b-12a6e1-openshift-default-5d85cdb88c-85478"
Oct 30 18:56:28.945770 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:28.942885    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245-credential-socket\") pod \"e2e-api-b-12a6e1-openshift-default-5d85cdb88c-85478\" (UID: \"cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245\") " pod="openshift-ingress/e2e-api-b-12a6e1-openshift-default-5d85cdb88c-85478"
Oct 30 18:56:28.945770 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:28.943128    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245-istio-data\") pod \"e2e-api-b-12a6e1-openshift-default-5d85cdb88c-85478\" (UID: \"cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245\") " pod="openshift-ingress/e2e-api-b-12a6e1-openshift-default-5d85cdb88c-85478"
Oct 30 18:56:28.953852 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:28.953820    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245-istiod-ca-cert\") pod \"e2e-api-b-12a6e1-openshift-default-5d85cdb88c-85478\" (UID: \"cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245\") " pod="openshift-ingress/e2e-api-b-12a6e1-openshift-default-5d85cdb88c-85478"
Oct 30 18:56:28.954911 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:28.954885    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245-workload-certs\") pod \"e2e-api-b-12a6e1-openshift-default-5d85cdb88c-85478\" (UID: \"cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245\") " pod="openshift-ingress/e2e-api-b-12a6e1-openshift-default-5d85cdb88c-85478"
Oct 30 18:56:28.982718 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:28.982630    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmbct\" (UniqueName: \"kubernetes.io/projected/cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245-kube-api-access-vmbct\") pod \"e2e-api-b-12a6e1-openshift-default-5d85cdb88c-85478\" (UID: \"cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245\") " pod="openshift-ingress/e2e-api-b-12a6e1-openshift-default-5d85cdb88c-85478"
Oct 30 18:56:28.983197 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:28.983175    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245-istio-envoy\") pod \"e2e-api-b-12a6e1-openshift-default-5d85cdb88c-85478\" (UID: \"cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245\") " pod="openshift-ingress/e2e-api-b-12a6e1-openshift-default-5d85cdb88c-85478"
Oct 30 18:56:28.983842 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:28.983820    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245-istio-podinfo\") pod \"e2e-api-b-12a6e1-openshift-default-5d85cdb88c-85478\" (UID: \"cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245\") " pod="openshift-ingress/e2e-api-b-12a6e1-openshift-default-5d85cdb88c-85478"
Oct 30 18:56:28.998715 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:28.998663    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245-istio-token\") pod \"e2e-api-b-12a6e1-openshift-default-5d85cdb88c-85478\" (UID: \"cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245\") " pod="openshift-ingress/e2e-api-b-12a6e1-openshift-default-5d85cdb88c-85478"
Oct 30 18:56:29.141252 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:29.138940    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9" event={"ID":"5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1","Type":"ContainerStarted","Data":"86a63fef72d63a6becfdf391e043a8dead722c8899591dd7ff0429ec8d41cdfe"}
Oct 30 18:56:29.141252 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:29.139887    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-12a6e1-openshift-default-5d85cdb88c-85478"
Oct 30 18:56:29.188748 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:29.187263    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx" event={"ID":"2d4927ad-ead6-477f-8105-4cdd3daadea5","Type":"ContainerStarted","Data":"b994d5f9166561d9b27b39cbee3ed61b6ab8e978e9ee2edd0f183935b47f78df"}
Oct 30 18:56:29.213706 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:29.209170    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx"
Oct 30 18:56:29.213706 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:29.209277    4525 patch_prober.go:29] interesting pod/e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Oct 30 18:56:29.214356 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:29.214016    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx" podUID="2d4927ad-ead6-477f-8105-4cdd3daadea5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Oct 30 18:56:29.246270 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:29.246245    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9"
Oct 30 18:56:29.246625 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:29.246477    4525 patch_prober.go:29] interesting pod/e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body=
Oct 30 18:56:29.246856 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:29.246833    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9" podUID="5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused"
Oct 30 18:56:29.635405 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:29.635373    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-12a6e1-openshift-default-5d85cdb88c-85478"]
Oct 30 18:56:29.636828 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:56:29.636787    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf3d8ea6_d7eb_48ae_ab8a_9a07611fc245.slice/crio-dd8cd67351d8c9b3fabe2411e9f17056a5aae8b417ddeb1e561fc94143beaf87 WatchSource:0}: Error finding container dd8cd67351d8c9b3fabe2411e9f17056a5aae8b417ddeb1e561fc94143beaf87: Status 404 returned error can't find the container with id dd8cd67351d8c9b3fabe2411e9f17056a5aae8b417ddeb1e561fc94143beaf87
Oct 30 18:56:29.640188 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:29.640149    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:56:29.640351 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:29.640242    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:56:29.640351 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:29.640286    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:56:30.280828 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:30.280794    4525 patch_prober.go:29] interesting pod/e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Oct 30 18:56:30.356238 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:30.353288    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx" podUID="2d4927ad-ead6-477f-8105-4cdd3daadea5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Oct 30 18:56:30.356238 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:30.291952    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32a67598-29l4p"]
Oct 30 18:56:30.356238 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:30.301976    4525 patch_prober.go:29] interesting pod/e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body=
Oct 30 18:56:30.356238 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:30.353413    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9" podUID="5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused"
Oct 30 18:56:30.360943 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:30.360910    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-12a6e1-openshift-default-5d85cdb88c-85478" event={"ID":"cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245","Type":"ContainerStarted","Data":"ccf8abf7f63a34b18fa795c7b6e0a9a68ae6d15366245e90a8ef885ff6e026cf"}
Oct 30 18:56:30.361064 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:30.360973    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-12a6e1-openshift-default-5d85cdb88c-85478" event={"ID":"cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245","Type":"ContainerStarted","Data":"dd8cd67351d8c9b3fabe2411e9f17056a5aae8b417ddeb1e561fc94143beaf87"}
Oct 30 18:56:30.365292 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:30.365268    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32a67598-29l4p"]
Oct 30 18:56:30.455799 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:30.454476    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-12a6e1-openshift-default-5d85cdb88c-85478" podStartSLOduration=2.454454832 podStartE2EDuration="2.454454832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:56:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:56:30.435977772 +0000 UTC m=+1213.061647737" watchObservedRunningTime="2026-10-30 18:56:30.454454832 +0000 UTC m=+1213.080124796"
Oct 30 18:56:31.140416 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:31.140374    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-12a6e1-openshift-default-5d85cdb88c-85478"
Oct 30 18:56:31.140763 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:31.140737    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96c25bd9-7mwc6"]
Oct 30 18:56:31.147558 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:31.147528    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96c25bd9-7mwc6"]
Oct 30 18:56:31.148537 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:31.148519    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-12a6e1-openshift-default-5d85cdb88c-85478"
Oct 30 18:56:31.226419 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:31.226381    4525 patch_prober.go:29] interesting pod/e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Oct 30 18:56:31.226627 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:31.226461    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx" podUID="2d4927ad-ead6-477f-8105-4cdd3daadea5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Oct 30 18:56:31.246391 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:31.246365    4525 patch_prober.go:29] interesting pod/e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body=
Oct 30 18:56:31.246532 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:31.246418    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9" podUID="5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused"
Oct 30 18:56:31.369272 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:31.369243    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-12a6e1-openshift-default-5d85cdb88c-85478"
Oct 30 18:56:31.370037 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:31.370014    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-12a6e1-openshift-default-5d85cdb88c-85478"
Oct 30 18:56:31.556575 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:31.556494    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24fb6a8c-100c-43d8-ae55-d1b6d6fcb437" path="/var/lib/kubelet/pods/24fb6a8c-100c-43d8-ae55-d1b6d6fcb437/volumes"
Oct 30 18:56:31.557600 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:31.557574    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97a1b489-5957-4e3c-93fd-866399427ee0" path="/var/lib/kubelet/pods/97a1b489-5957-4e3c-93fd-866399427ee0/volumes"
Oct 30 18:56:32.233709 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:32.218099    4525 patch_prober.go:29] interesting pod/e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Oct 30 18:56:32.233709 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:32.218175    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx" podUID="2d4927ad-ead6-477f-8105-4cdd3daadea5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Oct 30 18:56:32.247446 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:32.246858    4525 patch_prober.go:29] interesting pod/e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body=
Oct 30 18:56:32.247446 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:32.246941    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9" podUID="5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused"
Oct 30 18:56:33.041966 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:33.041831    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9982ad-269q9"]
Oct 30 18:56:33.047993 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:33.047957    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9982ad-269q9"
Oct 30 18:56:33.057799 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:33.057771    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f2aca4d-b6k2n"]
Oct 30 18:56:33.057998 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:33.057925    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:56:33.058149 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:33.057939    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94hcn\""
Oct 30 18:56:33.061528 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:33.061509    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9982ad-269q9"]
Oct 30 18:56:33.069782 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:33.069756    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f2aca4d-b6k2n"]
Oct 30 18:56:33.215707 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:33.215112    4525 patch_prober.go:29] interesting pod/e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Oct 30 18:56:33.215707 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:33.215174    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx" podUID="2d4927ad-ead6-477f-8105-4cdd3daadea5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Oct 30 18:56:33.233653 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:33.233616    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qmlx\" (UniqueName: \"kubernetes.io/projected/8a500c5e-4436-4fe8-a298-a1f6be60d66b-kube-api-access-2qmlx\") pod \"maas-api-revoke-keys-e2e-api-b-9982ad-269q9\" (UID: \"8a500c5e-4436-4fe8-a298-a1f6be60d66b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9982ad-269q9"
Oct 30 18:56:33.233845 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:33.233792    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a500c5e-4436-4fe8-a298-a1f6be60d66b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-9982ad-269q9\" (UID: \"8a500c5e-4436-4fe8-a298-a1f6be60d66b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9982ad-269q9"
Oct 30 18:56:33.234027 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:33.233975    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a500c5e-4436-4fe8-a298-a1f6be60d66b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-9982ad-269q9\" (UID: \"8a500c5e-4436-4fe8-a298-a1f6be60d66b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9982ad-269q9"
Oct 30 18:56:33.246160 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:33.246126    4525 patch_prober.go:29] interesting pod/e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body=
Oct 30 18:56:33.246302 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:33.246180    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9" podUID="5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused"
Oct 30 18:56:33.335364 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:33.335276    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qmlx\" (UniqueName: \"kubernetes.io/projected/8a500c5e-4436-4fe8-a298-a1f6be60d66b-kube-api-access-2qmlx\") pod \"maas-api-revoke-keys-e2e-api-b-9982ad-269q9\" (UID: \"8a500c5e-4436-4fe8-a298-a1f6be60d66b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9982ad-269q9"
Oct 30 18:56:33.335542 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:33.335387    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a500c5e-4436-4fe8-a298-a1f6be60d66b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-9982ad-269q9\" (UID: \"8a500c5e-4436-4fe8-a298-a1f6be60d66b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9982ad-269q9"
Oct 30 18:56:33.335542 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:33.335445    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a500c5e-4436-4fe8-a298-a1f6be60d66b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-9982ad-269q9\" (UID: \"8a500c5e-4436-4fe8-a298-a1f6be60d66b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9982ad-269q9"
Oct 30 18:56:33.336280 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:33.336249    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a500c5e-4436-4fe8-a298-a1f6be60d66b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-9982ad-269q9\" (UID: \"8a500c5e-4436-4fe8-a298-a1f6be60d66b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9982ad-269q9"
Oct 30 18:56:33.338737 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:33.338706    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a500c5e-4436-4fe8-a298-a1f6be60d66b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-9982ad-269q9\" (UID: \"8a500c5e-4436-4fe8-a298-a1f6be60d66b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9982ad-269q9"
Oct 30 18:56:33.349864 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:33.349840    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qmlx\" (UniqueName: \"kubernetes.io/projected/8a500c5e-4436-4fe8-a298-a1f6be60d66b-kube-api-access-2qmlx\") pod \"maas-api-revoke-keys-e2e-api-b-9982ad-269q9\" (UID: \"8a500c5e-4436-4fe8-a298-a1f6be60d66b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9982ad-269q9"
Oct 30 18:56:33.375328 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:33.375305    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9982ad-269q9"
Oct 30 18:56:33.568503 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:33.566140    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13b3b9e4-817c-48c5-8803-c80d0cccb410" path="/var/lib/kubelet/pods/13b3b9e4-817c-48c5-8803-c80d0cccb410/volumes"
Oct 30 18:56:33.844927 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:33.844905    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9982ad-269q9"]
Oct 30 18:56:34.230825 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:34.230769    4525 patch_prober.go:29] interesting pod/e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Oct 30 18:56:34.231448 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:34.231388    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx" podUID="2d4927ad-ead6-477f-8105-4cdd3daadea5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Oct 30 18:56:34.246645 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:34.246606    4525 patch_prober.go:29] interesting pod/e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body=
Oct 30 18:56:34.246899 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:34.246878    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9" podUID="5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused"
Oct 30 18:56:34.724707 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:34.703523    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9982ad-269q9" event={"ID":"8a500c5e-4436-4fe8-a298-a1f6be60d66b","Type":"ContainerStarted","Data":"800b5fff6186ff365f07f3ff91dc0101b397897ffe04493b1bb4e12d3272de28"}
Oct 30 18:56:35.218850 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:35.208970    4525 patch_prober.go:29] interesting pod/e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Oct 30 18:56:35.218850 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:35.209032    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx" podUID="2d4927ad-ead6-477f-8105-4cdd3daadea5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Oct 30 18:56:35.260067 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:35.260027    4525 patch_prober.go:29] interesting pod/e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body=
Oct 30 18:56:35.260799 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:35.260768    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9" podUID="5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused"
Oct 30 18:56:35.465706 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:35.464123    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9"
Oct 30 18:56:35.718251 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:35.718213    4525 generic.go:356] "Generic (PLEG): container finished" podID="8a500c5e-4436-4fe8-a298-a1f6be60d66b" containerID="c49dd1bddf33ca71f80ed1baaf158d83718d9edd42cf13dfc708add4f17e0e04" exitCode=0
Oct 30 18:56:35.718614 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:35.718315    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9982ad-269q9" event={"ID":"8a500c5e-4436-4fe8-a298-a1f6be60d66b","Type":"ContainerDied","Data":"c49dd1bddf33ca71f80ed1baaf158d83718d9edd42cf13dfc708add4f17e0e04"}
Oct 30 18:56:36.213905 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:36.213875    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx"
Oct 30 18:56:36.214361 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:36.214334    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx"
Oct 30 18:56:36.214965 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:36.214947    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx"
Oct 30 18:56:36.251043 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:36.251013    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9"
Oct 30 18:56:36.251888 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:36.251865    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9"
Oct 30 18:56:36.975275 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:36.975238    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9982ad-269q9"
Oct 30 18:56:37.014165 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:37.014135    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a500c5e-4436-4fe8-a298-a1f6be60d66b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a500c5e-4436-4fe8-a298-a1f6be60d66b-maas-api-openshift-ca\") pod \"8a500c5e-4436-4fe8-a298-a1f6be60d66b\" (UID: \"8a500c5e-4436-4fe8-a298-a1f6be60d66b\") "
Oct 30 18:56:37.014301 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:37.014205    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a500c5e-4436-4fe8-a298-a1f6be60d66b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a500c5e-4436-4fe8-a298-a1f6be60d66b-maas-api-xks-ca\") pod \"8a500c5e-4436-4fe8-a298-a1f6be60d66b\" (UID: \"8a500c5e-4436-4fe8-a298-a1f6be60d66b\") "
Oct 30 18:56:37.014301 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:37.014234    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a500c5e-4436-4fe8-a298-a1f6be60d66b-kube-api-access-2qmlx\" (UniqueName: \"kubernetes.io/projected/8a500c5e-4436-4fe8-a298-a1f6be60d66b-kube-api-access-2qmlx\") pod \"8a500c5e-4436-4fe8-a298-a1f6be60d66b\" (UID: \"8a500c5e-4436-4fe8-a298-a1f6be60d66b\") "
Oct 30 18:56:37.014569 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:37.014542    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a500c5e-4436-4fe8-a298-a1f6be60d66b-maas-api-openshift-ca" pod "8a500c5e-4436-4fe8-a298-a1f6be60d66b" (UID: "8a500c5e-4436-4fe8-a298-a1f6be60d66b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:56:37.015032 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:37.015009    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a500c5e-4436-4fe8-a298-a1f6be60d66b-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:56:37.016440 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:37.016418    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a500c5e-4436-4fe8-a298-a1f6be60d66b-maas-api-xks-ca" pod "8a500c5e-4436-4fe8-a298-a1f6be60d66b" (UID: "8a500c5e-4436-4fe8-a298-a1f6be60d66b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:56:37.016532 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:37.016516    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a500c5e-4436-4fe8-a298-a1f6be60d66b-kube-api-access-2qmlx" pod "8a500c5e-4436-4fe8-a298-a1f6be60d66b" (UID: "8a500c5e-4436-4fe8-a298-a1f6be60d66b"). InnerVolumeSpecName "kube-api-access-2qmlx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:56:37.116815 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:37.116783    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a500c5e-4436-4fe8-a298-a1f6be60d66b-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:56:37.116815 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:37.116816    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qmlx\" (UniqueName: \"kubernetes.io/projected/8a500c5e-4436-4fe8-a298-a1f6be60d66b-kube-api-access-2qmlx\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:56:37.646714 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:37.645182    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-da58b769-rgc48"]
Oct 30 18:56:37.646912 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:37.646720    4525 cpu_manager.go:414] "containerMap: removing container" podUID="8a500c5e-4436-4fe8-a298-a1f6be60d66b" containerName="revoke-keys"
Oct 30 18:56:37.646912 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:37.646741    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a500c5e-4436-4fe8-a298-a1f6be60d66b" containerName="revoke-keys"
Oct 30 18:56:37.647088 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:37.647069    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a500c5e-4436-4fe8-a298-a1f6be60d66b" containerName="revoke-keys"
Oct 30 18:56:37.657229 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:37.657203    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-da58b769-rgc48"
Oct 30 18:56:37.663077 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:37.663050    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-da58b769-rgc48"]
Oct 30 18:56:37.739510 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:37.739474    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b92316a6-58ed-436d-bb91-4e41c7358128-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-da58b769-rgc48\" (UID: \"b92316a6-58ed-436d-bb91-4e41c7358128\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-da58b769-rgc48"
Oct 30 18:56:37.739729 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:37.739702    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4k79\" (UniqueName: \"kubernetes.io/projected/b92316a6-58ed-436d-bb91-4e41c7358128-kube-api-access-g4k79\") pod \"maas-api-revoke-keys-e2e-mt-da58b769-rgc48\" (UID: \"b92316a6-58ed-436d-bb91-4e41c7358128\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-da58b769-rgc48"
Oct 30 18:56:37.739875 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:37.739851    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b92316a6-58ed-436d-bb91-4e41c7358128-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-da58b769-rgc48\" (UID: \"b92316a6-58ed-436d-bb91-4e41c7358128\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-da58b769-rgc48"
Oct 30 18:56:37.780180 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:37.780150    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9982ad-269q9"
Oct 30 18:56:37.780372 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:37.780186    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9982ad-269q9" event={"ID":"8a500c5e-4436-4fe8-a298-a1f6be60d66b","Type":"ContainerDied","Data":"800b5fff6186ff365f07f3ff91dc0101b397897ffe04493b1bb4e12d3272de28"}
Oct 30 18:56:37.780372 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:37.780229    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="800b5fff6186ff365f07f3ff91dc0101b397897ffe04493b1bb4e12d3272de28"
Oct 30 18:56:37.840787 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:37.840663    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4k79\" (UniqueName: \"kubernetes.io/projected/b92316a6-58ed-436d-bb91-4e41c7358128-kube-api-access-g4k79\") pod \"maas-api-revoke-keys-e2e-mt-da58b769-rgc48\" (UID: \"b92316a6-58ed-436d-bb91-4e41c7358128\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-da58b769-rgc48"
Oct 30 18:56:37.840956 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:37.840826    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b92316a6-58ed-436d-bb91-4e41c7358128-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-da58b769-rgc48\" (UID: \"b92316a6-58ed-436d-bb91-4e41c7358128\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-da58b769-rgc48"
Oct 30 18:56:37.840956 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:37.840900    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b92316a6-58ed-436d-bb91-4e41c7358128-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-da58b769-rgc48\" (UID: \"b92316a6-58ed-436d-bb91-4e41c7358128\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-da58b769-rgc48"
Oct 30 18:56:37.841631 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:37.841611    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b92316a6-58ed-436d-bb91-4e41c7358128-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-da58b769-rgc48\" (UID: \"b92316a6-58ed-436d-bb91-4e41c7358128\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-da58b769-rgc48"
Oct 30 18:56:37.843463 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:37.843445    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b92316a6-58ed-436d-bb91-4e41c7358128-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-da58b769-rgc48\" (UID: \"b92316a6-58ed-436d-bb91-4e41c7358128\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-da58b769-rgc48"
Oct 30 18:56:37.848329 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:37.848311    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4k79\" (UniqueName: \"kubernetes.io/projected/b92316a6-58ed-436d-bb91-4e41c7358128-kube-api-access-g4k79\") pod \"maas-api-revoke-keys-e2e-mt-da58b769-rgc48\" (UID: \"b92316a6-58ed-436d-bb91-4e41c7358128\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-da58b769-rgc48"
Oct 30 18:56:37.975274 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:37.975247    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-da58b769-rgc48"
Oct 30 18:56:38.299760 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:56:38.299406    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb92316a6_58ed_436d_bb91_4e41c7358128.slice/crio-2164b506ec502465fdb4856157eaac6f34cf9b903ab89a6388ad63f115986665 WatchSource:0}: Error finding container 2164b506ec502465fdb4856157eaac6f34cf9b903ab89a6388ad63f115986665: Status 404 returned error can't find the container with id 2164b506ec502465fdb4856157eaac6f34cf9b903ab89a6388ad63f115986665
Oct 30 18:56:38.301130 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:38.301099    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-da58b769-rgc48"]
Oct 30 18:56:38.791206 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:38.791172    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-da58b769-rgc48" event={"ID":"b92316a6-58ed-436d-bb91-4e41c7358128","Type":"ContainerStarted","Data":"e5d771e47fd5c5b6ac5f11c1293eb2e4c1596f43efb936fc1f4e62f83c73b687"}
Oct 30 18:56:38.791390 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:38.791236    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-da58b769-rgc48" event={"ID":"b92316a6-58ed-436d-bb91-4e41c7358128","Type":"ContainerStarted","Data":"2164b506ec502465fdb4856157eaac6f34cf9b903ab89a6388ad63f115986665"}
Oct 30 18:56:38.810110 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:38.810051    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-da58b769-rgc48" podStartSLOduration=1.8100379709999999 podStartE2EDuration="1.810037971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:56:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:56:38.80601886 +0000 UTC m=+1221.431688825" watchObservedRunningTime="2026-10-30 18:56:38.810037971 +0000 UTC m=+1221.435707934"
Oct 30 18:56:39.801919 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:39.801880    4525 generic.go:356] "Generic (PLEG): container finished" podID="b92316a6-58ed-436d-bb91-4e41c7358128" containerID="e5d771e47fd5c5b6ac5f11c1293eb2e4c1596f43efb936fc1f4e62f83c73b687" exitCode=0
Oct 30 18:56:39.802435 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:39.801923    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-da58b769-rgc48" event={"ID":"b92316a6-58ed-436d-bb91-4e41c7358128","Type":"ContainerDied","Data":"e5d771e47fd5c5b6ac5f11c1293eb2e4c1596f43efb936fc1f4e62f83c73b687"}
Oct 30 18:56:41.694331 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:41.694298    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-da58b769-rgc48"
Oct 30 18:56:41.811483 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:41.811442    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b92316a6-58ed-436d-bb91-4e41c7358128-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b92316a6-58ed-436d-bb91-4e41c7358128-maas-api-openshift-ca\") pod \"b92316a6-58ed-436d-bb91-4e41c7358128\" (UID: \"b92316a6-58ed-436d-bb91-4e41c7358128\") "
Oct 30 18:56:41.811739 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:41.811565    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b92316a6-58ed-436d-bb91-4e41c7358128-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b92316a6-58ed-436d-bb91-4e41c7358128-maas-api-xks-ca\") pod \"b92316a6-58ed-436d-bb91-4e41c7358128\" (UID: \"b92316a6-58ed-436d-bb91-4e41c7358128\") "
Oct 30 18:56:41.811739 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:41.811640    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b92316a6-58ed-436d-bb91-4e41c7358128-kube-api-access-g4k79\" (UniqueName: \"kubernetes.io/projected/b92316a6-58ed-436d-bb91-4e41c7358128-kube-api-access-g4k79\") pod \"b92316a6-58ed-436d-bb91-4e41c7358128\" (UID: \"b92316a6-58ed-436d-bb91-4e41c7358128\") "
Oct 30 18:56:41.811885 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:41.811856    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b92316a6-58ed-436d-bb91-4e41c7358128-maas-api-openshift-ca" pod "b92316a6-58ed-436d-bb91-4e41c7358128" (UID: "b92316a6-58ed-436d-bb91-4e41c7358128"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:56:41.812600 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:41.812573    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b92316a6-58ed-436d-bb91-4e41c7358128-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:56:41.813952 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:41.813910    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b92316a6-58ed-436d-bb91-4e41c7358128-maas-api-xks-ca" pod "b92316a6-58ed-436d-bb91-4e41c7358128" (UID: "b92316a6-58ed-436d-bb91-4e41c7358128"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:56:41.814071 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:41.814018    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b92316a6-58ed-436d-bb91-4e41c7358128-kube-api-access-g4k79" pod "b92316a6-58ed-436d-bb91-4e41c7358128" (UID: "b92316a6-58ed-436d-bb91-4e41c7358128"). InnerVolumeSpecName "kube-api-access-g4k79". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:56:41.837376 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:41.837285    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-da58b769-rgc48" event={"ID":"b92316a6-58ed-436d-bb91-4e41c7358128","Type":"ContainerDied","Data":"2164b506ec502465fdb4856157eaac6f34cf9b903ab89a6388ad63f115986665"}
Oct 30 18:56:41.837580 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:41.837378    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2164b506ec502465fdb4856157eaac6f34cf9b903ab89a6388ad63f115986665"
Oct 30 18:56:41.837580 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:41.837450    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-da58b769-rgc48"
Oct 30 18:56:41.914251 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:41.914022    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b92316a6-58ed-436d-bb91-4e41c7358128-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:56:41.914251 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:41.914059    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4k79\" (UniqueName: \"kubernetes.io/projected/b92316a6-58ed-436d-bb91-4e41c7358128-kube-api-access-g4k79\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:56:42.043722 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:42.043527    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2721a29-vtccj"]
Oct 30 18:56:42.049768 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:42.049727    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2721a29-vtccj"]
Oct 30 18:56:43.591732 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:43.591671    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af853a58-3659-4146-85bc-b81f55b5e39a" path="/var/lib/kubelet/pods/af853a58-3659-4146-85bc-b81f55b5e39a/volumes"
Oct 30 18:56:47.398496 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:56:47.398457    4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d49l" containerName="extractor"
Oct 30 18:56:48.026389 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:48.026353    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e57547ca-tcbxq"]
Oct 30 18:56:48.032108 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:48.032071    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e57547ca-tcbxq"]
Oct 30 18:56:49.556921 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:49.556878    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="caf20f32-8f96-49cf-91a0-e4e651f6f48a" path="/var/lib/kubelet/pods/caf20f32-8f96-49cf-91a0-e4e651f6f48a/volumes"
Oct 30 18:56:50.935470 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:50.935430    4525 generic.go:356] "Generic (PLEG): container finished" podID="9c3b7950-7451-4f37-8119-f9574c3045f3" containerID="18a288522f8406d4217f3b2d58d5f6c7858fac117df2b308e65e0868deb47272" exitCode=0
Oct 30 18:56:50.935987 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:50.935476    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8f1c37-openshift-default-7cd9c57fdf-kwhps" event={"ID":"9c3b7950-7451-4f37-8119-f9574c3045f3","Type":"ContainerDied","Data":"18a288522f8406d4217f3b2d58d5f6c7858fac117df2b308e65e0868deb47272"}
Oct 30 18:56:50.936046 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:50.935984    4525 scope.go:122] "RemoveContainer" containerID="18a288522f8406d4217f3b2d58d5f6c7858fac117df2b308e65e0868deb47272"
Oct 30 18:56:50.937951 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:50.937919    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:56:50.938081 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:50.938007    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:56:50.938081 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:50.938055    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:56:51.946576 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:51.946538    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8f1c37-openshift-default-7cd9c57fdf-kwhps" event={"ID":"9c3b7950-7451-4f37-8119-f9574c3045f3","Type":"ContainerStarted","Data":"df8e6b96b307a54a195cec9e44b62ae7933fc6cee4c29828a7055634d15ed074"}
Oct 30 18:56:52.382666 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:52.376050    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-8f1c37-openshift-default-7cd9c57fdf-kwhps"
Oct 30 18:56:52.386833 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:52.386800    4525 patch_prober.go:29] interesting pod/e2e-api-a-8f1c37-openshift-default-7cd9c57fdf-kwhps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Oct 30 18:56:52.392644 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:52.392605    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8f1c37-openshift-default-7cd9c57fdf-kwhps" podUID="9c3b7950-7451-4f37-8119-f9574c3045f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Oct 30 18:56:52.734757 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:52.734642    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-da58b769-rgc48"]
Oct 30 18:56:52.739438 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:52.739406    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-da58b769-rgc48"]
Oct 30 18:56:53.028223 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:53.028141    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50aa509a-nx7xh"]
Oct 30 18:56:53.033327 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:53.033299    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50aa509a-nx7xh"]
Oct 30 18:56:53.350493 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:53.350408    4525 patch_prober.go:29] interesting pod/e2e-api-a-8f1c37-openshift-default-7cd9c57fdf-kwhps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Oct 30 18:56:53.350493 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:53.350466    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8f1c37-openshift-default-7cd9c57fdf-kwhps" podUID="9c3b7950-7451-4f37-8119-f9574c3045f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Oct 30 18:56:53.569285 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:53.569250    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="103a5ab0-3191-4bc8-9d7f-9bae95dd1fd2" path="/var/lib/kubelet/pods/103a5ab0-3191-4bc8-9d7f-9bae95dd1fd2/volumes"
Oct 30 18:56:53.569836 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:53.569819    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b92316a6-58ed-436d-bb91-4e41c7358128" path="/var/lib/kubelet/pods/b92316a6-58ed-436d-bb91-4e41c7358128/volumes"
Oct 30 18:56:54.059707 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:54.059184    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31dbf6f3-xhmf2"]
Oct 30 18:56:54.067513 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:54.067488    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31dbf6f3-xhmf2"]
Oct 30 18:56:54.350982 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:54.350900    4525 patch_prober.go:29] interesting pod/e2e-api-a-8f1c37-openshift-default-7cd9c57fdf-kwhps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Oct 30 18:56:54.350982 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:54.350950    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8f1c37-openshift-default-7cd9c57fdf-kwhps" podUID="9c3b7950-7451-4f37-8119-f9574c3045f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Oct 30 18:56:55.350601 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:55.350568    4525 patch_prober.go:29] interesting pod/e2e-api-a-8f1c37-openshift-default-7cd9c57fdf-kwhps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Oct 30 18:56:55.351027 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:55.350622    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8f1c37-openshift-default-7cd9c57fdf-kwhps" podUID="9c3b7950-7451-4f37-8119-f9574c3045f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Oct 30 18:56:55.546159 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:55.546124    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f42ffdee-b882-41e9-ae44-c8a3ba102853" path="/var/lib/kubelet/pods/f42ffdee-b882-41e9-ae44-c8a3ba102853/volumes"
Oct 30 18:56:56.350675 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:56.350640    4525 patch_prober.go:29] interesting pod/e2e-api-a-8f1c37-openshift-default-7cd9c57fdf-kwhps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Oct 30 18:56:56.351124 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:56.350743    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8f1c37-openshift-default-7cd9c57fdf-kwhps" podUID="9c3b7950-7451-4f37-8119-f9574c3045f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Oct 30 18:56:57.350134 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:57.350099    4525 patch_prober.go:29] interesting pod/e2e-api-a-8f1c37-openshift-default-7cd9c57fdf-kwhps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Oct 30 18:56:57.350350 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:57.350161    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8f1c37-openshift-default-7cd9c57fdf-kwhps" podUID="9c3b7950-7451-4f37-8119-f9574c3045f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Oct 30 18:56:58.365964 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:58.365921    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-8f1c37-openshift-default-7cd9c57fdf-kwhps"
Oct 30 18:56:58.366984 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:58.366940    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-8f1c37-openshift-default-7cd9c57fdf-kwhps"
Oct 30 18:56:58.367787 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:58.367760    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-8f1c37-openshift-default-7cd9c57fdf-kwhps"
Oct 30 18:56:59.016765 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:59.016733    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8f1c37-lvvpr"]
Oct 30 18:56:59.017333 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:59.017318    4525 cpu_manager.go:414] "containerMap: removing container" podUID="b92316a6-58ed-436d-bb91-4e41c7358128" containerName="revoke-keys"
Oct 30 18:56:59.017380 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:59.017335    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b92316a6-58ed-436d-bb91-4e41c7358128" containerName="revoke-keys"
Oct 30 18:56:59.017482 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:59.017473    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="b92316a6-58ed-436d-bb91-4e41c7358128" containerName="revoke-keys"
Oct 30 18:56:59.022762 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:59.022733    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8f1c37-lvvpr"
Oct 30 18:56:59.026811 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:59.026786    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:56:59.026939 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:59.026838    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94hcn\""
Oct 30 18:56:59.029146 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:59.029123    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8f1c37-lvvpr"]
Oct 30 18:56:59.092045 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:59.092011    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c97ea986-7c6e-448f-b9e4-3c212159631c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-8f1c37-lvvpr\" (UID: \"c97ea986-7c6e-448f-b9e4-3c212159631c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8f1c37-lvvpr"
Oct 30 18:56:59.092210 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:59.092153    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzm9f\" (UniqueName: \"kubernetes.io/projected/c97ea986-7c6e-448f-b9e4-3c212159631c-kube-api-access-wzm9f\") pod \"maas-api-revoke-keys-e2e-api-a-8f1c37-lvvpr\" (UID: \"c97ea986-7c6e-448f-b9e4-3c212159631c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8f1c37-lvvpr"
Oct 30 18:56:59.092295 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:59.092280    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c97ea986-7c6e-448f-b9e4-3c212159631c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-8f1c37-lvvpr\" (UID: \"c97ea986-7c6e-448f-b9e4-3c212159631c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8f1c37-lvvpr"
Oct 30 18:56:59.193454 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:59.193411    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzm9f\" (UniqueName: \"kubernetes.io/projected/c97ea986-7c6e-448f-b9e4-3c212159631c-kube-api-access-wzm9f\") pod \"maas-api-revoke-keys-e2e-api-a-8f1c37-lvvpr\" (UID: \"c97ea986-7c6e-448f-b9e4-3c212159631c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8f1c37-lvvpr"
Oct 30 18:56:59.193737 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:59.193635    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c97ea986-7c6e-448f-b9e4-3c212159631c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-8f1c37-lvvpr\" (UID: \"c97ea986-7c6e-448f-b9e4-3c212159631c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8f1c37-lvvpr"
Oct 30 18:56:59.193887 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:59.193871    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c97ea986-7c6e-448f-b9e4-3c212159631c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-8f1c37-lvvpr\" (UID: \"c97ea986-7c6e-448f-b9e4-3c212159631c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8f1c37-lvvpr"
Oct 30 18:56:59.194531 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:59.194506    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c97ea986-7c6e-448f-b9e4-3c212159631c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-8f1c37-lvvpr\" (UID: \"c97ea986-7c6e-448f-b9e4-3c212159631c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8f1c37-lvvpr"
Oct 30 18:56:59.196238 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:59.196216    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c97ea986-7c6e-448f-b9e4-3c212159631c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-8f1c37-lvvpr\" (UID: \"c97ea986-7c6e-448f-b9e4-3c212159631c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8f1c37-lvvpr"
Oct 30 18:56:59.201507 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:59.201484    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzm9f\" (UniqueName: \"kubernetes.io/projected/c97ea986-7c6e-448f-b9e4-3c212159631c-kube-api-access-wzm9f\") pod \"maas-api-revoke-keys-e2e-api-a-8f1c37-lvvpr\" (UID: \"c97ea986-7c6e-448f-b9e4-3c212159631c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8f1c37-lvvpr"
Oct 30 18:56:59.348240 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:59.348156    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8f1c37-lvvpr"
Oct 30 18:56:59.629251 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:56:59.629222    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc97ea986_7c6e_448f_b9e4_3c212159631c.slice/crio-34a4c1a18545eaf32e57d9e859c89324af8169eca9259f07dfa0b372856aff0e WatchSource:0}: Error finding container 34a4c1a18545eaf32e57d9e859c89324af8169eca9259f07dfa0b372856aff0e: Status 404 returned error can't find the container with id 34a4c1a18545eaf32e57d9e859c89324af8169eca9259f07dfa0b372856aff0e
Oct 30 18:56:59.630800 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:56:59.630758    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8f1c37-lvvpr"]
Oct 30 18:57:00.132829 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:00.132794    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8f1c37-lvvpr" event={"ID":"c97ea986-7c6e-448f-b9e4-3c212159631c","Type":"ContainerStarted","Data":"ba37595a8a500eed0b051adbe65314f2291f930670f1fac99ba4c0ee2f2a51ef"}
Oct 30 18:57:00.133013 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:00.132862    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8f1c37-lvvpr" event={"ID":"c97ea986-7c6e-448f-b9e4-3c212159631c","Type":"ContainerStarted","Data":"34a4c1a18545eaf32e57d9e859c89324af8169eca9259f07dfa0b372856aff0e"}
Oct 30 18:57:00.155614 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:00.155522    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8f1c37-lvvpr" podStartSLOduration=2.155508771 podStartE2EDuration="2.155508771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:56:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:57:00.149502679 +0000 UTC m=+1242.775172644" watchObservedRunningTime="2026-10-30 18:57:00.155508771 +0000 UTC m=+1242.781178754"
Oct 30 18:57:01.142615 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:01.142578    4525 generic.go:356] "Generic (PLEG): container finished" podID="c97ea986-7c6e-448f-b9e4-3c212159631c" containerID="ba37595a8a500eed0b051adbe65314f2291f930670f1fac99ba4c0ee2f2a51ef" exitCode=0
Oct 30 18:57:01.143060 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:01.142664    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8f1c37-lvvpr" event={"ID":"c97ea986-7c6e-448f-b9e4-3c212159631c","Type":"ContainerDied","Data":"ba37595a8a500eed0b051adbe65314f2291f930670f1fac99ba4c0ee2f2a51ef"}
Oct 30 18:57:02.254704 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:02.239749    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9982ad-269q9"]
Oct 30 18:57:02.272524 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:02.272485    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9982ad-269q9"]
Oct 30 18:57:02.634738 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:02.634695    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8f1c37-lvvpr"
Oct 30 18:57:02.775964 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:02.775878    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c97ea986-7c6e-448f-b9e4-3c212159631c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c97ea986-7c6e-448f-b9e4-3c212159631c-maas-api-xks-ca\") pod \"c97ea986-7c6e-448f-b9e4-3c212159631c\" (UID: \"c97ea986-7c6e-448f-b9e4-3c212159631c\") "
Oct 30 18:57:02.776139 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:02.775999    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c97ea986-7c6e-448f-b9e4-3c212159631c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c97ea986-7c6e-448f-b9e4-3c212159631c-maas-api-openshift-ca\") pod \"c97ea986-7c6e-448f-b9e4-3c212159631c\" (UID: \"c97ea986-7c6e-448f-b9e4-3c212159631c\") "
Oct 30 18:57:02.776139 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:02.776122    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c97ea986-7c6e-448f-b9e4-3c212159631c-kube-api-access-wzm9f\" (UniqueName: \"kubernetes.io/projected/c97ea986-7c6e-448f-b9e4-3c212159631c-kube-api-access-wzm9f\") pod \"c97ea986-7c6e-448f-b9e4-3c212159631c\" (UID: \"c97ea986-7c6e-448f-b9e4-3c212159631c\") "
Oct 30 18:57:02.776394 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:02.776371    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c97ea986-7c6e-448f-b9e4-3c212159631c-maas-api-openshift-ca" pod "c97ea986-7c6e-448f-b9e4-3c212159631c" (UID: "c97ea986-7c6e-448f-b9e4-3c212159631c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:57:02.776707 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:02.776671    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c97ea986-7c6e-448f-b9e4-3c212159631c-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:02.778287 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:02.778267    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c97ea986-7c6e-448f-b9e4-3c212159631c-maas-api-xks-ca" pod "c97ea986-7c6e-448f-b9e4-3c212159631c" (UID: "c97ea986-7c6e-448f-b9e4-3c212159631c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:57:02.778447 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:02.778426    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c97ea986-7c6e-448f-b9e4-3c212159631c-kube-api-access-wzm9f" pod "c97ea986-7c6e-448f-b9e4-3c212159631c" (UID: "c97ea986-7c6e-448f-b9e4-3c212159631c"). InnerVolumeSpecName "kube-api-access-wzm9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:57:02.877650 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:02.877611    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c97ea986-7c6e-448f-b9e4-3c212159631c-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:02.877650 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:02.877647    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzm9f\" (UniqueName: \"kubernetes.io/projected/c97ea986-7c6e-448f-b9e4-3c212159631c-kube-api-access-wzm9f\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:03.160988 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:03.160958    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8f1c37-lvvpr"
Oct 30 18:57:03.161797 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:03.160991    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8f1c37-lvvpr" event={"ID":"c97ea986-7c6e-448f-b9e4-3c212159631c","Type":"ContainerDied","Data":"34a4c1a18545eaf32e57d9e859c89324af8169eca9259f07dfa0b372856aff0e"}
Oct 30 18:57:03.161797 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:03.161039    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34a4c1a18545eaf32e57d9e859c89324af8169eca9259f07dfa0b372856aff0e"
Oct 30 18:57:03.582665 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:03.582565    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a500c5e-4436-4fe8-a298-a1f6be60d66b" path="/var/lib/kubelet/pods/8a500c5e-4436-4fe8-a298-a1f6be60d66b/volumes"
Oct 30 18:57:05.037229 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:05.037188    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac06a57d-kxzjp"]
Oct 30 18:57:05.042012 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:05.041984    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac06a57d-kxzjp"]
Oct 30 18:57:05.552783 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:05.552741    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc4bf99a-34e8-47f1-9380-9a0c9676423f" path="/var/lib/kubelet/pods/bc4bf99a-34e8-47f1-9380-9a0c9676423f/volumes"
Oct 30 18:57:07.033064 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:07.033024    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f61c55b0-w4hbz"]
Oct 30 18:57:07.040979 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:07.040943    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f61c55b0-w4hbz"]
Oct 30 18:57:07.545902 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:07.545872    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="558bb0e9-54ce-4848-92ca-1e65c3c05957" path="/var/lib/kubelet/pods/558bb0e9-54ce-4848-92ca-1e65c3c05957/volumes"
Oct 30 18:57:08.024991 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:08.024957    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e29de711-qpch8"]
Oct 30 18:57:08.029395 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:08.029359    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e29de711-qpch8"]
Oct 30 18:57:09.406715 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:09.376544    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-9982ad-openshift-default-6f478df445-gr62t"]
Oct 30 18:57:09.407969 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:09.407914    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-9982ad-openshift-default-6f478df445-gr62t" podUID="ed65886b-3abc-4b57-9206-8e0603411200" containerName="istio-proxy" containerID="cri-o://cbd730893f8345ba95b708f10b7c44fa10f0d9ca38bdc0a39e14e24a9626f3ae" gracePeriod=30
Oct 30 18:57:09.546988 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:09.546956    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="070b5c5e-06a4-4ecd-a12e-37ddf440f097" path="/var/lib/kubelet/pods/070b5c5e-06a4-4ecd-a12e-37ddf440f097/volumes"
Oct 30 18:57:13.078874 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:13.078825    4525 patch_prober.go:29] interesting pod/e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 18:57:13.079303 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:13.078901    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td" podUID="bce8221a-1318-4f64-908c-d927050e1ac1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:57:14.756895 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:14.756757    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9982ad-openshift-default-6f478df445-gr62t"
Oct 30 18:57:14.834384 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:14.834348    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed65886b-3abc-4b57-9206-8e0603411200-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed65886b-3abc-4b57-9206-8e0603411200-credential-socket\") pod \"ed65886b-3abc-4b57-9206-8e0603411200\" (UID: \"ed65886b-3abc-4b57-9206-8e0603411200\") "
Oct 30 18:57:14.834384 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:14.834397    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed65886b-3abc-4b57-9206-8e0603411200-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed65886b-3abc-4b57-9206-8e0603411200-workload-certs\") pod \"ed65886b-3abc-4b57-9206-8e0603411200\" (UID: \"ed65886b-3abc-4b57-9206-8e0603411200\") "
Oct 30 18:57:14.834657 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:14.834563    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ed65886b-3abc-4b57-9206-8e0603411200-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed65886b-3abc-4b57-9206-8e0603411200-istio-podinfo\") pod \"ed65886b-3abc-4b57-9206-8e0603411200\" (UID: \"ed65886b-3abc-4b57-9206-8e0603411200\") "
Oct 30 18:57:14.834657 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:14.834624    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed65886b-3abc-4b57-9206-8e0603411200-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed65886b-3abc-4b57-9206-8e0603411200-workload-socket\") pod \"ed65886b-3abc-4b57-9206-8e0603411200\" (UID: \"ed65886b-3abc-4b57-9206-8e0603411200\") "
Oct 30 18:57:14.834796 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:14.834710    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed65886b-3abc-4b57-9206-8e0603411200-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed65886b-3abc-4b57-9206-8e0603411200-istio-data\") pod \"ed65886b-3abc-4b57-9206-8e0603411200\" (UID: \"ed65886b-3abc-4b57-9206-8e0603411200\") "
Oct 30 18:57:14.834796 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:14.834727    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed65886b-3abc-4b57-9206-8e0603411200-credential-socket" pod "ed65886b-3abc-4b57-9206-8e0603411200" (UID: "ed65886b-3abc-4b57-9206-8e0603411200"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:57:14.834796 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:14.834741    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed65886b-3abc-4b57-9206-8e0603411200-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed65886b-3abc-4b57-9206-8e0603411200-istiod-ca-cert\") pod \"ed65886b-3abc-4b57-9206-8e0603411200\" (UID: \"ed65886b-3abc-4b57-9206-8e0603411200\") "
Oct 30 18:57:14.835379 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:14.834869    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed65886b-3abc-4b57-9206-8e0603411200-workload-certs" pod "ed65886b-3abc-4b57-9206-8e0603411200" (UID: "ed65886b-3abc-4b57-9206-8e0603411200"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:57:14.835379 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:14.834881    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed65886b-3abc-4b57-9206-8e0603411200-kube-api-access-vd5fl\" (UniqueName: \"kubernetes.io/projected/ed65886b-3abc-4b57-9206-8e0603411200-kube-api-access-vd5fl\") pod \"ed65886b-3abc-4b57-9206-8e0603411200\" (UID: \"ed65886b-3abc-4b57-9206-8e0603411200\") "
Oct 30 18:57:14.835379 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:14.835090    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed65886b-3abc-4b57-9206-8e0603411200-workload-socket" pod "ed65886b-3abc-4b57-9206-8e0603411200" (UID: "ed65886b-3abc-4b57-9206-8e0603411200"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:57:14.835379 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:14.835118    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed65886b-3abc-4b57-9206-8e0603411200-istiod-ca-cert" pod "ed65886b-3abc-4b57-9206-8e0603411200" (UID: "ed65886b-3abc-4b57-9206-8e0603411200"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:57:14.835379 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:14.835161    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed65886b-3abc-4b57-9206-8e0603411200-istio-data" pod "ed65886b-3abc-4b57-9206-8e0603411200" (UID: "ed65886b-3abc-4b57-9206-8e0603411200"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:57:14.835379 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:14.835243    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed65886b-3abc-4b57-9206-8e0603411200-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed65886b-3abc-4b57-9206-8e0603411200-istio-envoy\") pod \"ed65886b-3abc-4b57-9206-8e0603411200\" (UID: \"ed65886b-3abc-4b57-9206-8e0603411200\") "
Oct 30 18:57:14.835379 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:14.835289    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed65886b-3abc-4b57-9206-8e0603411200-istio-token\" (UniqueName: \"kubernetes.io/projected/ed65886b-3abc-4b57-9206-8e0603411200-istio-token\") pod \"ed65886b-3abc-4b57-9206-8e0603411200\" (UID: \"ed65886b-3abc-4b57-9206-8e0603411200\") "
Oct 30 18:57:14.836045 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:14.835987    4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed65886b-3abc-4b57-9206-8e0603411200-workload-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:14.836045 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:14.836012    4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed65886b-3abc-4b57-9206-8e0603411200-istio-data\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:14.836045 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:14.836027    4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed65886b-3abc-4b57-9206-8e0603411200-istiod-ca-cert\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:14.836045 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:14.836041    4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed65886b-3abc-4b57-9206-8e0603411200-credential-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:14.836298 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:14.836055    4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed65886b-3abc-4b57-9206-8e0603411200-workload-certs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:14.838146 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:14.838116    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed65886b-3abc-4b57-9206-8e0603411200-kube-api-access-vd5fl" pod "ed65886b-3abc-4b57-9206-8e0603411200" (UID: "ed65886b-3abc-4b57-9206-8e0603411200"). InnerVolumeSpecName "kube-api-access-vd5fl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:57:14.838331 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:14.838307    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ed65886b-3abc-4b57-9206-8e0603411200-istio-podinfo" pod "ed65886b-3abc-4b57-9206-8e0603411200" (UID: "ed65886b-3abc-4b57-9206-8e0603411200"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 18:57:14.842474 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:14.838703    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed65886b-3abc-4b57-9206-8e0603411200-istio-token" pod "ed65886b-3abc-4b57-9206-8e0603411200" (UID: "ed65886b-3abc-4b57-9206-8e0603411200"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:57:14.842474 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:14.842413    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed65886b-3abc-4b57-9206-8e0603411200-istio-envoy" pod "ed65886b-3abc-4b57-9206-8e0603411200" (UID: "ed65886b-3abc-4b57-9206-8e0603411200"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:57:14.937161 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:14.937131    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vd5fl\" (UniqueName: \"kubernetes.io/projected/ed65886b-3abc-4b57-9206-8e0603411200-kube-api-access-vd5fl\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:14.937161 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:14.937158    4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed65886b-3abc-4b57-9206-8e0603411200-istio-envoy\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:14.937161 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:14.937170    4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed65886b-3abc-4b57-9206-8e0603411200-istio-token\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:14.937382 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:14.937179    4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed65886b-3abc-4b57-9206-8e0603411200-istio-podinfo\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:15.408792 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:15.408762    4525 generic.go:356] "Generic (PLEG): container finished" podID="ed65886b-3abc-4b57-9206-8e0603411200" containerID="cbd730893f8345ba95b708f10b7c44fa10f0d9ca38bdc0a39e14e24a9626f3ae" exitCode=0
Oct 30 18:57:15.412915 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:15.408842    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9982ad-openshift-default-6f478df445-gr62t"
Oct 30 18:57:15.412915 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:15.408834    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9982ad-openshift-default-6f478df445-gr62t" event={"ID":"ed65886b-3abc-4b57-9206-8e0603411200","Type":"ContainerDied","Data":"cbd730893f8345ba95b708f10b7c44fa10f0d9ca38bdc0a39e14e24a9626f3ae"}
Oct 30 18:57:15.412915 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:15.408968    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9982ad-openshift-default-6f478df445-gr62t" event={"ID":"ed65886b-3abc-4b57-9206-8e0603411200","Type":"ContainerDied","Data":"49658b08c9f13adb96358104bd4b7ddbd8f59d9918922281b2836fb4199e4a2e"}
Oct 30 18:57:15.412915 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:15.408998    4525 scope.go:122] "RemoveContainer" containerID="cbd730893f8345ba95b708f10b7c44fa10f0d9ca38bdc0a39e14e24a9626f3ae"
Oct 30 18:57:15.428147 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:15.428122    4525 scope.go:122] "RemoveContainer" containerID="68154ca9a57e6d106cc15aab5cacacf31c1c64634f8c4eba89fa726c0adcb905"
Oct 30 18:57:15.443476 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:15.443449    4525 scope.go:122] "RemoveContainer" containerID="cbd730893f8345ba95b708f10b7c44fa10f0d9ca38bdc0a39e14e24a9626f3ae"
Oct 30 18:57:15.445948 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:57:15.445471    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbd730893f8345ba95b708f10b7c44fa10f0d9ca38bdc0a39e14e24a9626f3ae\": container with ID starting with cbd730893f8345ba95b708f10b7c44fa10f0d9ca38bdc0a39e14e24a9626f3ae not found: ID does not exist" containerID="cbd730893f8345ba95b708f10b7c44fa10f0d9ca38bdc0a39e14e24a9626f3ae"
Oct 30 18:57:15.445948 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:15.445514    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbd730893f8345ba95b708f10b7c44fa10f0d9ca38bdc0a39e14e24a9626f3ae"} err="failed to get container status \"cbd730893f8345ba95b708f10b7c44fa10f0d9ca38bdc0a39e14e24a9626f3ae\": rpc error: code = NotFound desc = could not find container \"cbd730893f8345ba95b708f10b7c44fa10f0d9ca38bdc0a39e14e24a9626f3ae\": container with ID starting with cbd730893f8345ba95b708f10b7c44fa10f0d9ca38bdc0a39e14e24a9626f3ae not found: ID does not exist"
Oct 30 18:57:15.445948 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:15.445539    4525 scope.go:122] "RemoveContainer" containerID="68154ca9a57e6d106cc15aab5cacacf31c1c64634f8c4eba89fa726c0adcb905"
Oct 30 18:57:15.445948 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:57:15.445919    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68154ca9a57e6d106cc15aab5cacacf31c1c64634f8c4eba89fa726c0adcb905\": container with ID starting with 68154ca9a57e6d106cc15aab5cacacf31c1c64634f8c4eba89fa726c0adcb905 not found: ID does not exist" containerID="68154ca9a57e6d106cc15aab5cacacf31c1c64634f8c4eba89fa726c0adcb905"
Oct 30 18:57:15.446234 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:15.445952    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68154ca9a57e6d106cc15aab5cacacf31c1c64634f8c4eba89fa726c0adcb905"} err="failed to get container status \"68154ca9a57e6d106cc15aab5cacacf31c1c64634f8c4eba89fa726c0adcb905\": rpc error: code = NotFound desc = could not find container \"68154ca9a57e6d106cc15aab5cacacf31c1c64634f8c4eba89fa726c0adcb905\": container with ID starting with 68154ca9a57e6d106cc15aab5cacacf31c1c64634f8c4eba89fa726c0adcb905 not found: ID does not exist"
Oct 30 18:57:15.446619 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:15.446601    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-9982ad-openshift-default-6f478df445-gr62t"]
Oct 30 18:57:15.453323 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:15.453272    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-9982ad-openshift-default-6f478df445-gr62t"]
Oct 30 18:57:15.546056 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:15.546026    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed65886b-3abc-4b57-9206-8e0603411200" path="/var/lib/kubelet/pods/ed65886b-3abc-4b57-9206-8e0603411200/volumes"
Oct 30 18:57:16.350672 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:16.350639    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-54bc2b-openshift-default-bf84f77dc-jqfxp"]
Oct 30 18:57:16.352295 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:16.352266    4525 cpu_manager.go:414] "containerMap: removing container" podUID="c97ea986-7c6e-448f-b9e4-3c212159631c" containerName="revoke-keys"
Oct 30 18:57:16.352456 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:16.352296    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c97ea986-7c6e-448f-b9e4-3c212159631c" containerName="revoke-keys"
Oct 30 18:57:16.352456 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:16.352332    4525 cpu_manager.go:414] "containerMap: removing container" podUID="ed65886b-3abc-4b57-9206-8e0603411200" containerName="istio-proxy"
Oct 30 18:57:16.352456 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:16.352341    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed65886b-3abc-4b57-9206-8e0603411200" containerName="istio-proxy"
Oct 30 18:57:16.352456 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:16.352357    4525 cpu_manager.go:414] "containerMap: removing container" podUID="ed65886b-3abc-4b57-9206-8e0603411200" containerName="istio-proxy"
Oct 30 18:57:16.352456 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:16.352364    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed65886b-3abc-4b57-9206-8e0603411200" containerName="istio-proxy"
Oct 30 18:57:16.352795 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:16.352521    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed65886b-3abc-4b57-9206-8e0603411200" containerName="istio-proxy"
Oct 30 18:57:16.352795 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:16.352537    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed65886b-3abc-4b57-9206-8e0603411200" containerName="istio-proxy"
Oct 30 18:57:16.352795 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:16.352545    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="c97ea986-7c6e-448f-b9e4-3c212159631c" containerName="revoke-keys"
Oct 30 18:57:16.360177 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:16.360154    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-54bc2b-openshift-default-bf84f77dc-jqfxp"
Oct 30 18:57:16.367971 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:16.367944    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-54bc2b-openshift-default-dockercfg-ttgqg\""
Oct 30 18:57:16.369056 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:16.369034    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-54bc2b-openshift-default-bf84f77dc-jqfxp"]
Oct 30 18:57:16.424004 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:16.423970    4525 generic.go:356] "Generic (PLEG): container finished" podID="bce8221a-1318-4f64-908c-d927050e1ac1" containerID="8c925841a89466566a44959880d802bb2800584d1ab4b27d6b60ceb240d95868" exitCode=0
Oct 30 18:57:16.424189 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:16.424043    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td" event={"ID":"bce8221a-1318-4f64-908c-d927050e1ac1","Type":"ContainerDied","Data":"8c925841a89466566a44959880d802bb2800584d1ab4b27d6b60ceb240d95868"}
Oct 30 18:57:16.424543 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:16.424519    4525 scope.go:122] "RemoveContainer" containerID="8c925841a89466566a44959880d802bb2800584d1ab4b27d6b60ceb240d95868"
Oct 30 18:57:16.426573 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:16.426541    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:57:16.426648 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:16.426625    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:57:16.426718 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:16.426672    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:57:16.481804 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:16.479300    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9aca1683-2018-4248-8c22-bd7a46096df3-istio-token\") pod \"e2e-ipp-a-54bc2b-openshift-default-bf84f77dc-jqfxp\" (UID: \"9aca1683-2018-4248-8c22-bd7a46096df3\") " pod="openshift-ingress/e2e-ipp-a-54bc2b-openshift-default-bf84f77dc-jqfxp"
Oct 30 18:57:16.481804 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:16.479480    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9aca1683-2018-4248-8c22-bd7a46096df3-istio-data\") pod \"e2e-ipp-a-54bc2b-openshift-default-bf84f77dc-jqfxp\" (UID: \"9aca1683-2018-4248-8c22-bd7a46096df3\") " pod="openshift-ingress/e2e-ipp-a-54bc2b-openshift-default-bf84f77dc-jqfxp"
Oct 30 18:57:16.481804 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:16.479511    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9aca1683-2018-4248-8c22-bd7a46096df3-istio-podinfo\") pod \"e2e-ipp-a-54bc2b-openshift-default-bf84f77dc-jqfxp\" (UID: \"9aca1683-2018-4248-8c22-bd7a46096df3\") " pod="openshift-ingress/e2e-ipp-a-54bc2b-openshift-default-bf84f77dc-jqfxp"
Oct 30 18:57:16.481804 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:16.479597    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9aca1683-2018-4248-8c22-bd7a46096df3-istiod-ca-cert\") pod \"e2e-ipp-a-54bc2b-openshift-default-bf84f77dc-jqfxp\" (UID: \"9aca1683-2018-4248-8c22-bd7a46096df3\") " pod="openshift-ingress/e2e-ipp-a-54bc2b-openshift-default-bf84f77dc-jqfxp"
Oct 30 18:57:16.481804 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:16.479644    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9aca1683-2018-4248-8c22-bd7a46096df3-workload-certs\") pod \"e2e-ipp-a-54bc2b-openshift-default-bf84f77dc-jqfxp\" (UID: \"9aca1683-2018-4248-8c22-bd7a46096df3\") " pod="openshift-ingress/e2e-ipp-a-54bc2b-openshift-default-bf84f77dc-jqfxp"
Oct 30 18:57:16.481804 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:16.479755    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km8bm\" (UniqueName: \"kubernetes.io/projected/9aca1683-2018-4248-8c22-bd7a46096df3-kube-api-access-km8bm\") pod \"e2e-ipp-a-54bc2b-openshift-default-bf84f77dc-jqfxp\" (UID: \"9aca1683-2018-4248-8c22-bd7a46096df3\") " pod="openshift-ingress/e2e-ipp-a-54bc2b-openshift-default-bf84f77dc-jqfxp"
Oct 30 18:57:16.481804 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:16.479787    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aca1683-2018-4248-8c22-bd7a46096df3-credential-socket\") pod \"e2e-ipp-a-54bc2b-openshift-default-bf84f77dc-jqfxp\" (UID: \"9aca1683-2018-4248-8c22-bd7a46096df3\") " pod="openshift-ingress/e2e-ipp-a-54bc2b-openshift-default-bf84f77dc-jqfxp"
Oct 30 18:57:16.481804 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:16.479816    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9aca1683-2018-4248-8c22-bd7a46096df3-istio-envoy\") pod \"e2e-ipp-a-54bc2b-openshift-default-bf84f77dc-jqfxp\" (UID: \"9aca1683-2018-4248-8c22-bd7a46096df3\") " pod="openshift-ingress/e2e-ipp-a-54bc2b-openshift-default-bf84f77dc-jqfxp"
Oct 30 18:57:16.481804 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:16.479843    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aca1683-2018-4248-8c22-bd7a46096df3-workload-socket\") pod \"e2e-ipp-a-54bc2b-openshift-default-bf84f77dc-jqfxp\" (UID: \"9aca1683-2018-4248-8c22-bd7a46096df3\") " pod="openshift-ingress/e2e-ipp-a-54bc2b-openshift-default-bf84f77dc-jqfxp"
Oct 30 18:57:16.751744 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:16.749934    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9aca1683-2018-4248-8c22-bd7a46096df3-istiod-ca-cert\") pod \"e2e-ipp-a-54bc2b-openshift-default-bf84f77dc-jqfxp\" (UID: \"9aca1683-2018-4248-8c22-bd7a46096df3\") " pod="openshift-ingress/e2e-ipp-a-54bc2b-openshift-default-bf84f77dc-jqfxp"
Oct 30 18:57:16.751744 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:16.750004    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9aca1683-2018-4248-8c22-bd7a46096df3-workload-certs\") pod \"e2e-ipp-a-54bc2b-openshift-default-bf84f77dc-jqfxp\" (UID: \"9aca1683-2018-4248-8c22-bd7a46096df3\") " pod="openshift-ingress/e2e-ipp-a-54bc2b-openshift-default-bf84f77dc-jqfxp"
Oct 30 18:57:16.751744 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:16.750078    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km8bm\" (UniqueName: \"kubernetes.io/projected/9aca1683-2018-4248-8c22-bd7a46096df3-kube-api-access-km8bm\") pod \"e2e-ipp-a-54bc2b-openshift-default-bf84f77dc-jqfxp\" (UID: \"9aca1683-2018-4248-8c22-bd7a46096df3\") " pod="openshift-ingress/e2e-ipp-a-54bc2b-openshift-default-bf84f77dc-jqfxp"
Oct 30 18:57:16.751744 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:16.750107    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aca1683-2018-4248-8c22-bd7a46096df3-credential-socket\") pod \"e2e-ipp-a-54bc2b-openshift-default-bf84f77dc-jqfxp\" (UID: \"9aca1683-2018-4248-8c22-bd7a46096df3\") " pod="openshift-ingress/e2e-ipp-a-54bc2b-openshift-default-bf84f77dc-jqfxp"
Oct 30 18:57:16.751744 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:16.750141    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9aca1683-2018-4248-8c22-bd7a46096df3-istio-envoy\") pod \"e2e-ipp-a-54bc2b-openshift-default-bf84f77dc-jqfxp\" (UID: \"9aca1683-2018-4248-8c22-bd7a46096df3\") " pod="openshift-ingress/e2e-ipp-a-54bc2b-openshift-default-bf84f77dc-jqfxp"
Oct 30 18:57:16.751744 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:16.750167    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aca1683-2018-4248-8c22-bd7a46096df3-workload-socket\") pod \"e2e-ipp-a-54bc2b-openshift-default-bf84f77dc-jqfxp\" (UID: \"9aca1683-2018-4248-8c22-bd7a46096df3\") " pod="openshift-ingress/e2e-ipp-a-54bc2b-openshift-default-bf84f77dc-jqfxp"
Oct 30 18:57:16.751744 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:16.750387    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9aca1683-2018-4248-8c22-bd7a46096df3-istio-token\") pod \"e2e-ipp-a-54bc2b-openshift-default-bf84f77dc-jqfxp\" (UID: \"9aca1683-2018-4248-8c22-bd7a46096df3\") " pod="openshift-ingress/e2e-ipp-a-54bc2b-openshift-default-bf84f77dc-jqfxp"
Oct 30 18:57:16.751744 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:16.750567    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9aca1683-2018-4248-8c22-bd7a46096df3-istio-data\") pod \"e2e-ipp-a-54bc2b-openshift-default-bf84f77dc-jqfxp\" (UID: \"9aca1683-2018-4248-8c22-bd7a46096df3\") " pod="openshift-ingress/e2e-ipp-a-54bc2b-openshift-default-bf84f77dc-jqfxp"
Oct 30 18:57:16.751744 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:16.750593    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9aca1683-2018-4248-8c22-bd7a46096df3-istio-podinfo\") pod \"e2e-ipp-a-54bc2b-openshift-default-bf84f77dc-jqfxp\" (UID: \"9aca1683-2018-4248-8c22-bd7a46096df3\") " pod="openshift-ingress/e2e-ipp-a-54bc2b-openshift-default-bf84f77dc-jqfxp"
Oct 30 18:57:16.751744 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:16.751110    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aca1683-2018-4248-8c22-bd7a46096df3-workload-socket\") pod \"e2e-ipp-a-54bc2b-openshift-default-bf84f77dc-jqfxp\" (UID: \"9aca1683-2018-4248-8c22-bd7a46096df3\") " pod="openshift-ingress/e2e-ipp-a-54bc2b-openshift-default-bf84f77dc-jqfxp"
Oct 30 18:57:16.751744 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:16.751197    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9aca1683-2018-4248-8c22-bd7a46096df3-workload-certs\") pod \"e2e-ipp-a-54bc2b-openshift-default-bf84f77dc-jqfxp\" (UID: \"9aca1683-2018-4248-8c22-bd7a46096df3\") " pod="openshift-ingress/e2e-ipp-a-54bc2b-openshift-default-bf84f77dc-jqfxp"
Oct 30 18:57:16.751744 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:16.751347    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9aca1683-2018-4248-8c22-bd7a46096df3-istiod-ca-cert\") pod \"e2e-ipp-a-54bc2b-openshift-default-bf84f77dc-jqfxp\" (UID: \"9aca1683-2018-4248-8c22-bd7a46096df3\") " pod="openshift-ingress/e2e-ipp-a-54bc2b-openshift-default-bf84f77dc-jqfxp"
Oct 30 18:57:16.752495 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:16.750567    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aca1683-2018-4248-8c22-bd7a46096df3-credential-socket\") pod \"e2e-ipp-a-54bc2b-openshift-default-bf84f77dc-jqfxp\" (UID: \"9aca1683-2018-4248-8c22-bd7a46096df3\") " pod="openshift-ingress/e2e-ipp-a-54bc2b-openshift-default-bf84f77dc-jqfxp"
Oct 30 18:57:16.752550 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:16.752529    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9aca1683-2018-4248-8c22-bd7a46096df3-istio-data\") pod \"e2e-ipp-a-54bc2b-openshift-default-bf84f77dc-jqfxp\" (UID: \"9aca1683-2018-4248-8c22-bd7a46096df3\") " pod="openshift-ingress/e2e-ipp-a-54bc2b-openshift-default-bf84f77dc-jqfxp"
Oct 30 18:57:16.754674 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:16.754651    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9aca1683-2018-4248-8c22-bd7a46096df3-istio-podinfo\") pod \"e2e-ipp-a-54bc2b-openshift-default-bf84f77dc-jqfxp\" (UID: \"9aca1683-2018-4248-8c22-bd7a46096df3\") " pod="openshift-ingress/e2e-ipp-a-54bc2b-openshift-default-bf84f77dc-jqfxp"
Oct 30 18:57:16.771846 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:16.769214    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9aca1683-2018-4248-8c22-bd7a46096df3-istio-envoy\") pod \"e2e-ipp-a-54bc2b-openshift-default-bf84f77dc-jqfxp\" (UID: \"9aca1683-2018-4248-8c22-bd7a46096df3\") " pod="openshift-ingress/e2e-ipp-a-54bc2b-openshift-default-bf84f77dc-jqfxp"
Oct 30 18:57:16.776302 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:16.773573    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-km8bm\" (UniqueName: \"kubernetes.io/projected/9aca1683-2018-4248-8c22-bd7a46096df3-kube-api-access-km8bm\") pod \"e2e-ipp-a-54bc2b-openshift-default-bf84f77dc-jqfxp\" (UID: \"9aca1683-2018-4248-8c22-bd7a46096df3\") " pod="openshift-ingress/e2e-ipp-a-54bc2b-openshift-default-bf84f77dc-jqfxp"
Oct 30 18:57:16.776302 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:16.775002    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9aca1683-2018-4248-8c22-bd7a46096df3-istio-token\") pod \"e2e-ipp-a-54bc2b-openshift-default-bf84f77dc-jqfxp\" (UID: \"9aca1683-2018-4248-8c22-bd7a46096df3\") " pod="openshift-ingress/e2e-ipp-a-54bc2b-openshift-default-bf84f77dc-jqfxp"
Oct 30 18:57:16.981492 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:16.981408    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-54bc2b-openshift-default-bf84f77dc-jqfxp"
Oct 30 18:57:17.270901 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:57:17.270853    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9aca1683_2018_4248_8c22_bd7a46096df3.slice/crio-ea2e05ea2c06ad2ee09ef69230f2b32fefa54ee0bda7335cc38a6d8d2a374d79 WatchSource:0}: Error finding container ea2e05ea2c06ad2ee09ef69230f2b32fefa54ee0bda7335cc38a6d8d2a374d79: Status 404 returned error can't find the container with id ea2e05ea2c06ad2ee09ef69230f2b32fefa54ee0bda7335cc38a6d8d2a374d79
Oct 30 18:57:17.276843 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:17.276746    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-54bc2b-openshift-default-bf84f77dc-jqfxp"]
Oct 30 18:57:17.283203 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:17.283155    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:57:17.283343 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:17.283246    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:57:17.283343 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:17.283289    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:57:17.425951 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:57:17.425919    4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d49l" containerName="extractor"
Oct 30 18:57:17.441282 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:17.441245    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td" event={"ID":"bce8221a-1318-4f64-908c-d927050e1ac1","Type":"ContainerStarted","Data":"8c7dee88d79b8b21cc93e60b2020ff2e7eddbbba8640bb638b4c76b7e68e12d5"}
Oct 30 18:57:17.443492 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:17.443457    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-54bc2b-openshift-default-bf84f77dc-jqfxp" event={"ID":"9aca1683-2018-4248-8c22-bd7a46096df3","Type":"ContainerStarted","Data":"2173c9aa51f156466dcd6a03ba7bb121bb8e74f6380583fcb039eb2bcd650caf"}
Oct 30 18:57:17.443636 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:17.443519    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-54bc2b-openshift-default-bf84f77dc-jqfxp" event={"ID":"9aca1683-2018-4248-8c22-bd7a46096df3","Type":"ContainerStarted","Data":"ea2e05ea2c06ad2ee09ef69230f2b32fefa54ee0bda7335cc38a6d8d2a374d79"}
Oct 30 18:57:17.515049 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:17.514987    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-54bc2b-openshift-default-bf84f77dc-jqfxp" podStartSLOduration=1.5149619429999999 podStartE2EDuration="1.514961943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:57:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:57:17.49430161 +0000 UTC m=+1260.119971575" watchObservedRunningTime="2026-10-30 18:57:17.514961943 +0000 UTC m=+1260.140631913"
Oct 30 18:57:17.664041 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:17.659132    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td"
Oct 30 18:57:17.664041 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:17.659282    4525 patch_prober.go:29] interesting pod/e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Oct 30 18:57:17.664041 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:17.659331    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td" podUID="bce8221a-1318-4f64-908c-d927050e1ac1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Oct 30 18:57:17.981909 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:17.981811    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-54bc2b-openshift-default-bf84f77dc-jqfxp"
Oct 30 18:57:18.531606 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:18.531578    4525 scope.go:122] "RemoveContainer" containerID="94693d8f259404209b001c5a7b33d3e73316681d5af18545a241feff6acfc4dd"
Oct 30 18:57:18.543736 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:18.543712    4525 scope.go:122] "RemoveContainer" containerID="8eb32162ab54c2160093181a6f3dc189bbde406f1530c47fbc347f9b0477976a"
Oct 30 18:57:18.659310 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:18.659274    4525 patch_prober.go:29] interesting pod/e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Oct 30 18:57:18.659484 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:18.659335    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td" podUID="bce8221a-1318-4f64-908c-d927050e1ac1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Oct 30 18:57:19.007178 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:19.007147    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-54bc2b-openshift-default-bf84f77dc-jqfxp"
Oct 30 18:57:19.024256 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:19.024223    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d4bed94-hrbfm"]
Oct 30 18:57:19.029068 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:19.029044    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d4bed94-hrbfm"]
Oct 30 18:57:19.461340 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:19.461307    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-54bc2b-openshift-default-bf84f77dc-jqfxp"
Oct 30 18:57:19.462279 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:19.462260    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-54bc2b-openshift-default-bf84f77dc-jqfxp"
Oct 30 18:57:19.550742 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:19.550709    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac945b70-9980-4d7c-8b6a-01b3fb349c44" path="/var/lib/kubelet/pods/ac945b70-9980-4d7c-8b6a-01b3fb349c44/volumes"
Oct 30 18:57:19.659619 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:19.659582    4525 patch_prober.go:29] interesting pod/e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Oct 30 18:57:19.659809 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:19.659641    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td" podUID="bce8221a-1318-4f64-908c-d927050e1ac1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Oct 30 18:57:20.659629 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:20.659592    4525 patch_prober.go:29] interesting pod/e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Oct 30 18:57:20.660283 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:20.659647    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td" podUID="bce8221a-1318-4f64-908c-d927050e1ac1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Oct 30 18:57:21.027036 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:21.026919    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-242299b0-f4z9m"]
Oct 30 18:57:21.049756 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:21.035175    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-242299b0-f4z9m"]
Oct 30 18:57:21.545230 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:21.545197    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ead1810-d291-4c15-ad2a-e310e653d399" path="/var/lib/kubelet/pods/4ead1810-d291-4c15-ad2a-e310e653d399/volumes"
Oct 30 18:57:21.658915 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:21.658879    4525 patch_prober.go:29] interesting pod/e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Oct 30 18:57:21.659093 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:21.658961    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td" podUID="bce8221a-1318-4f64-908c-d927050e1ac1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Oct 30 18:57:22.659589 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:22.659559    4525 patch_prober.go:29] interesting pod/e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Oct 30 18:57:22.660050 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:22.659618    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td" podUID="bce8221a-1318-4f64-908c-d927050e1ac1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Oct 30 18:57:23.662975 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:23.662946    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td"
Oct 30 18:57:23.663471 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:23.663333    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td"
Oct 30 18:57:23.663957 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:23.663936    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td"
Oct 30 18:57:25.799786 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:25.799743    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8f1c37-lvvpr"]
Oct 30 18:57:25.804350 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:25.804318    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8f1c37-lvvpr"]
Oct 30 18:57:27.546890 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:27.546856    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c97ea986-7c6e-448f-b9e4-3c212159631c" path="/var/lib/kubelet/pods/c97ea986-7c6e-448f-b9e4-3c212159631c/volumes"
Oct 30 18:57:27.592856 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:27.592823    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-040d9c-kjp8n"]
Oct 30 18:57:27.599076 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:27.599054    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-040d9c-kjp8n"
Oct 30 18:57:27.612060 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:27.612029    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-040d9c-kjp8n"]
Oct 30 18:57:27.639744 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:27.639707    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bc09eb5-a343-46d0-97cd-10bed2b6e11f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-040d9c-kjp8n\" (UID: \"2bc09eb5-a343-46d0-97cd-10bed2b6e11f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-040d9c-kjp8n"
Oct 30 18:57:27.639950 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:27.639918    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bc09eb5-a343-46d0-97cd-10bed2b6e11f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-040d9c-kjp8n\" (UID: \"2bc09eb5-a343-46d0-97cd-10bed2b6e11f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-040d9c-kjp8n"
Oct 30 18:57:27.640169 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:27.640134    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kbn7\" (UniqueName: \"kubernetes.io/projected/2bc09eb5-a343-46d0-97cd-10bed2b6e11f-kube-api-access-5kbn7\") pod \"maas-api-revoke-keys-e2e-ipp-a-040d9c-kjp8n\" (UID: \"2bc09eb5-a343-46d0-97cd-10bed2b6e11f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-040d9c-kjp8n"
Oct 30 18:57:27.696781 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:27.696748    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:57:27.697007 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:27.696926    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94hcn\""
Oct 30 18:57:27.742237 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:27.742195    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kbn7\" (UniqueName: \"kubernetes.io/projected/2bc09eb5-a343-46d0-97cd-10bed2b6e11f-kube-api-access-5kbn7\") pod \"maas-api-revoke-keys-e2e-ipp-a-040d9c-kjp8n\" (UID: \"2bc09eb5-a343-46d0-97cd-10bed2b6e11f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-040d9c-kjp8n"
Oct 30 18:57:27.742427 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:27.742272    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bc09eb5-a343-46d0-97cd-10bed2b6e11f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-040d9c-kjp8n\" (UID: \"2bc09eb5-a343-46d0-97cd-10bed2b6e11f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-040d9c-kjp8n"
Oct 30 18:57:27.742427 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:27.742372    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bc09eb5-a343-46d0-97cd-10bed2b6e11f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-040d9c-kjp8n\" (UID: \"2bc09eb5-a343-46d0-97cd-10bed2b6e11f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-040d9c-kjp8n"
Oct 30 18:57:27.743279 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:27.743252    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bc09eb5-a343-46d0-97cd-10bed2b6e11f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-040d9c-kjp8n\" (UID: \"2bc09eb5-a343-46d0-97cd-10bed2b6e11f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-040d9c-kjp8n"
Oct 30 18:57:27.744904 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:27.744877    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bc09eb5-a343-46d0-97cd-10bed2b6e11f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-040d9c-kjp8n\" (UID: \"2bc09eb5-a343-46d0-97cd-10bed2b6e11f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-040d9c-kjp8n"
Oct 30 18:57:27.752499 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:27.752479    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kbn7\" (UniqueName: \"kubernetes.io/projected/2bc09eb5-a343-46d0-97cd-10bed2b6e11f-kube-api-access-5kbn7\") pod \"maas-api-revoke-keys-e2e-ipp-a-040d9c-kjp8n\" (UID: \"2bc09eb5-a343-46d0-97cd-10bed2b6e11f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-040d9c-kjp8n"
Oct 30 18:57:27.921867 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:27.921836    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-040d9c-kjp8n"
Oct 30 18:57:28.030431 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:28.030394    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6926d01d-cqhns"]
Oct 30 18:57:28.039845 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:28.039811    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6926d01d-cqhns"]
Oct 30 18:57:28.215774 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:28.215622    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-040d9c-kjp8n"]
Oct 30 18:57:28.220346 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:57:28.220316    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bc09eb5_a343_46d0_97cd_10bed2b6e11f.slice/crio-d67a92086bc1975318478f5188902fe6479f242054aef6b42de5fb10c6ff6e9c WatchSource:0}: Error finding container d67a92086bc1975318478f5188902fe6479f242054aef6b42de5fb10c6ff6e9c: Status 404 returned error can't find the container with id d67a92086bc1975318478f5188902fe6479f242054aef6b42de5fb10c6ff6e9c
Oct 30 18:57:28.567657 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:28.567622    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-040d9c-kjp8n" event={"ID":"2bc09eb5-a343-46d0-97cd-10bed2b6e11f","Type":"ContainerStarted","Data":"f7f580731b8bb57c3fd705f500cc319552464ed4b6f44cfc5beb57ec1e41e0c9"}
Oct 30 18:57:28.568065 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:28.567694    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-040d9c-kjp8n" event={"ID":"2bc09eb5-a343-46d0-97cd-10bed2b6e11f","Type":"ContainerStarted","Data":"d67a92086bc1975318478f5188902fe6479f242054aef6b42de5fb10c6ff6e9c"}
Oct 30 18:57:28.587222 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:28.587157    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-040d9c-kjp8n" podStartSLOduration=1.5871434309999999 podStartE2EDuration="1.587143431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:57:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:57:28.584763152 +0000 UTC m=+1271.210433115" watchObservedRunningTime="2026-10-30 18:57:28.587143431 +0000 UTC m=+1271.212813394"
Oct 30 18:57:29.552702 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:29.551874    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="262b6401-e723-497a-9637-5d80e4d722d9" path="/var/lib/kubelet/pods/262b6401-e723-497a-9637-5d80e4d722d9/volumes"
Oct 30 18:57:29.581026 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:29.580992    4525 generic.go:356] "Generic (PLEG): container finished" podID="2bc09eb5-a343-46d0-97cd-10bed2b6e11f" containerID="f7f580731b8bb57c3fd705f500cc319552464ed4b6f44cfc5beb57ec1e41e0c9" exitCode=0
Oct 30 18:57:29.581528 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:29.581229    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-040d9c-kjp8n" event={"ID":"2bc09eb5-a343-46d0-97cd-10bed2b6e11f","Type":"ContainerDied","Data":"f7f580731b8bb57c3fd705f500cc319552464ed4b6f44cfc5beb57ec1e41e0c9"}
Oct 30 18:57:30.789609 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:30.789583    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-040d9c-kjp8n"
Oct 30 18:57:30.911849 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:30.911759    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bc09eb5-a343-46d0-97cd-10bed2b6e11f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bc09eb5-a343-46d0-97cd-10bed2b6e11f-maas-api-xks-ca\") pod \"2bc09eb5-a343-46d0-97cd-10bed2b6e11f\" (UID: \"2bc09eb5-a343-46d0-97cd-10bed2b6e11f\") "
Oct 30 18:57:30.911999 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:30.911955    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bc09eb5-a343-46d0-97cd-10bed2b6e11f-kube-api-access-5kbn7\" (UniqueName: \"kubernetes.io/projected/2bc09eb5-a343-46d0-97cd-10bed2b6e11f-kube-api-access-5kbn7\") pod \"2bc09eb5-a343-46d0-97cd-10bed2b6e11f\" (UID: \"2bc09eb5-a343-46d0-97cd-10bed2b6e11f\") "
Oct 30 18:57:30.912080 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:30.912066    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bc09eb5-a343-46d0-97cd-10bed2b6e11f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bc09eb5-a343-46d0-97cd-10bed2b6e11f-maas-api-openshift-ca\") pod \"2bc09eb5-a343-46d0-97cd-10bed2b6e11f\" (UID: \"2bc09eb5-a343-46d0-97cd-10bed2b6e11f\") "
Oct 30 18:57:30.912434 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:30.912410    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bc09eb5-a343-46d0-97cd-10bed2b6e11f-maas-api-openshift-ca" pod "2bc09eb5-a343-46d0-97cd-10bed2b6e11f" (UID: "2bc09eb5-a343-46d0-97cd-10bed2b6e11f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:57:30.912619 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:30.912603    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bc09eb5-a343-46d0-97cd-10bed2b6e11f-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:30.913956 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:30.913927    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bc09eb5-a343-46d0-97cd-10bed2b6e11f-maas-api-xks-ca" pod "2bc09eb5-a343-46d0-97cd-10bed2b6e11f" (UID: "2bc09eb5-a343-46d0-97cd-10bed2b6e11f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:57:30.914478 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:30.914451    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bc09eb5-a343-46d0-97cd-10bed2b6e11f-kube-api-access-5kbn7" pod "2bc09eb5-a343-46d0-97cd-10bed2b6e11f" (UID: "2bc09eb5-a343-46d0-97cd-10bed2b6e11f"). InnerVolumeSpecName "kube-api-access-5kbn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:57:31.014250 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:31.014217    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bc09eb5-a343-46d0-97cd-10bed2b6e11f-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:31.014250 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:31.014245    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kbn7\" (UniqueName: \"kubernetes.io/projected/2bc09eb5-a343-46d0-97cd-10bed2b6e11f-kube-api-access-5kbn7\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:31.628803 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:31.628721    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-040d9c-kjp8n" event={"ID":"2bc09eb5-a343-46d0-97cd-10bed2b6e11f","Type":"ContainerDied","Data":"d67a92086bc1975318478f5188902fe6479f242054aef6b42de5fb10c6ff6e9c"}
Oct 30 18:57:31.628803 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:31.628785    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d67a92086bc1975318478f5188902fe6479f242054aef6b42de5fb10c6ff6e9c"
Oct 30 18:57:31.629669 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:31.628844    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-040d9c-kjp8n"
Oct 30 18:57:32.025885 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:32.025851    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-073ea9f1-pnfqr"]
Oct 30 18:57:32.031814 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:32.031780    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-541d2bf7-w25bt"]
Oct 30 18:57:32.035264 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:32.035242    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-073ea9f1-pnfqr"]
Oct 30 18:57:32.039082 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:32.039058    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-541d2bf7-w25bt"]
Oct 30 18:57:33.435054 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:33.435022    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-8f1c37-openshift-default-7cd9c57fdf-kwhps"]
Oct 30 18:57:33.435479 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:33.435226    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-8f1c37-openshift-default-7cd9c57fdf-kwhps" podUID="9c3b7950-7451-4f37-8119-f9574c3045f3" containerName="istio-proxy" containerID="cri-o://df8e6b96b307a54a195cec9e44b62ae7933fc6cee4c29828a7055634d15ed074" gracePeriod=30
Oct 30 18:57:33.631726 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:33.631530    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="328d84f1-7f29-4e6c-b62d-df7a0709914a" path="/var/lib/kubelet/pods/328d84f1-7f29-4e6c-b62d-df7a0709914a/volumes"
Oct 30 18:57:33.634035 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:33.632601    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84f528c3-fc82-45a6-9c25-f72970220236" path="/var/lib/kubelet/pods/84f528c3-fc82-45a6-9c25-f72970220236/volumes"
Oct 30 18:57:33.900623 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:33.900591    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-12a6e1-llk89"]
Oct 30 18:57:33.901287 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:33.901269    4525 cpu_manager.go:414] "containerMap: removing container" podUID="2bc09eb5-a343-46d0-97cd-10bed2b6e11f" containerName="revoke-keys"
Oct 30 18:57:33.901287 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:33.901285    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bc09eb5-a343-46d0-97cd-10bed2b6e11f" containerName="revoke-keys"
Oct 30 18:57:33.901452 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:33.901372    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bc09eb5-a343-46d0-97cd-10bed2b6e11f" containerName="revoke-keys"
Oct 30 18:57:33.906397 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:33.906369    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-12a6e1-llk89"
Oct 30 18:57:33.919881 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:33.919855    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-12a6e1-llk89"]
Oct 30 18:57:33.975939 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:33.975893    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:57:33.976176 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:33.976156    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94hcn\""
Oct 30 18:57:34.093360 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:34.093319    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2e90b1d-8d01-4b0b-b661-287c52b8bb4f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-12a6e1-llk89\" (UID: \"a2e90b1d-8d01-4b0b-b661-287c52b8bb4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-12a6e1-llk89"
Oct 30 18:57:34.093562 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:34.093494    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7g7v\" (UniqueName: \"kubernetes.io/projected/a2e90b1d-8d01-4b0b-b661-287c52b8bb4f-kube-api-access-c7g7v\") pod \"maas-api-revoke-keys-e2e-api-b-12a6e1-llk89\" (UID: \"a2e90b1d-8d01-4b0b-b661-287c52b8bb4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-12a6e1-llk89"
Oct 30 18:57:34.093562 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:34.093534    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2e90b1d-8d01-4b0b-b661-287c52b8bb4f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-12a6e1-llk89\" (UID: \"a2e90b1d-8d01-4b0b-b661-287c52b8bb4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-12a6e1-llk89"
Oct 30 18:57:34.194938 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:34.194848    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7g7v\" (UniqueName: \"kubernetes.io/projected/a2e90b1d-8d01-4b0b-b661-287c52b8bb4f-kube-api-access-c7g7v\") pod \"maas-api-revoke-keys-e2e-api-b-12a6e1-llk89\" (UID: \"a2e90b1d-8d01-4b0b-b661-287c52b8bb4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-12a6e1-llk89"
Oct 30 18:57:34.194938 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:34.194892    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2e90b1d-8d01-4b0b-b661-287c52b8bb4f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-12a6e1-llk89\" (UID: \"a2e90b1d-8d01-4b0b-b661-287c52b8bb4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-12a6e1-llk89"
Oct 30 18:57:34.195207 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:34.195096    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2e90b1d-8d01-4b0b-b661-287c52b8bb4f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-12a6e1-llk89\" (UID: \"a2e90b1d-8d01-4b0b-b661-287c52b8bb4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-12a6e1-llk89"
Oct 30 18:57:34.195647 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:34.195625    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2e90b1d-8d01-4b0b-b661-287c52b8bb4f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-12a6e1-llk89\" (UID: \"a2e90b1d-8d01-4b0b-b661-287c52b8bb4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-12a6e1-llk89"
Oct 30 18:57:34.197728 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:34.197707    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2e90b1d-8d01-4b0b-b661-287c52b8bb4f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-12a6e1-llk89\" (UID: \"a2e90b1d-8d01-4b0b-b661-287c52b8bb4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-12a6e1-llk89"
Oct 30 18:57:34.202297 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:34.202274    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7g7v\" (UniqueName: \"kubernetes.io/projected/a2e90b1d-8d01-4b0b-b661-287c52b8bb4f-kube-api-access-c7g7v\") pod \"maas-api-revoke-keys-e2e-api-b-12a6e1-llk89\" (UID: \"a2e90b1d-8d01-4b0b-b661-287c52b8bb4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-12a6e1-llk89"
Oct 30 18:57:34.231243 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:34.231214    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-12a6e1-llk89"
Oct 30 18:57:34.518039 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:57:34.517993    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2e90b1d_8d01_4b0b_b661_287c52b8bb4f.slice/crio-e0a1ce14ef32f96eb2c908696ba9be674b8063cbd6b2e352367899302ed90dc6 WatchSource:0}: Error finding container e0a1ce14ef32f96eb2c908696ba9be674b8063cbd6b2e352367899302ed90dc6: Status 404 returned error can't find the container with id e0a1ce14ef32f96eb2c908696ba9be674b8063cbd6b2e352367899302ed90dc6
Oct 30 18:57:34.518442 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:34.518158    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-12a6e1-llk89"]
Oct 30 18:57:34.738856 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:34.738739    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-12a6e1-llk89" event={"ID":"a2e90b1d-8d01-4b0b-b661-287c52b8bb4f","Type":"ContainerStarted","Data":"e0a1ce14ef32f96eb2c908696ba9be674b8063cbd6b2e352367899302ed90dc6"}
Oct 30 18:57:35.023892 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:35.023857    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c4ef7c1-bfk6v"]
Oct 30 18:57:35.028149 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:35.028126    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c4ef7c1-bfk6v"]
Oct 30 18:57:35.545540 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:35.545510    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04ac81d7-49c8-4c5a-bf2d-f6740ac448ba" path="/var/lib/kubelet/pods/04ac81d7-49c8-4c5a-bf2d-f6740ac448ba/volumes"
Oct 30 18:57:35.748103 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:35.748015    4525 generic.go:356] "Generic (PLEG): container finished" podID="a2e90b1d-8d01-4b0b-b661-287c52b8bb4f" containerID="f757d57200c0cf51301ccc9edf446344169219ba5f3790ada7f50a04b1cf87ac" exitCode=0
Oct 30 18:57:35.748273 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:35.748107    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-12a6e1-llk89" event={"ID":"a2e90b1d-8d01-4b0b-b661-287c52b8bb4f","Type":"ContainerDied","Data":"f757d57200c0cf51301ccc9edf446344169219ba5f3790ada7f50a04b1cf87ac"}
Oct 30 18:57:36.926161 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:36.926135    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-12a6e1-llk89"
Oct 30 18:57:37.038064 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:37.037984    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2e90b1d-8d01-4b0b-b661-287c52b8bb4f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2e90b1d-8d01-4b0b-b661-287c52b8bb4f-maas-api-openshift-ca\") pod \"a2e90b1d-8d01-4b0b-b661-287c52b8bb4f\" (UID: \"a2e90b1d-8d01-4b0b-b661-287c52b8bb4f\") "
Oct 30 18:57:37.038064 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:37.038041    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2e90b1d-8d01-4b0b-b661-287c52b8bb4f-kube-api-access-c7g7v\" (UniqueName: \"kubernetes.io/projected/a2e90b1d-8d01-4b0b-b661-287c52b8bb4f-kube-api-access-c7g7v\") pod \"a2e90b1d-8d01-4b0b-b661-287c52b8bb4f\" (UID: \"a2e90b1d-8d01-4b0b-b661-287c52b8bb4f\") "
Oct 30 18:57:37.038311 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:37.038079    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2e90b1d-8d01-4b0b-b661-287c52b8bb4f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2e90b1d-8d01-4b0b-b661-287c52b8bb4f-maas-api-xks-ca\") pod \"a2e90b1d-8d01-4b0b-b661-287c52b8bb4f\" (UID: \"a2e90b1d-8d01-4b0b-b661-287c52b8bb4f\") "
Oct 30 18:57:37.038376 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:37.038341    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2e90b1d-8d01-4b0b-b661-287c52b8bb4f-maas-api-openshift-ca" pod "a2e90b1d-8d01-4b0b-b661-287c52b8bb4f" (UID: "a2e90b1d-8d01-4b0b-b661-287c52b8bb4f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:57:37.038700 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:37.038666    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2e90b1d-8d01-4b0b-b661-287c52b8bb4f-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:37.040230 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:37.040209    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2e90b1d-8d01-4b0b-b661-287c52b8bb4f-kube-api-access-c7g7v" pod "a2e90b1d-8d01-4b0b-b661-287c52b8bb4f" (UID: "a2e90b1d-8d01-4b0b-b661-287c52b8bb4f"). InnerVolumeSpecName "kube-api-access-c7g7v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:57:37.040372 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:37.040352    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2e90b1d-8d01-4b0b-b661-287c52b8bb4f-maas-api-xks-ca" pod "a2e90b1d-8d01-4b0b-b661-287c52b8bb4f" (UID: "a2e90b1d-8d01-4b0b-b661-287c52b8bb4f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:57:37.139896 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:37.139866    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7g7v\" (UniqueName: \"kubernetes.io/projected/a2e90b1d-8d01-4b0b-b661-287c52b8bb4f-kube-api-access-c7g7v\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:37.139896 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:37.139894    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2e90b1d-8d01-4b0b-b661-287c52b8bb4f-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:37.780978 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:37.780933    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-12a6e1-llk89"
Oct 30 18:57:37.780978 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:37.780954    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-12a6e1-llk89" event={"ID":"a2e90b1d-8d01-4b0b-b661-287c52b8bb4f","Type":"ContainerDied","Data":"e0a1ce14ef32f96eb2c908696ba9be674b8063cbd6b2e352367899302ed90dc6"}
Oct 30 18:57:37.781231 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:37.781017    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e0a1ce14ef32f96eb2c908696ba9be674b8063cbd6b2e352367899302ed90dc6"
Oct 30 18:57:38.777954 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:38.777925    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-8f1c37-openshift-default-7cd9c57fdf-kwhps"
Oct 30 18:57:38.791782 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:38.791498    4525 generic.go:356] "Generic (PLEG): container finished" podID="9c3b7950-7451-4f37-8119-f9574c3045f3" containerID="df8e6b96b307a54a195cec9e44b62ae7933fc6cee4c29828a7055634d15ed074" exitCode=0
Oct 30 18:57:38.791782 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:38.791554    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-8f1c37-openshift-default-7cd9c57fdf-kwhps"
Oct 30 18:57:38.791782 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:38.791547    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8f1c37-openshift-default-7cd9c57fdf-kwhps" event={"ID":"9c3b7950-7451-4f37-8119-f9574c3045f3","Type":"ContainerDied","Data":"df8e6b96b307a54a195cec9e44b62ae7933fc6cee4c29828a7055634d15ed074"}
Oct 30 18:57:38.791782 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:38.791617    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8f1c37-openshift-default-7cd9c57fdf-kwhps" event={"ID":"9c3b7950-7451-4f37-8119-f9574c3045f3","Type":"ContainerDied","Data":"6456efc3c3b473993b9bd1ad703bd38538b2ff81507ce95a4bf405fe988c0480"}
Oct 30 18:57:38.791782 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:38.791643    4525 scope.go:122] "RemoveContainer" containerID="df8e6b96b307a54a195cec9e44b62ae7933fc6cee4c29828a7055634d15ed074"
Oct 30 18:57:38.805093 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:38.805062    4525 scope.go:122] "RemoveContainer" containerID="18a288522f8406d4217f3b2d58d5f6c7858fac117df2b308e65e0868deb47272"
Oct 30 18:57:38.817982 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:38.817954    4525 scope.go:122] "RemoveContainer" containerID="df8e6b96b307a54a195cec9e44b62ae7933fc6cee4c29828a7055634d15ed074"
Oct 30 18:57:38.818343 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:57:38.818313    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df8e6b96b307a54a195cec9e44b62ae7933fc6cee4c29828a7055634d15ed074\": container with ID starting with df8e6b96b307a54a195cec9e44b62ae7933fc6cee4c29828a7055634d15ed074 not found: ID does not exist" containerID="df8e6b96b307a54a195cec9e44b62ae7933fc6cee4c29828a7055634d15ed074"
Oct 30 18:57:38.818454 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:38.818354    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df8e6b96b307a54a195cec9e44b62ae7933fc6cee4c29828a7055634d15ed074"} err="failed to get container status \"df8e6b96b307a54a195cec9e44b62ae7933fc6cee4c29828a7055634d15ed074\": rpc error: code = NotFound desc = could not find container \"df8e6b96b307a54a195cec9e44b62ae7933fc6cee4c29828a7055634d15ed074\": container with ID starting with df8e6b96b307a54a195cec9e44b62ae7933fc6cee4c29828a7055634d15ed074 not found: ID does not exist"
Oct 30 18:57:38.818454 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:38.818372    4525 scope.go:122] "RemoveContainer" containerID="18a288522f8406d4217f3b2d58d5f6c7858fac117df2b308e65e0868deb47272"
Oct 30 18:57:38.818724 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:57:38.818698    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18a288522f8406d4217f3b2d58d5f6c7858fac117df2b308e65e0868deb47272\": container with ID starting with 18a288522f8406d4217f3b2d58d5f6c7858fac117df2b308e65e0868deb47272 not found: ID does not exist" containerID="18a288522f8406d4217f3b2d58d5f6c7858fac117df2b308e65e0868deb47272"
Oct 30 18:57:38.818838 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:38.818728    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18a288522f8406d4217f3b2d58d5f6c7858fac117df2b308e65e0868deb47272"} err="failed to get container status \"18a288522f8406d4217f3b2d58d5f6c7858fac117df2b308e65e0868deb47272\": rpc error: code = NotFound desc = could not find container \"18a288522f8406d4217f3b2d58d5f6c7858fac117df2b308e65e0868deb47272\": container with ID starting with 18a288522f8406d4217f3b2d58d5f6c7858fac117df2b308e65e0868deb47272 not found: ID does not exist"
Oct 30 18:57:38.885428 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:38.885378    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c3b7950-7451-4f37-8119-f9574c3045f3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c3b7950-7451-4f37-8119-f9574c3045f3-workload-socket\") pod \"9c3b7950-7451-4f37-8119-f9574c3045f3\" (UID: \"9c3b7950-7451-4f37-8119-f9574c3045f3\") "
Oct 30 18:57:38.891706 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:38.886033    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c3b7950-7451-4f37-8119-f9574c3045f3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c3b7950-7451-4f37-8119-f9574c3045f3-istio-envoy\") pod \"9c3b7950-7451-4f37-8119-f9574c3045f3\" (UID: \"9c3b7950-7451-4f37-8119-f9574c3045f3\") "
Oct 30 18:57:38.891706 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:38.886139    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c3b7950-7451-4f37-8119-f9574c3045f3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c3b7950-7451-4f37-8119-f9574c3045f3-istiod-ca-cert\") pod \"9c3b7950-7451-4f37-8119-f9574c3045f3\" (UID: \"9c3b7950-7451-4f37-8119-f9574c3045f3\") "
Oct 30 18:57:38.891706 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:38.886198    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c3b7950-7451-4f37-8119-f9574c3045f3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c3b7950-7451-4f37-8119-f9574c3045f3-workload-certs\") pod \"9c3b7950-7451-4f37-8119-f9574c3045f3\" (UID: \"9c3b7950-7451-4f37-8119-f9574c3045f3\") "
Oct 30 18:57:38.891706 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:38.886277    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9c3b7950-7451-4f37-8119-f9574c3045f3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c3b7950-7451-4f37-8119-f9574c3045f3-istio-podinfo\") pod \"9c3b7950-7451-4f37-8119-f9574c3045f3\" (UID: \"9c3b7950-7451-4f37-8119-f9574c3045f3\") "
Oct 30 18:57:38.891706 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:38.886321    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c3b7950-7451-4f37-8119-f9574c3045f3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c3b7950-7451-4f37-8119-f9574c3045f3-credential-socket\") pod \"9c3b7950-7451-4f37-8119-f9574c3045f3\" (UID: \"9c3b7950-7451-4f37-8119-f9574c3045f3\") "
Oct 30 18:57:38.891706 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:38.886478    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c3b7950-7451-4f37-8119-f9574c3045f3-istio-token\" (UniqueName: \"kubernetes.io/projected/9c3b7950-7451-4f37-8119-f9574c3045f3-istio-token\") pod \"9c3b7950-7451-4f37-8119-f9574c3045f3\" (UID: \"9c3b7950-7451-4f37-8119-f9574c3045f3\") "
Oct 30 18:57:38.891706 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:38.886591    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c3b7950-7451-4f37-8119-f9574c3045f3-kube-api-access-qjhrn\" (UniqueName: \"kubernetes.io/projected/9c3b7950-7451-4f37-8119-f9574c3045f3-kube-api-access-qjhrn\") pod \"9c3b7950-7451-4f37-8119-f9574c3045f3\" (UID: \"9c3b7950-7451-4f37-8119-f9574c3045f3\") "
Oct 30 18:57:38.891706 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:38.886642    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c3b7950-7451-4f37-8119-f9574c3045f3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c3b7950-7451-4f37-8119-f9574c3045f3-istio-data\") pod \"9c3b7950-7451-4f37-8119-f9574c3045f3\" (UID: \"9c3b7950-7451-4f37-8119-f9574c3045f3\") "
Oct 30 18:57:38.895178 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:38.895145    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c3b7950-7451-4f37-8119-f9574c3045f3-istio-data" pod "9c3b7950-7451-4f37-8119-f9574c3045f3" (UID: "9c3b7950-7451-4f37-8119-f9574c3045f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:57:38.902926 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:38.902889    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c3b7950-7451-4f37-8119-f9574c3045f3-istiod-ca-cert" pod "9c3b7950-7451-4f37-8119-f9574c3045f3" (UID: "9c3b7950-7451-4f37-8119-f9574c3045f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:57:38.903199 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:38.903173    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c3b7950-7451-4f37-8119-f9574c3045f3-workload-socket" pod "9c3b7950-7451-4f37-8119-f9574c3045f3" (UID: "9c3b7950-7451-4f37-8119-f9574c3045f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:57:38.903375 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:38.903355    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c3b7950-7451-4f37-8119-f9574c3045f3-credential-socket" pod "9c3b7950-7451-4f37-8119-f9574c3045f3" (UID: "9c3b7950-7451-4f37-8119-f9574c3045f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:57:38.903542 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:38.903524    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c3b7950-7451-4f37-8119-f9574c3045f3-workload-certs" pod "9c3b7950-7451-4f37-8119-f9574c3045f3" (UID: "9c3b7950-7451-4f37-8119-f9574c3045f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:57:38.914181 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:38.914147    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c3b7950-7451-4f37-8119-f9574c3045f3-istio-envoy" pod "9c3b7950-7451-4f37-8119-f9574c3045f3" (UID: "9c3b7950-7451-4f37-8119-f9574c3045f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:57:38.918298 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:38.915095    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9c3b7950-7451-4f37-8119-f9574c3045f3-istio-podinfo" pod "9c3b7950-7451-4f37-8119-f9574c3045f3" (UID: "9c3b7950-7451-4f37-8119-f9574c3045f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 18:57:39.023735 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:39.020005    4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c3b7950-7451-4f37-8119-f9574c3045f3-istio-data\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:39.023735 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:39.020041    4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c3b7950-7451-4f37-8119-f9574c3045f3-workload-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:39.023735 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:39.020059    4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c3b7950-7451-4f37-8119-f9574c3045f3-istio-envoy\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:39.023735 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:39.020073    4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c3b7950-7451-4f37-8119-f9574c3045f3-istiod-ca-cert\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:39.023735 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:39.020087    4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c3b7950-7451-4f37-8119-f9574c3045f3-workload-certs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:39.023735 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:39.020100    4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c3b7950-7451-4f37-8119-f9574c3045f3-istio-podinfo\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:39.023735 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:39.020114    4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c3b7950-7451-4f37-8119-f9574c3045f3-credential-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:39.025745 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:39.025671    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c3b7950-7451-4f37-8119-f9574c3045f3-istio-token" pod "9c3b7950-7451-4f37-8119-f9574c3045f3" (UID: "9c3b7950-7451-4f37-8119-f9574c3045f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:57:39.050906 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:39.050849    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c3b7950-7451-4f37-8119-f9574c3045f3-kube-api-access-qjhrn" pod "9c3b7950-7451-4f37-8119-f9574c3045f3" (UID: "9c3b7950-7451-4f37-8119-f9574c3045f3"). InnerVolumeSpecName "kube-api-access-qjhrn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:57:39.122155 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:39.122102    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-8f1c37-openshift-default-7cd9c57fdf-kwhps"]
Oct 30 18:57:39.122365 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:39.122258    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjhrn\" (UniqueName: \"kubernetes.io/projected/9c3b7950-7451-4f37-8119-f9574c3045f3-kube-api-access-qjhrn\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:39.122365 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:39.122277    4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c3b7950-7451-4f37-8119-f9574c3045f3-istio-token\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:39.126002 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:39.125974    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-8f1c37-openshift-default-7cd9c57fdf-kwhps"]
Oct 30 18:57:39.184100 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:39.184002    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-040d9c-kjp8n"]
Oct 30 18:57:39.189872 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:39.189826    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-040d9c-kjp8n"]
Oct 30 18:57:39.551493 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:39.551460    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bc09eb5-a343-46d0-97cd-10bed2b6e11f" path="/var/lib/kubelet/pods/2bc09eb5-a343-46d0-97cd-10bed2b6e11f/volumes"
Oct 30 18:57:39.552212 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:39.552191    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c3b7950-7451-4f37-8119-f9574c3045f3" path="/var/lib/kubelet/pods/9c3b7950-7451-4f37-8119-f9574c3045f3/volumes"
Oct 30 18:57:42.541142 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:42.541111    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-da58b769-openshift-default-69887ddf8f-n79ls"]
Oct 30 18:57:42.541731 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:42.541315    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-da58b769-openshift-default-69887ddf8f-n79ls" podUID="81739e60-d074-4a42-ade3-a0f68ea9329d" containerName="istio-proxy" containerID="cri-o://24cc207c0ffbce5490bd5d432ae75292eb84c13ddee7f6e809a74eb1175fc456" gracePeriod=30
Oct 30 18:57:44.168442 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:44.168404    4525 patch_prober.go:29] interesting pod/e2e-mt-da58b769-openshift-default-69887ddf8f-n79ls container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 18:57:44.168986 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:44.168481    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-da58b769-openshift-default-69887ddf8f-n79ls" podUID="81739e60-d074-4a42-ade3-a0f68ea9329d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:57:45.141306 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:45.139887    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-642ba060-xf9kw"]
Oct 30 18:57:45.148204 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:45.148155    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-642ba060-xf9kw"]
Oct 30 18:57:45.629516 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:45.628909    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0d69f71-d7aa-4ab2-94fe-72582bbec250" path="/var/lib/kubelet/pods/f0d69f71-d7aa-4ab2-94fe-72582bbec250/volumes"
Oct 30 18:57:46.023329 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:46.023241    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2ea8d05-9xc2z"]
Oct 30 18:57:46.027474 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:46.027450    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2ea8d05-9xc2z"]
Oct 30 18:57:46.917025 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:46.916993    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx"]
Oct 30 18:57:46.917832 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:46.917784    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx" podUID="2d4927ad-ead6-477f-8105-4cdd3daadea5" containerName="istio-proxy" containerID="cri-o://b994d5f9166561d9b27b39cbee3ed61b6ab8e978e9ee2edd0f183935b47f78df" gracePeriod=30
Oct 30 18:57:47.437743 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:57:47.436878    4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d49l" containerName="extractor"
Oct 30 18:57:47.438323 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:47.438296    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e7e190-swpr4"]
Oct 30 18:57:47.439218 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:47.439195    4525 cpu_manager.go:414] "containerMap: removing container" podUID="9c3b7950-7451-4f37-8119-f9574c3045f3" containerName="istio-proxy"
Oct 30 18:57:47.439324 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:47.439220    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c3b7950-7451-4f37-8119-f9574c3045f3" containerName="istio-proxy"
Oct 30 18:57:47.439324 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:47.439234    4525 cpu_manager.go:414] "containerMap: removing container" podUID="a2e90b1d-8d01-4b0b-b661-287c52b8bb4f" containerName="revoke-keys"
Oct 30 18:57:47.439324 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:47.439242    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2e90b1d-8d01-4b0b-b661-287c52b8bb4f" containerName="revoke-keys"
Oct 30 18:57:47.439460 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:47.439408    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c3b7950-7451-4f37-8119-f9574c3045f3" containerName="istio-proxy"
Oct 30 18:57:47.439460 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:47.439418    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2e90b1d-8d01-4b0b-b661-287c52b8bb4f" containerName="revoke-keys"
Oct 30 18:57:47.445144 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:47.445120    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e7e190-swpr4"
Oct 30 18:57:47.448107 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:47.448072    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:57:47.448803 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:47.448775    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94hcn\""
Oct 30 18:57:47.449760 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:47.449706    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e7e190-swpr4"]
Oct 30 18:57:47.462650 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:47.462616    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4pd4\" (UniqueName: \"kubernetes.io/projected/614a95b4-19f0-4490-b402-e681c1bc6e39-kube-api-access-p4pd4\") pod \"maas-api-revoke-keys-e2e-ipp-b-e7e190-swpr4\" (UID: \"614a95b4-19f0-4490-b402-e681c1bc6e39\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e7e190-swpr4"
Oct 30 18:57:47.462907 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:47.462806    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/614a95b4-19f0-4490-b402-e681c1bc6e39-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e7e190-swpr4\" (UID: \"614a95b4-19f0-4490-b402-e681c1bc6e39\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e7e190-swpr4"
Oct 30 18:57:47.463015 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:47.462967    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/614a95b4-19f0-4490-b402-e681c1bc6e39-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e7e190-swpr4\" (UID: \"614a95b4-19f0-4490-b402-e681c1bc6e39\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e7e190-swpr4"
Oct 30 18:57:47.545965 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:47.545929    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bd81572-1116-4a66-bbc1-a08ce150a2ba" path="/var/lib/kubelet/pods/8bd81572-1116-4a66-bbc1-a08ce150a2ba/volumes"
Oct 30 18:57:47.564103 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:47.564075    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/614a95b4-19f0-4490-b402-e681c1bc6e39-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e7e190-swpr4\" (UID: \"614a95b4-19f0-4490-b402-e681c1bc6e39\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e7e190-swpr4"
Oct 30 18:57:47.564242 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:47.564148    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/614a95b4-19f0-4490-b402-e681c1bc6e39-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e7e190-swpr4\" (UID: \"614a95b4-19f0-4490-b402-e681c1bc6e39\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e7e190-swpr4"
Oct 30 18:57:47.564242 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:47.564241    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4pd4\" (UniqueName: \"kubernetes.io/projected/614a95b4-19f0-4490-b402-e681c1bc6e39-kube-api-access-p4pd4\") pod \"maas-api-revoke-keys-e2e-ipp-b-e7e190-swpr4\" (UID: \"614a95b4-19f0-4490-b402-e681c1bc6e39\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e7e190-swpr4"
Oct 30 18:57:47.564862 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:47.564844    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/614a95b4-19f0-4490-b402-e681c1bc6e39-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e7e190-swpr4\" (UID: \"614a95b4-19f0-4490-b402-e681c1bc6e39\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e7e190-swpr4"
Oct 30 18:57:47.566522 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:47.566497    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/614a95b4-19f0-4490-b402-e681c1bc6e39-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e7e190-swpr4\" (UID: \"614a95b4-19f0-4490-b402-e681c1bc6e39\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e7e190-swpr4"
Oct 30 18:57:47.572963 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:47.572938    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4pd4\" (UniqueName: \"kubernetes.io/projected/614a95b4-19f0-4490-b402-e681c1bc6e39-kube-api-access-p4pd4\") pod \"maas-api-revoke-keys-e2e-ipp-b-e7e190-swpr4\" (UID: \"614a95b4-19f0-4490-b402-e681c1bc6e39\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e7e190-swpr4"
Oct 30 18:57:47.767293 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:47.767258    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e7e190-swpr4"
Oct 30 18:57:47.903091 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:47.903062    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-da58b769-openshift-default-69887ddf8f-n79ls"
Oct 30 18:57:47.999906 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:47.975512    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81739e60-d074-4a42-ade3-a0f68ea9329d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81739e60-d074-4a42-ade3-a0f68ea9329d-istio-data\") pod \"81739e60-d074-4a42-ade3-a0f68ea9329d\" (UID: \"81739e60-d074-4a42-ade3-a0f68ea9329d\") "
Oct 30 18:57:47.999906 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:47.975656    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/81739e60-d074-4a42-ade3-a0f68ea9329d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81739e60-d074-4a42-ade3-a0f68ea9329d-istio-podinfo\") pod \"81739e60-d074-4a42-ade3-a0f68ea9329d\" (UID: \"81739e60-d074-4a42-ade3-a0f68ea9329d\") "
Oct 30 18:57:47.999906 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:47.975788    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81739e60-d074-4a42-ade3-a0f68ea9329d-istio-token\" (UniqueName: \"kubernetes.io/projected/81739e60-d074-4a42-ade3-a0f68ea9329d-istio-token\") pod \"81739e60-d074-4a42-ade3-a0f68ea9329d\" (UID: \"81739e60-d074-4a42-ade3-a0f68ea9329d\") "
Oct 30 18:57:47.999906 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:47.975830    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81739e60-d074-4a42-ade3-a0f68ea9329d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81739e60-d074-4a42-ade3-a0f68ea9329d-workload-certs\") pod \"81739e60-d074-4a42-ade3-a0f68ea9329d\" (UID: \"81739e60-d074-4a42-ade3-a0f68ea9329d\") "
Oct 30 18:57:47.999906 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:47.975891    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81739e60-d074-4a42-ade3-a0f68ea9329d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81739e60-d074-4a42-ade3-a0f68ea9329d-istiod-ca-cert\") pod \"81739e60-d074-4a42-ade3-a0f68ea9329d\" (UID: \"81739e60-d074-4a42-ade3-a0f68ea9329d\") "
Oct 30 18:57:47.999906 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:47.975944    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81739e60-d074-4a42-ade3-a0f68ea9329d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81739e60-d074-4a42-ade3-a0f68ea9329d-istio-envoy\") pod \"81739e60-d074-4a42-ade3-a0f68ea9329d\" (UID: \"81739e60-d074-4a42-ade3-a0f68ea9329d\") "
Oct 30 18:57:47.999906 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:47.976019    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81739e60-d074-4a42-ade3-a0f68ea9329d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81739e60-d074-4a42-ade3-a0f68ea9329d-workload-socket\") pod \"81739e60-d074-4a42-ade3-a0f68ea9329d\" (UID: \"81739e60-d074-4a42-ade3-a0f68ea9329d\") "
Oct 30 18:57:47.999906 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:47.976081    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81739e60-d074-4a42-ade3-a0f68ea9329d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81739e60-d074-4a42-ade3-a0f68ea9329d-credential-socket\") pod \"81739e60-d074-4a42-ade3-a0f68ea9329d\" (UID: \"81739e60-d074-4a42-ade3-a0f68ea9329d\") "
Oct 30 18:57:47.999906 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:47.976181    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81739e60-d074-4a42-ade3-a0f68ea9329d-kube-api-access-47ncm\" (UniqueName: \"kubernetes.io/projected/81739e60-d074-4a42-ade3-a0f68ea9329d-kube-api-access-47ncm\") pod \"81739e60-d074-4a42-ade3-a0f68ea9329d\" (UID: \"81739e60-d074-4a42-ade3-a0f68ea9329d\") "
Oct 30 18:57:48.011841 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:48.007259    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81739e60-d074-4a42-ade3-a0f68ea9329d-workload-certs" pod "81739e60-d074-4a42-ade3-a0f68ea9329d" (UID: "81739e60-d074-4a42-ade3-a0f68ea9329d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:57:48.011841 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:48.007591    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81739e60-d074-4a42-ade3-a0f68ea9329d-istio-data" pod "81739e60-d074-4a42-ade3-a0f68ea9329d" (UID: "81739e60-d074-4a42-ade3-a0f68ea9329d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:57:48.011841 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:48.011819    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81739e60-d074-4a42-ade3-a0f68ea9329d-workload-socket" pod "81739e60-d074-4a42-ade3-a0f68ea9329d" (UID: "81739e60-d074-4a42-ade3-a0f68ea9329d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:57:48.031623 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:48.012517    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81739e60-d074-4a42-ade3-a0f68ea9329d-credential-socket" pod "81739e60-d074-4a42-ade3-a0f68ea9329d" (UID: "81739e60-d074-4a42-ade3-a0f68ea9329d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:57:48.033597 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:48.032803    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81739e60-d074-4a42-ade3-a0f68ea9329d-istiod-ca-cert" pod "81739e60-d074-4a42-ade3-a0f68ea9329d" (UID: "81739e60-d074-4a42-ade3-a0f68ea9329d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:57:48.037887 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:48.037258    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9384137-s9nhl"]
Oct 30 18:57:48.048656 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:48.046487    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81739e60-d074-4a42-ade3-a0f68ea9329d-kube-api-access-47ncm" pod "81739e60-d074-4a42-ade3-a0f68ea9329d" (UID: "81739e60-d074-4a42-ade3-a0f68ea9329d"). InnerVolumeSpecName "kube-api-access-47ncm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:57:48.048656 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:48.047568    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81739e60-d074-4a42-ade3-a0f68ea9329d-istio-envoy" pod "81739e60-d074-4a42-ade3-a0f68ea9329d" (UID: "81739e60-d074-4a42-ade3-a0f68ea9329d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:57:48.048656 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:48.048088    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9384137-s9nhl"]
Oct 30 18:57:48.050114 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:48.048774    4525 generic.go:356] "Generic (PLEG): container finished" podID="81739e60-d074-4a42-ade3-a0f68ea9329d" containerID="24cc207c0ffbce5490bd5d432ae75292eb84c13ddee7f6e809a74eb1175fc456" exitCode=0
Oct 30 18:57:48.050114 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:48.048822    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/81739e60-d074-4a42-ade3-a0f68ea9329d-istio-podinfo" pod "81739e60-d074-4a42-ade3-a0f68ea9329d" (UID: "81739e60-d074-4a42-ade3-a0f68ea9329d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 18:57:48.050114 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:48.048881    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-da58b769-openshift-default-69887ddf8f-n79ls" event={"ID":"81739e60-d074-4a42-ade3-a0f68ea9329d","Type":"ContainerDied","Data":"24cc207c0ffbce5490bd5d432ae75292eb84c13ddee7f6e809a74eb1175fc456"}
Oct 30 18:57:48.050114 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:48.048963    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-da58b769-openshift-default-69887ddf8f-n79ls"
Oct 30 18:57:48.050114 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:48.048986    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-da58b769-openshift-default-69887ddf8f-n79ls" event={"ID":"81739e60-d074-4a42-ade3-a0f68ea9329d","Type":"ContainerDied","Data":"a9e53e895cc3982b47223434f429e0cc2d970db44cc294a70c3fb71e570a98d0"}
Oct 30 18:57:48.050114 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:48.049372    4525 scope.go:122] "RemoveContainer" containerID="24cc207c0ffbce5490bd5d432ae75292eb84c13ddee7f6e809a74eb1175fc456"
Oct 30 18:57:48.051999 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:48.051975    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81739e60-d074-4a42-ade3-a0f68ea9329d-istio-token" pod "81739e60-d074-4a42-ade3-a0f68ea9329d" (UID: "81739e60-d074-4a42-ade3-a0f68ea9329d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:57:48.065906 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:48.065884    4525 scope.go:122] "RemoveContainer" containerID="3eb61f7ea9b79c37b1f93a3a5f2a93a34be89622062ad9a76e12470e5e20c9a3"
Oct 30 18:57:48.090723 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:48.087920    4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81739e60-d074-4a42-ade3-a0f68ea9329d-istio-podinfo\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:48.090723 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:48.087965    4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81739e60-d074-4a42-ade3-a0f68ea9329d-istio-token\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:48.090723 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:48.087984    4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81739e60-d074-4a42-ade3-a0f68ea9329d-workload-certs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:48.090723 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:48.087998    4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81739e60-d074-4a42-ade3-a0f68ea9329d-istiod-ca-cert\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:48.090723 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:48.088012    4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81739e60-d074-4a42-ade3-a0f68ea9329d-istio-envoy\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:48.090723 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:48.088026    4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81739e60-d074-4a42-ade3-a0f68ea9329d-workload-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:48.090723 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:48.088040    4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81739e60-d074-4a42-ade3-a0f68ea9329d-credential-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:48.090723 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:48.088056    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47ncm\" (UniqueName: \"kubernetes.io/projected/81739e60-d074-4a42-ade3-a0f68ea9329d-kube-api-access-47ncm\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:48.090723 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:48.088072    4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81739e60-d074-4a42-ade3-a0f68ea9329d-istio-data\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:48.351759 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:48.330191    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e7e190-swpr4"]
Oct 30 18:57:48.351759 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:48.335810    4525 scope.go:122] "RemoveContainer" containerID="24cc207c0ffbce5490bd5d432ae75292eb84c13ddee7f6e809a74eb1175fc456"
Oct 30 18:57:48.370723 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:57:48.358202    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24cc207c0ffbce5490bd5d432ae75292eb84c13ddee7f6e809a74eb1175fc456\": container with ID starting with 24cc207c0ffbce5490bd5d432ae75292eb84c13ddee7f6e809a74eb1175fc456 not found: ID does not exist" containerID="24cc207c0ffbce5490bd5d432ae75292eb84c13ddee7f6e809a74eb1175fc456"
Oct 30 18:57:48.370723 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:48.358248    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24cc207c0ffbce5490bd5d432ae75292eb84c13ddee7f6e809a74eb1175fc456"} err="failed to get container status \"24cc207c0ffbce5490bd5d432ae75292eb84c13ddee7f6e809a74eb1175fc456\": rpc error: code = NotFound desc = could not find container \"24cc207c0ffbce5490bd5d432ae75292eb84c13ddee7f6e809a74eb1175fc456\": container with ID starting with 24cc207c0ffbce5490bd5d432ae75292eb84c13ddee7f6e809a74eb1175fc456 not found: ID does not exist"
Oct 30 18:57:48.370723 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:48.358275    4525 scope.go:122] "RemoveContainer" containerID="3eb61f7ea9b79c37b1f93a3a5f2a93a34be89622062ad9a76e12470e5e20c9a3"
Oct 30 18:57:48.370723 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:57:48.364058    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3eb61f7ea9b79c37b1f93a3a5f2a93a34be89622062ad9a76e12470e5e20c9a3\": container with ID starting with 3eb61f7ea9b79c37b1f93a3a5f2a93a34be89622062ad9a76e12470e5e20c9a3 not found: ID does not exist" containerID="3eb61f7ea9b79c37b1f93a3a5f2a93a34be89622062ad9a76e12470e5e20c9a3"
Oct 30 18:57:48.370723 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:48.364096    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3eb61f7ea9b79c37b1f93a3a5f2a93a34be89622062ad9a76e12470e5e20c9a3"} err="failed to get container status \"3eb61f7ea9b79c37b1f93a3a5f2a93a34be89622062ad9a76e12470e5e20c9a3\": rpc error: code = NotFound desc = could not find container \"3eb61f7ea9b79c37b1f93a3a5f2a93a34be89622062ad9a76e12470e5e20c9a3\": container with ID starting with 3eb61f7ea9b79c37b1f93a3a5f2a93a34be89622062ad9a76e12470e5e20c9a3 not found: ID does not exist"
Oct 30 18:57:48.429301 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:48.429275    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-da58b769-openshift-default-69887ddf8f-n79ls"]
Oct 30 18:57:48.436314 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:48.436283    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-da58b769-openshift-default-69887ddf8f-n79ls"]
Oct 30 18:57:49.083767 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:49.070804    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-812953cf-w4g6v"]
Oct 30 18:57:49.095932 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:49.092913    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-812953cf-w4g6v"]
Oct 30 18:57:49.095932 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:49.095489    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e7e190-swpr4" event={"ID":"614a95b4-19f0-4490-b402-e681c1bc6e39","Type":"ContainerStarted","Data":"6f6e7c38cf81f9888734a208501f27d045f7cfa123a722c9067feba60023bf88"}
Oct 30 18:57:49.095932 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:49.095553    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e7e190-swpr4" event={"ID":"614a95b4-19f0-4490-b402-e681c1bc6e39","Type":"ContainerStarted","Data":"dc0cc59cd716d373e05701e840d853a1703be1b1b33587d9576ac80b49c65a0c"}
Oct 30 18:57:49.144743 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:49.144654    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e7e190-swpr4" podStartSLOduration=2.144636384 podStartE2EDuration="2.144636384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:57:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:57:49.119883599 +0000 UTC m=+1291.745553564" watchObservedRunningTime="2026-10-30 18:57:49.144636384 +0000 UTC m=+1291.770306372"
Oct 30 18:57:49.576145 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:49.576103    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="756ed288-3ee5-4f9a-9ae3-114ec19838c6" path="/var/lib/kubelet/pods/756ed288-3ee5-4f9a-9ae3-114ec19838c6/volumes"
Oct 30 18:57:49.623939 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:49.604289    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81739e60-d074-4a42-ade3-a0f68ea9329d" path="/var/lib/kubelet/pods/81739e60-d074-4a42-ade3-a0f68ea9329d/volumes"
Oct 30 18:57:49.629849 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:49.625733    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f19f3cd6-d6f8-4f67-9bc1-76acf8cffb17" path="/var/lib/kubelet/pods/f19f3cd6-d6f8-4f67-9bc1-76acf8cffb17/volumes"
Oct 30 18:57:50.165272 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:50.165237    4525 generic.go:356] "Generic (PLEG): container finished" podID="614a95b4-19f0-4490-b402-e681c1bc6e39" containerID="6f6e7c38cf81f9888734a208501f27d045f7cfa123a722c9067feba60023bf88" exitCode=0
Oct 30 18:57:50.165657 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:50.165493    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e7e190-swpr4" event={"ID":"614a95b4-19f0-4490-b402-e681c1bc6e39","Type":"ContainerDied","Data":"6f6e7c38cf81f9888734a208501f27d045f7cfa123a722c9067feba60023bf88"}
Oct 30 18:57:51.262483 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:51.256330    4525 patch_prober.go:29] interesting pod/e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 18:57:51.262483 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:51.256393    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx" podUID="2d4927ad-ead6-477f-8105-4cdd3daadea5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:57:51.563373 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:51.562991    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e7e190-swpr4"
Oct 30 18:57:51.734432 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:51.734387    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/614a95b4-19f0-4490-b402-e681c1bc6e39-kube-api-access-p4pd4\" (UniqueName: \"kubernetes.io/projected/614a95b4-19f0-4490-b402-e681c1bc6e39-kube-api-access-p4pd4\") pod \"614a95b4-19f0-4490-b402-e681c1bc6e39\" (UID: \"614a95b4-19f0-4490-b402-e681c1bc6e39\") "
Oct 30 18:57:51.734607 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:51.734492    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/614a95b4-19f0-4490-b402-e681c1bc6e39-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/614a95b4-19f0-4490-b402-e681c1bc6e39-maas-api-openshift-ca\") pod \"614a95b4-19f0-4490-b402-e681c1bc6e39\" (UID: \"614a95b4-19f0-4490-b402-e681c1bc6e39\") "
Oct 30 18:57:51.734864 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:51.734763    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/614a95b4-19f0-4490-b402-e681c1bc6e39-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/614a95b4-19f0-4490-b402-e681c1bc6e39-maas-api-xks-ca\") pod \"614a95b4-19f0-4490-b402-e681c1bc6e39\" (UID: \"614a95b4-19f0-4490-b402-e681c1bc6e39\") "
Oct 30 18:57:51.734985 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:51.734936    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/614a95b4-19f0-4490-b402-e681c1bc6e39-maas-api-openshift-ca" pod "614a95b4-19f0-4490-b402-e681c1bc6e39" (UID: "614a95b4-19f0-4490-b402-e681c1bc6e39"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:57:51.735725 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:51.735518    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/614a95b4-19f0-4490-b402-e681c1bc6e39-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:51.737099 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:51.737077    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/614a95b4-19f0-4490-b402-e681c1bc6e39-maas-api-xks-ca" pod "614a95b4-19f0-4490-b402-e681c1bc6e39" (UID: "614a95b4-19f0-4490-b402-e681c1bc6e39"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:57:51.737231 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:51.737216    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/614a95b4-19f0-4490-b402-e681c1bc6e39-kube-api-access-p4pd4" pod "614a95b4-19f0-4490-b402-e681c1bc6e39" (UID: "614a95b4-19f0-4490-b402-e681c1bc6e39"). InnerVolumeSpecName "kube-api-access-p4pd4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:57:51.840785 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:51.839825    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4pd4\" (UniqueName: \"kubernetes.io/projected/614a95b4-19f0-4490-b402-e681c1bc6e39-kube-api-access-p4pd4\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:51.840785 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:51.839867    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/614a95b4-19f0-4490-b402-e681c1bc6e39-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:52.221206 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:52.221175    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx"
Oct 30 18:57:52.306711 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:52.300296    4525 generic.go:356] "Generic (PLEG): container finished" podID="2d4927ad-ead6-477f-8105-4cdd3daadea5" containerID="b994d5f9166561d9b27b39cbee3ed61b6ab8e978e9ee2edd0f183935b47f78df" exitCode=0
Oct 30 18:57:52.306711 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:52.300572    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx" event={"ID":"2d4927ad-ead6-477f-8105-4cdd3daadea5","Type":"ContainerDied","Data":"b994d5f9166561d9b27b39cbee3ed61b6ab8e978e9ee2edd0f183935b47f78df"}
Oct 30 18:57:52.306711 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:52.300663    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx" event={"ID":"2d4927ad-ead6-477f-8105-4cdd3daadea5","Type":"ContainerDied","Data":"d540e119762d2f4a3c87a7147d05c6e0a6acad805352038d834f68a9d5e5045d"}
Oct 30 18:57:52.306711 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:52.300718    4525 scope.go:122] "RemoveContainer" containerID="b994d5f9166561d9b27b39cbee3ed61b6ab8e978e9ee2edd0f183935b47f78df"
Oct 30 18:57:52.306711 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:52.300875    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx"
Oct 30 18:57:52.306711 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:52.303178    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e7e190-swpr4"
Oct 30 18:57:52.306711 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:52.303194    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e7e190-swpr4" event={"ID":"614a95b4-19f0-4490-b402-e681c1bc6e39","Type":"ContainerDied","Data":"dc0cc59cd716d373e05701e840d853a1703be1b1b33587d9576ac80b49c65a0c"}
Oct 30 18:57:52.306711 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:52.303238    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc0cc59cd716d373e05701e840d853a1703be1b1b33587d9576ac80b49c65a0c"
Oct 30 18:57:52.316563 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:52.316538    4525 scope.go:122] "RemoveContainer" containerID="7024b7c83690f48ca73015a8dcfe2ab56c7816c4e0b933f9c0cff271502866ac"
Oct 30 18:57:52.330722 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:52.330698    4525 scope.go:122] "RemoveContainer" containerID="b994d5f9166561d9b27b39cbee3ed61b6ab8e978e9ee2edd0f183935b47f78df"
Oct 30 18:57:52.331037 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:57:52.331015    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b994d5f9166561d9b27b39cbee3ed61b6ab8e978e9ee2edd0f183935b47f78df\": container with ID starting with b994d5f9166561d9b27b39cbee3ed61b6ab8e978e9ee2edd0f183935b47f78df not found: ID does not exist" containerID="b994d5f9166561d9b27b39cbee3ed61b6ab8e978e9ee2edd0f183935b47f78df"
Oct 30 18:57:52.331145 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:52.331048    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b994d5f9166561d9b27b39cbee3ed61b6ab8e978e9ee2edd0f183935b47f78df"} err="failed to get container status \"b994d5f9166561d9b27b39cbee3ed61b6ab8e978e9ee2edd0f183935b47f78df\": rpc error: code = NotFound desc = could not find container \"b994d5f9166561d9b27b39cbee3ed61b6ab8e978e9ee2edd0f183935b47f78df\": container with ID starting with b994d5f9166561d9b27b39cbee3ed61b6ab8e978e9ee2edd0f183935b47f78df not found: ID does not exist"
Oct 30 18:57:52.331145 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:52.331070    4525 scope.go:122] "RemoveContainer" containerID="7024b7c83690f48ca73015a8dcfe2ab56c7816c4e0b933f9c0cff271502866ac"
Oct 30 18:57:52.331442 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:57:52.331419    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7024b7c83690f48ca73015a8dcfe2ab56c7816c4e0b933f9c0cff271502866ac\": container with ID starting with 7024b7c83690f48ca73015a8dcfe2ab56c7816c4e0b933f9c0cff271502866ac not found: ID does not exist" containerID="7024b7c83690f48ca73015a8dcfe2ab56c7816c4e0b933f9c0cff271502866ac"
Oct 30 18:57:52.331502 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:52.331452    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7024b7c83690f48ca73015a8dcfe2ab56c7816c4e0b933f9c0cff271502866ac"} err="failed to get container status \"7024b7c83690f48ca73015a8dcfe2ab56c7816c4e0b933f9c0cff271502866ac\": rpc error: code = NotFound desc = could not find container \"7024b7c83690f48ca73015a8dcfe2ab56c7816c4e0b933f9c0cff271502866ac\": container with ID starting with 7024b7c83690f48ca73015a8dcfe2ab56c7816c4e0b933f9c0cff271502866ac not found: ID does not exist"
Oct 30 18:57:52.347194 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:52.347171    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d4927ad-ead6-477f-8105-4cdd3daadea5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d4927ad-ead6-477f-8105-4cdd3daadea5-istio-data\") pod \"2d4927ad-ead6-477f-8105-4cdd3daadea5\" (UID: \"2d4927ad-ead6-477f-8105-4cdd3daadea5\") "
Oct 30 18:57:52.347317 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:52.347212    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d4927ad-ead6-477f-8105-4cdd3daadea5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d4927ad-ead6-477f-8105-4cdd3daadea5-workload-certs\") pod \"2d4927ad-ead6-477f-8105-4cdd3daadea5\" (UID: \"2d4927ad-ead6-477f-8105-4cdd3daadea5\") "
Oct 30 18:57:52.347317 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:52.347277    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2d4927ad-ead6-477f-8105-4cdd3daadea5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d4927ad-ead6-477f-8105-4cdd3daadea5-istio-podinfo\") pod \"2d4927ad-ead6-477f-8105-4cdd3daadea5\" (UID: \"2d4927ad-ead6-477f-8105-4cdd3daadea5\") "
Oct 30 18:57:52.347429 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:52.347384    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d4927ad-ead6-477f-8105-4cdd3daadea5-kube-api-access-tsmjp\" (UniqueName: \"kubernetes.io/projected/2d4927ad-ead6-477f-8105-4cdd3daadea5-kube-api-access-tsmjp\") pod \"2d4927ad-ead6-477f-8105-4cdd3daadea5\" (UID: \"2d4927ad-ead6-477f-8105-4cdd3daadea5\") "
Oct 30 18:57:52.347482 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:52.347457    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d4927ad-ead6-477f-8105-4cdd3daadea5-istio-data" pod "2d4927ad-ead6-477f-8105-4cdd3daadea5" (UID: "2d4927ad-ead6-477f-8105-4cdd3daadea5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:57:52.347573 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:52.347554    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d4927ad-ead6-477f-8105-4cdd3daadea5-istio-token\" (UniqueName: \"kubernetes.io/projected/2d4927ad-ead6-477f-8105-4cdd3daadea5-istio-token\") pod \"2d4927ad-ead6-477f-8105-4cdd3daadea5\" (UID: \"2d4927ad-ead6-477f-8105-4cdd3daadea5\") "
Oct 30 18:57:52.347610 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:52.347599    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d4927ad-ead6-477f-8105-4cdd3daadea5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d4927ad-ead6-477f-8105-4cdd3daadea5-workload-socket\") pod \"2d4927ad-ead6-477f-8105-4cdd3daadea5\" (UID: \"2d4927ad-ead6-477f-8105-4cdd3daadea5\") "
Oct 30 18:57:52.347840 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:52.347734    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d4927ad-ead6-477f-8105-4cdd3daadea5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d4927ad-ead6-477f-8105-4cdd3daadea5-istio-envoy\") pod \"2d4927ad-ead6-477f-8105-4cdd3daadea5\" (UID: \"2d4927ad-ead6-477f-8105-4cdd3daadea5\") "
Oct 30 18:57:52.347945 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:52.347897    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d4927ad-ead6-477f-8105-4cdd3daadea5-workload-socket" pod "2d4927ad-ead6-477f-8105-4cdd3daadea5" (UID: "2d4927ad-ead6-477f-8105-4cdd3daadea5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:57:52.348461 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:52.348055    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d4927ad-ead6-477f-8105-4cdd3daadea5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d4927ad-ead6-477f-8105-4cdd3daadea5-istiod-ca-cert\") pod \"2d4927ad-ead6-477f-8105-4cdd3daadea5\" (UID: \"2d4927ad-ead6-477f-8105-4cdd3daadea5\") "
Oct 30 18:57:52.348461 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:52.348097    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d4927ad-ead6-477f-8105-4cdd3daadea5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d4927ad-ead6-477f-8105-4cdd3daadea5-credential-socket\") pod \"2d4927ad-ead6-477f-8105-4cdd3daadea5\" (UID: \"2d4927ad-ead6-477f-8105-4cdd3daadea5\") "
Oct 30 18:57:52.348461 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:52.348425    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d4927ad-ead6-477f-8105-4cdd3daadea5-istiod-ca-cert" pod "2d4927ad-ead6-477f-8105-4cdd3daadea5" (UID: "2d4927ad-ead6-477f-8105-4cdd3daadea5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:57:52.348696 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:52.348589    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d4927ad-ead6-477f-8105-4cdd3daadea5-credential-socket" pod "2d4927ad-ead6-477f-8105-4cdd3daadea5" (UID: "2d4927ad-ead6-477f-8105-4cdd3daadea5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:57:52.348851 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:52.348830    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d4927ad-ead6-477f-8105-4cdd3daadea5-workload-certs" pod "2d4927ad-ead6-477f-8105-4cdd3daadea5" (UID: "2d4927ad-ead6-477f-8105-4cdd3daadea5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:57:52.349000 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:52.348972    4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d4927ad-ead6-477f-8105-4cdd3daadea5-istiod-ca-cert\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:52.349000 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:52.349001    4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d4927ad-ead6-477f-8105-4cdd3daadea5-credential-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:52.349172 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:52.349015    4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d4927ad-ead6-477f-8105-4cdd3daadea5-istio-data\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:52.349172 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:52.349029    4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d4927ad-ead6-477f-8105-4cdd3daadea5-workload-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:52.350021 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:52.349985    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d4927ad-ead6-477f-8105-4cdd3daadea5-kube-api-access-tsmjp" pod "2d4927ad-ead6-477f-8105-4cdd3daadea5" (UID: "2d4927ad-ead6-477f-8105-4cdd3daadea5"). InnerVolumeSpecName "kube-api-access-tsmjp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:57:52.350289 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:52.350239    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d4927ad-ead6-477f-8105-4cdd3daadea5-istio-podinfo" pod "2d4927ad-ead6-477f-8105-4cdd3daadea5" (UID: "2d4927ad-ead6-477f-8105-4cdd3daadea5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 18:57:52.350473 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:52.350424    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d4927ad-ead6-477f-8105-4cdd3daadea5-istio-token" pod "2d4927ad-ead6-477f-8105-4cdd3daadea5" (UID: "2d4927ad-ead6-477f-8105-4cdd3daadea5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:57:52.351117 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:52.351088    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d4927ad-ead6-477f-8105-4cdd3daadea5-istio-envoy" pod "2d4927ad-ead6-477f-8105-4cdd3daadea5" (UID: "2d4927ad-ead6-477f-8105-4cdd3daadea5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:57:52.450888 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:52.450781    4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d4927ad-ead6-477f-8105-4cdd3daadea5-istio-envoy\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:52.450888 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:52.450821    4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d4927ad-ead6-477f-8105-4cdd3daadea5-workload-certs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:52.450888 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:52.450837    4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d4927ad-ead6-477f-8105-4cdd3daadea5-istio-podinfo\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:52.450888 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:52.450851    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsmjp\" (UniqueName: \"kubernetes.io/projected/2d4927ad-ead6-477f-8105-4cdd3daadea5-kube-api-access-tsmjp\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:52.450888 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:52.450867    4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d4927ad-ead6-477f-8105-4cdd3daadea5-istio-token\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:52.626756 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:52.626721    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx"]
Oct 30 18:57:52.631498 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:52.631464    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-040d9c-openshift-default-5f6d95fff9-2pjrx"]
Oct 30 18:57:53.546753 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:53.546708    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d4927ad-ead6-477f-8105-4cdd3daadea5" path="/var/lib/kubelet/pods/2d4927ad-ead6-477f-8105-4cdd3daadea5/volumes"
Oct 30 18:57:56.347987 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:56.347880    4525 generic.go:356] "Generic (PLEG): container finished" podID="cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245" containerID="ccf8abf7f63a34b18fa795c7b6e0a9a68ae6d15366245e90a8ef885ff6e026cf" exitCode=0
Oct 30 18:57:56.348749 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:56.347958    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-12a6e1-openshift-default-5d85cdb88c-85478" event={"ID":"cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245","Type":"ContainerDied","Data":"ccf8abf7f63a34b18fa795c7b6e0a9a68ae6d15366245e90a8ef885ff6e026cf"}
Oct 30 18:57:56.348749 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:56.348585    4525 scope.go:122] "RemoveContainer" containerID="ccf8abf7f63a34b18fa795c7b6e0a9a68ae6d15366245e90a8ef885ff6e026cf"
Oct 30 18:57:56.350720 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:56.350653    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:57:56.351003 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:56.350786    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:57:56.351003 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:56.350839    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:57:57.410387 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:57.410342    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-12a6e1-openshift-default-5d85cdb88c-85478" event={"ID":"cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245","Type":"ContainerStarted","Data":"5938ab25204596f14251be67359f4ad4b4dbab1d8255124b4769285d1e5c33e0"}
Oct 30 18:57:58.140532 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:58.140495    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-12a6e1-openshift-default-5d85cdb88c-85478"
Oct 30 18:57:58.142111 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:58.142084    4525 patch_prober.go:29] interesting pod/e2e-api-b-12a6e1-openshift-default-5d85cdb88c-85478 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Oct 30 18:57:58.142240 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:58.142135    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-12a6e1-openshift-default-5d85cdb88c-85478" podUID="cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Oct 30 18:57:59.141140 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:59.141106    4525 patch_prober.go:29] interesting pod/e2e-api-b-12a6e1-openshift-default-5d85cdb88c-85478 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Oct 30 18:57:59.141583 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:57:59.141163    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-12a6e1-openshift-default-5d85cdb88c-85478" podUID="cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Oct 30 18:58:00.035709 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:00.029433    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e7e190-swpr4"]
Oct 30 18:58:00.098547 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:00.095886    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e7e190-swpr4"]
Oct 30 18:58:00.140661 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:00.140610    4525 patch_prober.go:29] interesting pod/e2e-api-b-12a6e1-openshift-default-5d85cdb88c-85478 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Oct 30 18:58:00.140880 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:00.140705    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-12a6e1-openshift-default-5d85cdb88c-85478" podUID="cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Oct 30 18:58:01.140984 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:01.140939    4525 patch_prober.go:29] interesting pod/e2e-api-b-12a6e1-openshift-default-5d85cdb88c-85478 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Oct 30 18:58:01.141508 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:01.141003    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-12a6e1-openshift-default-5d85cdb88c-85478" podUID="cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Oct 30 18:58:01.369580 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:01.369536    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-12a6e1-openshift-default-5d85cdb88c-85478"
Oct 30 18:58:01.449655 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:01.449570    4525 generic.go:356] "Generic (PLEG): container finished" podID="9aca1683-2018-4248-8c22-bd7a46096df3" containerID="2173c9aa51f156466dcd6a03ba7bb121bb8e74f6380583fcb039eb2bcd650caf" exitCode=0
Oct 30 18:58:01.449655 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:01.449639    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-54bc2b-openshift-default-bf84f77dc-jqfxp" event={"ID":"9aca1683-2018-4248-8c22-bd7a46096df3","Type":"ContainerDied","Data":"2173c9aa51f156466dcd6a03ba7bb121bb8e74f6380583fcb039eb2bcd650caf"}
Oct 30 18:58:01.450141 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:01.450126    4525 scope.go:122] "RemoveContainer" containerID="2173c9aa51f156466dcd6a03ba7bb121bb8e74f6380583fcb039eb2bcd650caf"
Oct 30 18:58:01.451673 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:01.451648    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:58:01.451755 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:01.451728    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:58:01.451793 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:01.451762    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:58:01.548035 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:01.548009    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="614a95b4-19f0-4490-b402-e681c1bc6e39" path="/var/lib/kubelet/pods/614a95b4-19f0-4490-b402-e681c1bc6e39/volumes"
Oct 30 18:58:02.140600 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:02.140561    4525 patch_prober.go:29] interesting pod/e2e-api-b-12a6e1-openshift-default-5d85cdb88c-85478 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Oct 30 18:58:02.140836 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:02.140616    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-12a6e1-openshift-default-5d85cdb88c-85478" podUID="cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Oct 30 18:58:02.461740 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:02.461648    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-54bc2b-openshift-default-bf84f77dc-jqfxp" event={"ID":"9aca1683-2018-4248-8c22-bd7a46096df3","Type":"ContainerStarted","Data":"075e22b9ed3d43cbd03af910a58084d3da7f6b091dbe8634deb7064d2e4a0d87"}
Oct 30 18:58:03.006710 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:03.005975    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-54bc2b-openshift-default-bf84f77dc-jqfxp"
Oct 30 18:58:03.008470 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:03.008438    4525 patch_prober.go:29] interesting pod/e2e-ipp-a-54bc2b-openshift-default-bf84f77dc-jqfxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 30 18:58:03.008662 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:03.008498    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-54bc2b-openshift-default-bf84f77dc-jqfxp" podUID="9aca1683-2018-4248-8c22-bd7a46096df3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 30 18:58:03.145484 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:03.145451    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-12a6e1-openshift-default-5d85cdb88c-85478"
Oct 30 18:58:03.146698 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:03.146663    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-12a6e1-openshift-default-5d85cdb88c-85478"
Oct 30 18:58:03.218016 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:03.217981    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-12a6e1-llk89"]
Oct 30 18:58:03.223665 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:03.223633    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-12a6e1-llk89"]
Oct 30 18:58:03.622933 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:03.622895    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2e90b1d-8d01-4b0b-b661-287c52b8bb4f" path="/var/lib/kubelet/pods/a2e90b1d-8d01-4b0b-b661-287c52b8bb4f/volumes"
Oct 30 18:58:03.995778 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:03.995653    4525 patch_prober.go:29] interesting pod/e2e-ipp-a-54bc2b-openshift-default-bf84f77dc-jqfxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 30 18:58:03.996067 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:03.996032    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-54bc2b-openshift-default-bf84f77dc-jqfxp" podUID="9aca1683-2018-4248-8c22-bd7a46096df3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 30 18:58:04.462007 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:04.461974    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-54bc2b-openshift-default-bf84f77dc-jqfxp"
Oct 30 18:58:04.995096 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:04.995058    4525 patch_prober.go:29] interesting pod/e2e-ipp-a-54bc2b-openshift-default-bf84f77dc-jqfxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 30 18:58:04.996778 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:04.996742    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-54bc2b-openshift-default-bf84f77dc-jqfxp" podUID="9aca1683-2018-4248-8c22-bd7a46096df3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 30 18:58:05.988706 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:05.984245    4525 patch_prober.go:29] interesting pod/e2e-ipp-a-54bc2b-openshift-default-bf84f77dc-jqfxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 30 18:58:05.988706 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:05.984336    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-54bc2b-openshift-default-bf84f77dc-jqfxp" podUID="9aca1683-2018-4248-8c22-bd7a46096df3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 30 18:58:06.993705 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:06.986063    4525 patch_prober.go:29] interesting pod/e2e-ipp-a-54bc2b-openshift-default-bf84f77dc-jqfxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 30 18:58:06.993705 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:06.986140    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-54bc2b-openshift-default-bf84f77dc-jqfxp" podUID="9aca1683-2018-4248-8c22-bd7a46096df3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 30 18:58:07.982015 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:07.981969    4525 patch_prober.go:29] interesting pod/e2e-ipp-a-54bc2b-openshift-default-bf84f77dc-jqfxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 30 18:58:07.982218 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:07.982038    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-54bc2b-openshift-default-bf84f77dc-jqfxp" podUID="9aca1683-2018-4248-8c22-bd7a46096df3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 30 18:58:09.077741 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:09.074173    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-54bc2b-openshift-default-bf84f77dc-jqfxp"
Oct 30 18:58:09.097334 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:09.097306    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-54bc2b-openshift-default-bf84f77dc-jqfxp"
Oct 30 18:58:10.403511 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:10.403471    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-a014d1-openshift-default-566cdcf765-4c97j"]
Oct 30 18:58:10.404516 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:10.404480    4525 cpu_manager.go:414] "containerMap: removing container" podUID="81739e60-d074-4a42-ade3-a0f68ea9329d" containerName="istio-proxy"
Oct 30 18:58:10.404516 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:10.404506    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81739e60-d074-4a42-ade3-a0f68ea9329d" containerName="istio-proxy"
Oct 30 18:58:10.404516 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:10.404519    4525 cpu_manager.go:414] "containerMap: removing container" podUID="614a95b4-19f0-4490-b402-e681c1bc6e39" containerName="revoke-keys"
Oct 30 18:58:10.404832 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:10.404527    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="614a95b4-19f0-4490-b402-e681c1bc6e39" containerName="revoke-keys"
Oct 30 18:58:10.404832 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:10.404549    4525 cpu_manager.go:414] "containerMap: removing container" podUID="81739e60-d074-4a42-ade3-a0f68ea9329d" containerName="istio-proxy"
Oct 30 18:58:10.404832 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:10.404556    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81739e60-d074-4a42-ade3-a0f68ea9329d" containerName="istio-proxy"
Oct 30 18:58:10.404832 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:10.404571    4525 cpu_manager.go:414] "containerMap: removing container" podUID="2d4927ad-ead6-477f-8105-4cdd3daadea5" containerName="istio-proxy"
Oct 30 18:58:10.404832 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:10.404578    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d4927ad-ead6-477f-8105-4cdd3daadea5" containerName="istio-proxy"
Oct 30 18:58:10.404832 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:10.404599    4525 cpu_manager.go:414] "containerMap: removing container" podUID="9c3b7950-7451-4f37-8119-f9574c3045f3" containerName="istio-proxy"
Oct 30 18:58:10.404832 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:10.404607    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c3b7950-7451-4f37-8119-f9574c3045f3" containerName="istio-proxy"
Oct 30 18:58:10.404832 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:10.404635    4525 cpu_manager.go:414] "containerMap: removing container" podUID="2d4927ad-ead6-477f-8105-4cdd3daadea5" containerName="istio-proxy"
Oct 30 18:58:10.404832 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:10.404641    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d4927ad-ead6-477f-8105-4cdd3daadea5" containerName="istio-proxy"
Oct 30 18:58:10.404832 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:10.404811    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c3b7950-7451-4f37-8119-f9574c3045f3" containerName="istio-proxy"
Oct 30 18:58:10.405440 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:10.404962    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d4927ad-ead6-477f-8105-4cdd3daadea5" containerName="istio-proxy"
Oct 30 18:58:10.405440 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:10.404987    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="81739e60-d074-4a42-ade3-a0f68ea9329d" containerName="istio-proxy"
Oct 30 18:58:10.405440 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:10.404998    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="614a95b4-19f0-4490-b402-e681c1bc6e39" containerName="revoke-keys"
Oct 30 18:58:10.405591 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:10.405578    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="81739e60-d074-4a42-ade3-a0f68ea9329d" containerName="istio-proxy"
Oct 30 18:58:10.405644 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:10.405598    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d4927ad-ead6-477f-8105-4cdd3daadea5" containerName="istio-proxy"
Oct 30 18:58:10.409240 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:10.409220    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-a014d1-openshift-default-566cdcf765-4c97j"
Oct 30 18:58:10.420294 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:10.418986    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-a014d1-openshift-default-dockercfg-pbb72\""
Oct 30 18:58:10.428334 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:10.428306    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-a014d1-openshift-default-566cdcf765-4c97j"]
Oct 30 18:58:10.569707 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:10.555607    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/116d5ed7-fbe2-4a19-8985-6fc29c87912d-workload-certs\") pod \"e2e-praxis-b-a014d1-openshift-default-566cdcf765-4c97j\" (UID: \"116d5ed7-fbe2-4a19-8985-6fc29c87912d\") " pod="openshift-ingress/e2e-praxis-b-a014d1-openshift-default-566cdcf765-4c97j"
Oct 30 18:58:10.570046 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:10.570009    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk45p\" (UniqueName: \"kubernetes.io/projected/116d5ed7-fbe2-4a19-8985-6fc29c87912d-kube-api-access-nk45p\") pod \"e2e-praxis-b-a014d1-openshift-default-566cdcf765-4c97j\" (UID: \"116d5ed7-fbe2-4a19-8985-6fc29c87912d\") " pod="openshift-ingress/e2e-praxis-b-a014d1-openshift-default-566cdcf765-4c97j"
Oct 30 18:58:10.570357 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:10.570337    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/116d5ed7-fbe2-4a19-8985-6fc29c87912d-istio-token\") pod \"e2e-praxis-b-a014d1-openshift-default-566cdcf765-4c97j\" (UID: \"116d5ed7-fbe2-4a19-8985-6fc29c87912d\") " pod="openshift-ingress/e2e-praxis-b-a014d1-openshift-default-566cdcf765-4c97j"
Oct 30 18:58:10.570632 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:10.570611    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/116d5ed7-fbe2-4a19-8985-6fc29c87912d-credential-socket\") pod \"e2e-praxis-b-a014d1-openshift-default-566cdcf765-4c97j\" (UID: \"116d5ed7-fbe2-4a19-8985-6fc29c87912d\") " pod="openshift-ingress/e2e-praxis-b-a014d1-openshift-default-566cdcf765-4c97j"
Oct 30 18:58:10.585705 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:10.570896    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/116d5ed7-fbe2-4a19-8985-6fc29c87912d-istio-podinfo\") pod \"e2e-praxis-b-a014d1-openshift-default-566cdcf765-4c97j\" (UID: \"116d5ed7-fbe2-4a19-8985-6fc29c87912d\") " pod="openshift-ingress/e2e-praxis-b-a014d1-openshift-default-566cdcf765-4c97j"
Oct 30 18:58:10.586026 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:10.585998    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/116d5ed7-fbe2-4a19-8985-6fc29c87912d-istiod-ca-cert\") pod \"e2e-praxis-b-a014d1-openshift-default-566cdcf765-4c97j\" (UID: \"116d5ed7-fbe2-4a19-8985-6fc29c87912d\") " pod="openshift-ingress/e2e-praxis-b-a014d1-openshift-default-566cdcf765-4c97j"
Oct 30 18:58:10.586230 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:10.586210    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/116d5ed7-fbe2-4a19-8985-6fc29c87912d-istio-data\") pod \"e2e-praxis-b-a014d1-openshift-default-566cdcf765-4c97j\" (UID: \"116d5ed7-fbe2-4a19-8985-6fc29c87912d\") " pod="openshift-ingress/e2e-praxis-b-a014d1-openshift-default-566cdcf765-4c97j"
Oct 30 18:58:10.586418 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:10.586405    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/116d5ed7-fbe2-4a19-8985-6fc29c87912d-workload-socket\") pod \"e2e-praxis-b-a014d1-openshift-default-566cdcf765-4c97j\" (UID: \"116d5ed7-fbe2-4a19-8985-6fc29c87912d\") " pod="openshift-ingress/e2e-praxis-b-a014d1-openshift-default-566cdcf765-4c97j"
Oct 30 18:58:10.586540 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:10.586527    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/116d5ed7-fbe2-4a19-8985-6fc29c87912d-istio-envoy\") pod \"e2e-praxis-b-a014d1-openshift-default-566cdcf765-4c97j\" (UID: \"116d5ed7-fbe2-4a19-8985-6fc29c87912d\") " pod="openshift-ingress/e2e-praxis-b-a014d1-openshift-default-566cdcf765-4c97j"
Oct 30 18:58:10.693515 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:10.693420    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/116d5ed7-fbe2-4a19-8985-6fc29c87912d-istio-podinfo\") pod \"e2e-praxis-b-a014d1-openshift-default-566cdcf765-4c97j\" (UID: \"116d5ed7-fbe2-4a19-8985-6fc29c87912d\") " pod="openshift-ingress/e2e-praxis-b-a014d1-openshift-default-566cdcf765-4c97j"
Oct 30 18:58:10.694436 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:10.694414    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/116d5ed7-fbe2-4a19-8985-6fc29c87912d-istiod-ca-cert\") pod \"e2e-praxis-b-a014d1-openshift-default-566cdcf765-4c97j\" (UID: \"116d5ed7-fbe2-4a19-8985-6fc29c87912d\") " pod="openshift-ingress/e2e-praxis-b-a014d1-openshift-default-566cdcf765-4c97j"
Oct 30 18:58:10.694648 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:10.694634    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/116d5ed7-fbe2-4a19-8985-6fc29c87912d-istio-data\") pod \"e2e-praxis-b-a014d1-openshift-default-566cdcf765-4c97j\" (UID: \"116d5ed7-fbe2-4a19-8985-6fc29c87912d\") " pod="openshift-ingress/e2e-praxis-b-a014d1-openshift-default-566cdcf765-4c97j"
Oct 30 18:58:10.694858 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:10.694845    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/116d5ed7-fbe2-4a19-8985-6fc29c87912d-workload-socket\") pod \"e2e-praxis-b-a014d1-openshift-default-566cdcf765-4c97j\" (UID: \"116d5ed7-fbe2-4a19-8985-6fc29c87912d\") " pod="openshift-ingress/e2e-praxis-b-a014d1-openshift-default-566cdcf765-4c97j"
Oct 30 18:58:10.700745 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:10.694939    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/116d5ed7-fbe2-4a19-8985-6fc29c87912d-istio-envoy\") pod \"e2e-praxis-b-a014d1-openshift-default-566cdcf765-4c97j\" (UID: \"116d5ed7-fbe2-4a19-8985-6fc29c87912d\") " pod="openshift-ingress/e2e-praxis-b-a014d1-openshift-default-566cdcf765-4c97j"
Oct 30 18:58:10.710896 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:10.701621    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/116d5ed7-fbe2-4a19-8985-6fc29c87912d-istio-podinfo\") pod \"e2e-praxis-b-a014d1-openshift-default-566cdcf765-4c97j\" (UID: \"116d5ed7-fbe2-4a19-8985-6fc29c87912d\") " pod="openshift-ingress/e2e-praxis-b-a014d1-openshift-default-566cdcf765-4c97j"
Oct 30 18:58:10.711149 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:10.702475    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/116d5ed7-fbe2-4a19-8985-6fc29c87912d-istiod-ca-cert\") pod \"e2e-praxis-b-a014d1-openshift-default-566cdcf765-4c97j\" (UID: \"116d5ed7-fbe2-4a19-8985-6fc29c87912d\") " pod="openshift-ingress/e2e-praxis-b-a014d1-openshift-default-566cdcf765-4c97j"
Oct 30 18:58:10.711149 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:10.703146    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/116d5ed7-fbe2-4a19-8985-6fc29c87912d-istio-data\") pod \"e2e-praxis-b-a014d1-openshift-default-566cdcf765-4c97j\" (UID: \"116d5ed7-fbe2-4a19-8985-6fc29c87912d\") " pod="openshift-ingress/e2e-praxis-b-a014d1-openshift-default-566cdcf765-4c97j"
Oct 30 18:58:10.711149 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:10.703456    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/116d5ed7-fbe2-4a19-8985-6fc29c87912d-workload-socket\") pod \"e2e-praxis-b-a014d1-openshift-default-566cdcf765-4c97j\" (UID: \"116d5ed7-fbe2-4a19-8985-6fc29c87912d\") " pod="openshift-ingress/e2e-praxis-b-a014d1-openshift-default-566cdcf765-4c97j"
Oct 30 18:58:10.711975 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:10.711948    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/116d5ed7-fbe2-4a19-8985-6fc29c87912d-workload-certs\") pod \"e2e-praxis-b-a014d1-openshift-default-566cdcf765-4c97j\" (UID: \"116d5ed7-fbe2-4a19-8985-6fc29c87912d\") " pod="openshift-ingress/e2e-praxis-b-a014d1-openshift-default-566cdcf765-4c97j"
Oct 30 18:58:10.712751 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:10.712436    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk45p\" (UniqueName: \"kubernetes.io/projected/116d5ed7-fbe2-4a19-8985-6fc29c87912d-kube-api-access-nk45p\") pod \"e2e-praxis-b-a014d1-openshift-default-566cdcf765-4c97j\" (UID: \"116d5ed7-fbe2-4a19-8985-6fc29c87912d\") " pod="openshift-ingress/e2e-praxis-b-a014d1-openshift-default-566cdcf765-4c97j"
Oct 30 18:58:10.712938 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:10.712361    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/116d5ed7-fbe2-4a19-8985-6fc29c87912d-workload-certs\") pod \"e2e-praxis-b-a014d1-openshift-default-566cdcf765-4c97j\" (UID: \"116d5ed7-fbe2-4a19-8985-6fc29c87912d\") " pod="openshift-ingress/e2e-praxis-b-a014d1-openshift-default-566cdcf765-4c97j"
Oct 30 18:58:10.713094 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:10.713072    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/116d5ed7-fbe2-4a19-8985-6fc29c87912d-istio-token\") pod \"e2e-praxis-b-a014d1-openshift-default-566cdcf765-4c97j\" (UID: \"116d5ed7-fbe2-4a19-8985-6fc29c87912d\") " pod="openshift-ingress/e2e-praxis-b-a014d1-openshift-default-566cdcf765-4c97j"
Oct 30 18:58:10.713962 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:10.713941    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/116d5ed7-fbe2-4a19-8985-6fc29c87912d-credential-socket\") pod \"e2e-praxis-b-a014d1-openshift-default-566cdcf765-4c97j\" (UID: \"116d5ed7-fbe2-4a19-8985-6fc29c87912d\") " pod="openshift-ingress/e2e-praxis-b-a014d1-openshift-default-566cdcf765-4c97j"
Oct 30 18:58:10.715083 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:10.715060    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/116d5ed7-fbe2-4a19-8985-6fc29c87912d-credential-socket\") pod \"e2e-praxis-b-a014d1-openshift-default-566cdcf765-4c97j\" (UID: \"116d5ed7-fbe2-4a19-8985-6fc29c87912d\") " pod="openshift-ingress/e2e-praxis-b-a014d1-openshift-default-566cdcf765-4c97j"
Oct 30 18:58:10.718416 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:10.718396    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/116d5ed7-fbe2-4a19-8985-6fc29c87912d-istio-envoy\") pod \"e2e-praxis-b-a014d1-openshift-default-566cdcf765-4c97j\" (UID: \"116d5ed7-fbe2-4a19-8985-6fc29c87912d\") " pod="openshift-ingress/e2e-praxis-b-a014d1-openshift-default-566cdcf765-4c97j"
Oct 30 18:58:10.725073 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:10.724944    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk45p\" (UniqueName: \"kubernetes.io/projected/116d5ed7-fbe2-4a19-8985-6fc29c87912d-kube-api-access-nk45p\") pod \"e2e-praxis-b-a014d1-openshift-default-566cdcf765-4c97j\" (UID: \"116d5ed7-fbe2-4a19-8985-6fc29c87912d\") " pod="openshift-ingress/e2e-praxis-b-a014d1-openshift-default-566cdcf765-4c97j"
Oct 30 18:58:10.729881 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:10.729858    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/116d5ed7-fbe2-4a19-8985-6fc29c87912d-istio-token\") pod \"e2e-praxis-b-a014d1-openshift-default-566cdcf765-4c97j\" (UID: \"116d5ed7-fbe2-4a19-8985-6fc29c87912d\") " pod="openshift-ingress/e2e-praxis-b-a014d1-openshift-default-566cdcf765-4c97j"
Oct 30 18:58:11.029195 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:11.029101    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-a014d1-openshift-default-566cdcf765-4c97j"
Oct 30 18:58:11.030205 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:11.030172    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a325894f-sm5gh"]
Oct 30 18:58:11.035915 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:11.035886    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a325894f-sm5gh"]
Oct 30 18:58:11.566756 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:11.565776    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2438ddf-fbd5-4d40-b271-cbdb10efd74f" path="/var/lib/kubelet/pods/a2438ddf-fbd5-4d40-b271-cbdb10efd74f/volumes"
Oct 30 18:58:11.566756 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:11.566237    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-12a6e1-openshift-default-5d85cdb88c-85478"]
Oct 30 18:58:11.566756 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:11.566478    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-12a6e1-openshift-default-5d85cdb88c-85478" podUID="cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245" containerName="istio-proxy" containerID="cri-o://5938ab25204596f14251be67359f4ad4b4dbab1d8255124b4769285d1e5c33e0" gracePeriod=30
Oct 30 18:58:11.612110 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:11.612069    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-a014d1-openshift-default-566cdcf765-4c97j"]
Oct 30 18:58:11.855124 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:58:11.854912    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod116d5ed7_fbe2_4a19_8985_6fc29c87912d.slice/crio-7dfce15f057502e7afa9da0019783a0ec5674c82ff0696a1f20d4fa91d153e45 WatchSource:0}: Error finding container 7dfce15f057502e7afa9da0019783a0ec5674c82ff0696a1f20d4fa91d153e45: Status 404 returned error can't find the container with id 7dfce15f057502e7afa9da0019783a0ec5674c82ff0696a1f20d4fa91d153e45
Oct 30 18:58:11.907486 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:11.903024    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:58:11.907486 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:11.903137    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:58:11.907486 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:11.903187    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:58:12.907393 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:12.907357    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-a014d1-openshift-default-566cdcf765-4c97j" event={"ID":"116d5ed7-fbe2-4a19-8985-6fc29c87912d","Type":"ContainerStarted","Data":"9e8bddf78828caf065396ba20548508f9b8ad9dd188fe59b4febb72343c4b0de"}
Oct 30 18:58:12.907946 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:12.907420    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-a014d1-openshift-default-566cdcf765-4c97j" event={"ID":"116d5ed7-fbe2-4a19-8985-6fc29c87912d","Type":"ContainerStarted","Data":"7dfce15f057502e7afa9da0019783a0ec5674c82ff0696a1f20d4fa91d153e45"}
Oct 30 18:58:12.931079 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:12.931018    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-a014d1-openshift-default-566cdcf765-4c97j" podStartSLOduration=2.930999576 podStartE2EDuration="2.930999576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:58:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:58:12.929836523 +0000 UTC m=+1315.555506489" watchObservedRunningTime="2026-10-30 18:58:12.930999576 +0000 UTC m=+1315.556669540"
Oct 30 18:58:13.029557 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:13.029519    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-a014d1-openshift-default-566cdcf765-4c97j"
Oct 30 18:58:13.040092 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:13.040059    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-a014d1-openshift-default-566cdcf765-4c97j"
Oct 30 18:58:14.120256 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:14.120076    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-a014d1-openshift-default-566cdcf765-4c97j"
Oct 30 18:58:14.130714 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:14.128875    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-a014d1-openshift-default-566cdcf765-4c97j"
Oct 30 18:58:17.367098 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:17.366595    4525 generic.go:356] "Generic (PLEG): container finished" podID="cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245" containerID="5938ab25204596f14251be67359f4ad4b4dbab1d8255124b4769285d1e5c33e0" exitCode=0
Oct 30 18:58:17.367098 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:17.366886    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-12a6e1-openshift-default-5d85cdb88c-85478" event={"ID":"cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245","Type":"ContainerDied","Data":"5938ab25204596f14251be67359f4ad4b4dbab1d8255124b4769285d1e5c33e0"}
Oct 30 18:58:17.367098 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:17.366955    4525 scope.go:122] "RemoveContainer" containerID="ccf8abf7f63a34b18fa795c7b6e0a9a68ae6d15366245e90a8ef885ff6e026cf"
Oct 30 18:58:17.410566 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:58:17.410522    4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d49l" containerName="extractor"
Oct 30 18:58:17.458340 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:17.458312    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-12a6e1-openshift-default-5d85cdb88c-85478"
Oct 30 18:58:17.545103 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:17.545072    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245-istio-data\") pod \"cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245\" (UID: \"cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245\") "
Oct 30 18:58:17.545265 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:17.545175    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245-workload-socket\") pod \"cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245\" (UID: \"cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245\") "
Oct 30 18:58:17.545265 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:17.545211    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245-istio-token\" (UniqueName: \"kubernetes.io/projected/cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245-istio-token\") pod \"cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245\" (UID: \"cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245\") "
Oct 30 18:58:17.545353 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:17.545312    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245-credential-socket\") pod \"cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245\" (UID: \"cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245\") "
Oct 30 18:58:17.545392 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:17.545341    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245-istio-data" pod "cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245" (UID: "cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:58:17.545445 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:17.545349    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245-istio-podinfo\") pod \"cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245\" (UID: \"cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245\") "
Oct 30 18:58:17.545500 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:17.545451    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245-workload-certs\") pod \"cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245\" (UID: \"cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245\") "
Oct 30 18:58:17.545500 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:17.545489    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245-kube-api-access-vmbct\" (UniqueName: \"kubernetes.io/projected/cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245-kube-api-access-vmbct\") pod \"cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245\" (UID: \"cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245\") "
Oct 30 18:58:17.545609 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:17.545517    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245-istio-envoy\") pod \"cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245\" (UID: \"cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245\") "
Oct 30 18:58:17.545609 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:17.545553    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245-workload-socket" pod "cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245" (UID: "cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:58:17.545609 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:17.545581    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245-istiod-ca-cert\") pod \"cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245\" (UID: \"cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245\") "
Oct 30 18:58:17.546371 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:17.545985    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245-credential-socket" pod "cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245" (UID: "cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:58:17.546371 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:17.546244    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245-workload-certs" pod "cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245" (UID: "cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:58:17.546371 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:17.546334    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245-istiod-ca-cert" pod "cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245" (UID: "cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:58:17.546697 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:17.546650    4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245-workload-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:17.546773 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:17.546721    4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245-credential-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:17.546773 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:17.546737    4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245-workload-certs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:17.546773 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:17.546750    4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245-istiod-ca-cert\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:17.546773 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:17.546764    4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245-istio-data\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:17.547933 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:17.547902    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245-istio-podinfo" pod "cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245" (UID: "cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 18:58:17.548021 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:17.547957    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245-istio-envoy" pod "cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245" (UID: "cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:58:17.548538 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:17.548517    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245-istio-token" pod "cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245" (UID: "cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:58:17.548607 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:17.548584    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245-kube-api-access-vmbct" pod "cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245" (UID: "cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245"). InnerVolumeSpecName "kube-api-access-vmbct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:58:17.647924 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:17.647897    4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245-istio-podinfo\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:17.647924 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:17.647921    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmbct\" (UniqueName: \"kubernetes.io/projected/cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245-kube-api-access-vmbct\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:17.647924 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:17.647931    4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245-istio-envoy\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:17.648138 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:17.647939    4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245-istio-token\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:18.376651 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:18.376624    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-12a6e1-openshift-default-5d85cdb88c-85478"
Oct 30 18:58:18.377115 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:18.376618    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-12a6e1-openshift-default-5d85cdb88c-85478" event={"ID":"cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245","Type":"ContainerDied","Data":"dd8cd67351d8c9b3fabe2411e9f17056a5aae8b417ddeb1e561fc94143beaf87"}
Oct 30 18:58:18.377115 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:18.376747    4525 scope.go:122] "RemoveContainer" containerID="5938ab25204596f14251be67359f4ad4b4dbab1d8255124b4769285d1e5c33e0"
Oct 30 18:58:18.403047 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:18.403019    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-12a6e1-openshift-default-5d85cdb88c-85478"]
Oct 30 18:58:18.412290 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:18.412266    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-12a6e1-openshift-default-5d85cdb88c-85478"]
Oct 30 18:58:18.617480 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:18.617449    4525 scope.go:122] "RemoveContainer" containerID="4fcdcafd6023518035a6f2fbbb6c76d543f529bed3c425ea730d0af9d5ff9e18"
Oct 30 18:58:18.628377 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:18.628356    4525 scope.go:122] "RemoveContainer" containerID="7e6df2e9d8225b4274a90bbb6f50bcedc6c8514fd6426e951516ab0129717cde"
Oct 30 18:58:18.638940 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:18.638919    4525 scope.go:122] "RemoveContainer" containerID="87a7f5ae89f7261636d240758e6ce01b456dd637bea2ffb3bab8e614e20c5301"
Oct 30 18:58:18.648518 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:18.648497    4525 scope.go:122] "RemoveContainer" containerID="5790f8ad72e52dd61d38c16fc2586ad064bbaf7094211564e869b69c19bd4ad5"
Oct 30 18:58:18.658746 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:18.658726    4525 scope.go:122] "RemoveContainer" containerID="d691dec3ecf13a08158a3f4f5de331d98b8b011daa40adb9fb2832333133cbd9"
Oct 30 18:58:18.669349 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:18.669325    4525 scope.go:122] "RemoveContainer" containerID="b829be9404f0f5a9c47703cd5a6783f89ae17af44b029a4b2fb3fc5b8c8d68b7"
Oct 30 18:58:18.679259 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:18.679242    4525 scope.go:122] "RemoveContainer" containerID="9a3ee946b74b86bd3ae429663137c93bb300aa73afb57d25310876739db101e2"
Oct 30 18:58:18.688369 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:18.688349    4525 scope.go:122] "RemoveContainer" containerID="5334ff9564b6f59d2da80847c259f6e2f0a2470603789f84363617710c95a1c1"
Oct 30 18:58:18.697637 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:18.697622    4525 scope.go:122] "RemoveContainer" containerID="d48ef5b500331c1faf8272f0c340885b96c30959e16a97c9ec2f9a5471306cd8"
Oct 30 18:58:18.707226 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:18.707211    4525 scope.go:122] "RemoveContainer" containerID="c1836498eff8463cf8132187faf31f99c815d4c9019c23eeeaaa4f3d5195354c"
Oct 30 18:58:18.722305 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:18.722290    4525 scope.go:122] "RemoveContainer" containerID="937da3f5011bdb4c24f0ddbfe6538d366643f24333e4ca37bced02eb87a0ff74"
Oct 30 18:58:18.731896 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:18.731879    4525 scope.go:122] "RemoveContainer" containerID="05a1bb9035bdbe67bbd29f0414f8aa603d1bb96d7cd7f6c7f3ff17ba68224865"
Oct 30 18:58:19.548648 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:19.548611    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245" path="/var/lib/kubelet/pods/cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245/volumes"
Oct 30 18:58:47.398063 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:58:47.398018    4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d49l" containerName="extractor"
Oct 30 18:58:49.944674 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:49.944641    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-praxis-a-6a327a/praxis-route-6a327a-kserve-5cb7997949-4vzjq"]
Oct 30 18:58:49.945298 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:49.945282    4525 cpu_manager.go:414] "containerMap: removing container" podUID="cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245" containerName="istio-proxy"
Oct 30 18:58:49.945348 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:49.945300    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245" containerName="istio-proxy"
Oct 30 18:58:49.945348 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:49.945327    4525 cpu_manager.go:414] "containerMap: removing container" podUID="cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245" containerName="istio-proxy"
Oct 30 18:58:49.945348 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:49.945332    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245" containerName="istio-proxy"
Oct 30 18:58:49.945445 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:49.945417    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245" containerName="istio-proxy"
Oct 30 18:58:49.949264 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:49.949243    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-6a327a/praxis-route-6a327a-kserve-5cb7997949-4vzjq"
Oct 30 18:58:49.956857 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:49.956827    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-6a327a/praxis-route-6a327a-kserve-5cb7997949-4vzjq"]
Oct 30 18:58:49.957526 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:49.957354    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-6a327a\"/\"kube-root-ca.crt\""
Oct 30 18:58:49.959280 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:49.958484    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-6a327a\"/\"default-dockercfg-rhnkl\""
Oct 30 18:58:49.959280 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:49.958544    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-6a327a\"/\"openshift-service-ca.crt\""
Oct 30 18:58:49.959280 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:49.958641    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-6a327a\"/\"praxis-route-6a327a-kserve-self-signed-certs\""
Oct 30 18:58:49.988187 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:49.988142    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0dcb92f5-5de5-4c91-87cc-27764bea9683-tls-certs\") pod \"praxis-route-6a327a-kserve-5cb7997949-4vzjq\" (UID: \"0dcb92f5-5de5-4c91-87cc-27764bea9683\") " pod="ai-tenant-e2e-praxis-a-6a327a/praxis-route-6a327a-kserve-5cb7997949-4vzjq"
Oct 30 18:58:49.988382 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:49.988286    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0dcb92f5-5de5-4c91-87cc-27764bea9683-model-cache\") pod \"praxis-route-6a327a-kserve-5cb7997949-4vzjq\" (UID: \"0dcb92f5-5de5-4c91-87cc-27764bea9683\") " pod="ai-tenant-e2e-praxis-a-6a327a/praxis-route-6a327a-kserve-5cb7997949-4vzjq"
Oct 30 18:58:49.988382 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:49.988328    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b8xw\" (UniqueName: \"kubernetes.io/projected/0dcb92f5-5de5-4c91-87cc-27764bea9683-kube-api-access-9b8xw\") pod \"praxis-route-6a327a-kserve-5cb7997949-4vzjq\" (UID: \"0dcb92f5-5de5-4c91-87cc-27764bea9683\") " pod="ai-tenant-e2e-praxis-a-6a327a/praxis-route-6a327a-kserve-5cb7997949-4vzjq"
Oct 30 18:58:49.988382 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:49.988369    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dcb92f5-5de5-4c91-87cc-27764bea9683-tmp-dir\") pod \"praxis-route-6a327a-kserve-5cb7997949-4vzjq\" (UID: \"0dcb92f5-5de5-4c91-87cc-27764bea9683\") " pod="ai-tenant-e2e-praxis-a-6a327a/praxis-route-6a327a-kserve-5cb7997949-4vzjq"
Oct 30 18:58:49.988580 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:49.988516    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0dcb92f5-5de5-4c91-87cc-27764bea9683-dshm\") pod \"praxis-route-6a327a-kserve-5cb7997949-4vzjq\" (UID: \"0dcb92f5-5de5-4c91-87cc-27764bea9683\") " pod="ai-tenant-e2e-praxis-a-6a327a/praxis-route-6a327a-kserve-5cb7997949-4vzjq"
Oct 30 18:58:49.988580 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:49.988565    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0dcb92f5-5de5-4c91-87cc-27764bea9683-home\") pod \"praxis-route-6a327a-kserve-5cb7997949-4vzjq\" (UID: \"0dcb92f5-5de5-4c91-87cc-27764bea9683\") " pod="ai-tenant-e2e-praxis-a-6a327a/praxis-route-6a327a-kserve-5cb7997949-4vzjq"
Oct 30 18:58:50.093707 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:50.091768    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0dcb92f5-5de5-4c91-87cc-27764bea9683-home\") pod \"praxis-route-6a327a-kserve-5cb7997949-4vzjq\" (UID: \"0dcb92f5-5de5-4c91-87cc-27764bea9683\") " pod="ai-tenant-e2e-praxis-a-6a327a/praxis-route-6a327a-kserve-5cb7997949-4vzjq"
Oct 30 18:58:50.093707 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:50.091871    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0dcb92f5-5de5-4c91-87cc-27764bea9683-tls-certs\") pod \"praxis-route-6a327a-kserve-5cb7997949-4vzjq\" (UID: \"0dcb92f5-5de5-4c91-87cc-27764bea9683\") " pod="ai-tenant-e2e-praxis-a-6a327a/praxis-route-6a327a-kserve-5cb7997949-4vzjq"
Oct 30 18:58:50.093707 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:50.091949    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0dcb92f5-5de5-4c91-87cc-27764bea9683-model-cache\") pod \"praxis-route-6a327a-kserve-5cb7997949-4vzjq\" (UID: \"0dcb92f5-5de5-4c91-87cc-27764bea9683\") " pod="ai-tenant-e2e-praxis-a-6a327a/praxis-route-6a327a-kserve-5cb7997949-4vzjq"
Oct 30 18:58:50.093707 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:50.091985    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b8xw\" (UniqueName: \"kubernetes.io/projected/0dcb92f5-5de5-4c91-87cc-27764bea9683-kube-api-access-9b8xw\") pod \"praxis-route-6a327a-kserve-5cb7997949-4vzjq\" (UID: \"0dcb92f5-5de5-4c91-87cc-27764bea9683\") " pod="ai-tenant-e2e-praxis-a-6a327a/praxis-route-6a327a-kserve-5cb7997949-4vzjq"
Oct 30 18:58:50.093707 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:50.092042    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dcb92f5-5de5-4c91-87cc-27764bea9683-tmp-dir\") pod \"praxis-route-6a327a-kserve-5cb7997949-4vzjq\" (UID: \"0dcb92f5-5de5-4c91-87cc-27764bea9683\") " pod="ai-tenant-e2e-praxis-a-6a327a/praxis-route-6a327a-kserve-5cb7997949-4vzjq"
Oct 30 18:58:50.093707 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:50.092176    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0dcb92f5-5de5-4c91-87cc-27764bea9683-dshm\") pod \"praxis-route-6a327a-kserve-5cb7997949-4vzjq\" (UID: \"0dcb92f5-5de5-4c91-87cc-27764bea9683\") " pod="ai-tenant-e2e-praxis-a-6a327a/praxis-route-6a327a-kserve-5cb7997949-4vzjq"
Oct 30 18:58:50.104397 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:50.102237    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0dcb92f5-5de5-4c91-87cc-27764bea9683-model-cache\") pod \"praxis-route-6a327a-kserve-5cb7997949-4vzjq\" (UID: \"0dcb92f5-5de5-4c91-87cc-27764bea9683\") " pod="ai-tenant-e2e-praxis-a-6a327a/praxis-route-6a327a-kserve-5cb7997949-4vzjq"
Oct 30 18:58:50.104397 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:50.102498    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0dcb92f5-5de5-4c91-87cc-27764bea9683-home\") pod \"praxis-route-6a327a-kserve-5cb7997949-4vzjq\" (UID: \"0dcb92f5-5de5-4c91-87cc-27764bea9683\") " pod="ai-tenant-e2e-praxis-a-6a327a/praxis-route-6a327a-kserve-5cb7997949-4vzjq"
Oct 30 18:58:50.113715 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:50.106416    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dcb92f5-5de5-4c91-87cc-27764bea9683-tmp-dir\") pod \"praxis-route-6a327a-kserve-5cb7997949-4vzjq\" (UID: \"0dcb92f5-5de5-4c91-87cc-27764bea9683\") " pod="ai-tenant-e2e-praxis-a-6a327a/praxis-route-6a327a-kserve-5cb7997949-4vzjq"
Oct 30 18:58:50.122407 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:50.122313    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0dcb92f5-5de5-4c91-87cc-27764bea9683-dshm\") pod \"praxis-route-6a327a-kserve-5cb7997949-4vzjq\" (UID: \"0dcb92f5-5de5-4c91-87cc-27764bea9683\") " pod="ai-tenant-e2e-praxis-a-6a327a/praxis-route-6a327a-kserve-5cb7997949-4vzjq"
Oct 30 18:58:50.141466 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:50.141395    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b8xw\" (UniqueName: \"kubernetes.io/projected/0dcb92f5-5de5-4c91-87cc-27764bea9683-kube-api-access-9b8xw\") pod \"praxis-route-6a327a-kserve-5cb7997949-4vzjq\" (UID: \"0dcb92f5-5de5-4c91-87cc-27764bea9683\") " pod="ai-tenant-e2e-praxis-a-6a327a/praxis-route-6a327a-kserve-5cb7997949-4vzjq"
Oct 30 18:58:50.152753 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:50.142177    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0dcb92f5-5de5-4c91-87cc-27764bea9683-tls-certs\") pod \"praxis-route-6a327a-kserve-5cb7997949-4vzjq\" (UID: \"0dcb92f5-5de5-4c91-87cc-27764bea9683\") " pod="ai-tenant-e2e-praxis-a-6a327a/praxis-route-6a327a-kserve-5cb7997949-4vzjq"
Oct 30 18:58:50.269669 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:50.269577    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-6a327a/praxis-route-6a327a-kserve-5cb7997949-4vzjq"
Oct 30 18:58:50.572742 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:50.572710    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-6a327a/praxis-route-6a327a-kserve-5cb7997949-4vzjq"]
Oct 30 18:58:50.573408 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:58:50.573382    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0dcb92f5_5de5_4c91_87cc_27764bea9683.slice/crio-a200cee14afdcb421767e20d4a6b78ca738c155d91c0ffe77c6b9e33cbc14a5e WatchSource:0}: Error finding container a200cee14afdcb421767e20d4a6b78ca738c155d91c0ffe77c6b9e33cbc14a5e: Status 404 returned error can't find the container with id a200cee14afdcb421767e20d4a6b78ca738c155d91c0ffe77c6b9e33cbc14a5e
Oct 30 18:58:50.804405 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:50.804365    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-6a327a/praxis-route-6a327a-kserve-5cb7997949-4vzjq" event={"ID":"0dcb92f5-5de5-4c91-87cc-27764bea9683","Type":"ContainerStarted","Data":"435d9aad2e4f57d0efe0cd6430c36f20c19b46aaf3151783674d1a6fa269b77f"}
Oct 30 18:58:50.804602 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:50.804422    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-6a327a/praxis-route-6a327a-kserve-5cb7997949-4vzjq" event={"ID":"0dcb92f5-5de5-4c91-87cc-27764bea9683","Type":"ContainerStarted","Data":"a200cee14afdcb421767e20d4a6b78ca738c155d91c0ffe77c6b9e33cbc14a5e"}
Oct 30 18:58:50.825615 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:50.825478    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-praxis-a-6a327a/praxis-route-6a327a-kserve-5cb7997949-4vzjq" podStartSLOduration=1.825462562 podStartE2EDuration="1.825462562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:58:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:58:50.821714116 +0000 UTC m=+1353.447384103" watchObservedRunningTime="2026-10-30 18:58:50.825462562 +0000 UTC m=+1353.451132526"
Oct 30 18:58:51.269969 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:51.269934    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-6a327a/praxis-route-6a327a-kserve-5cb7997949-4vzjq"
Oct 30 18:58:52.271920 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:52.271885    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-vd5tv"]
Oct 30 18:58:52.272818 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:52.272791    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf3d8ea6-d7eb-48ae-ab8a-9a07611fc245" containerName="istio-proxy"
Oct 30 18:58:52.276469 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:52.276453    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-vd5tv"
Oct 30 18:58:52.286481 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:52.286456    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-fe0612-openshift-default-dockercfg-d4m8b\""
Oct 30 18:58:52.288304 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:52.288279    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-vd5tv"]
Oct 30 18:58:52.436148 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:52.436106    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32211654-2ca8-4305-a4fc-fe06a98bcbd3-istio-data\") pod \"e2e-ipp-b-fe0612-openshift-default-884c77ff7-vd5tv\" (UID: \"32211654-2ca8-4305-a4fc-fe06a98bcbd3\") " pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-vd5tv"
Oct 30 18:58:52.436148 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:52.436151    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32211654-2ca8-4305-a4fc-fe06a98bcbd3-workload-socket\") pod \"e2e-ipp-b-fe0612-openshift-default-884c77ff7-vd5tv\" (UID: \"32211654-2ca8-4305-a4fc-fe06a98bcbd3\") " pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-vd5tv"
Oct 30 18:58:52.436389 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:52.436284    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32211654-2ca8-4305-a4fc-fe06a98bcbd3-istio-envoy\") pod \"e2e-ipp-b-fe0612-openshift-default-884c77ff7-vd5tv\" (UID: \"32211654-2ca8-4305-a4fc-fe06a98bcbd3\") " pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-vd5tv"
Oct 30 18:58:52.436389 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:52.436340    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32211654-2ca8-4305-a4fc-fe06a98bcbd3-workload-certs\") pod \"e2e-ipp-b-fe0612-openshift-default-884c77ff7-vd5tv\" (UID: \"32211654-2ca8-4305-a4fc-fe06a98bcbd3\") " pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-vd5tv"
Oct 30 18:58:52.436477 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:52.436416    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32211654-2ca8-4305-a4fc-fe06a98bcbd3-istiod-ca-cert\") pod \"e2e-ipp-b-fe0612-openshift-default-884c77ff7-vd5tv\" (UID: \"32211654-2ca8-4305-a4fc-fe06a98bcbd3\") " pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-vd5tv"
Oct 30 18:58:52.436594 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:52.436573    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32211654-2ca8-4305-a4fc-fe06a98bcbd3-istio-token\") pod \"e2e-ipp-b-fe0612-openshift-default-884c77ff7-vd5tv\" (UID: \"32211654-2ca8-4305-a4fc-fe06a98bcbd3\") " pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-vd5tv"
Oct 30 18:58:52.436660 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:52.436638    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32211654-2ca8-4305-a4fc-fe06a98bcbd3-credential-socket\") pod \"e2e-ipp-b-fe0612-openshift-default-884c77ff7-vd5tv\" (UID: \"32211654-2ca8-4305-a4fc-fe06a98bcbd3\") " pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-vd5tv"
Oct 30 18:58:52.436793 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:52.436775    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32211654-2ca8-4305-a4fc-fe06a98bcbd3-istio-podinfo\") pod \"e2e-ipp-b-fe0612-openshift-default-884c77ff7-vd5tv\" (UID: \"32211654-2ca8-4305-a4fc-fe06a98bcbd3\") " pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-vd5tv"
Oct 30 18:58:52.436880 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:52.436865    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpbqb\" (UniqueName: \"kubernetes.io/projected/32211654-2ca8-4305-a4fc-fe06a98bcbd3-kube-api-access-jpbqb\") pod \"e2e-ipp-b-fe0612-openshift-default-884c77ff7-vd5tv\" (UID: \"32211654-2ca8-4305-a4fc-fe06a98bcbd3\") " pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-vd5tv"
Oct 30 18:58:52.538489 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:52.538401    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32211654-2ca8-4305-a4fc-fe06a98bcbd3-istio-token\") pod \"e2e-ipp-b-fe0612-openshift-default-884c77ff7-vd5tv\" (UID: \"32211654-2ca8-4305-a4fc-fe06a98bcbd3\") " pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-vd5tv"
Oct 30 18:58:52.538489 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:52.538444    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32211654-2ca8-4305-a4fc-fe06a98bcbd3-credential-socket\") pod \"e2e-ipp-b-fe0612-openshift-default-884c77ff7-vd5tv\" (UID: \"32211654-2ca8-4305-a4fc-fe06a98bcbd3\") " pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-vd5tv"
Oct 30 18:58:52.538729 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:52.538493    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32211654-2ca8-4305-a4fc-fe06a98bcbd3-istio-podinfo\") pod \"e2e-ipp-b-fe0612-openshift-default-884c77ff7-vd5tv\" (UID: \"32211654-2ca8-4305-a4fc-fe06a98bcbd3\") " pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-vd5tv"
Oct 30 18:58:52.538729 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:52.538553    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpbqb\" (UniqueName: \"kubernetes.io/projected/32211654-2ca8-4305-a4fc-fe06a98bcbd3-kube-api-access-jpbqb\") pod \"e2e-ipp-b-fe0612-openshift-default-884c77ff7-vd5tv\" (UID: \"32211654-2ca8-4305-a4fc-fe06a98bcbd3\") " pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-vd5tv"
Oct 30 18:58:52.538729 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:52.538601    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32211654-2ca8-4305-a4fc-fe06a98bcbd3-istio-data\") pod \"e2e-ipp-b-fe0612-openshift-default-884c77ff7-vd5tv\" (UID: \"32211654-2ca8-4305-a4fc-fe06a98bcbd3\") " pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-vd5tv"
Oct 30 18:58:52.538729 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:52.538628    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32211654-2ca8-4305-a4fc-fe06a98bcbd3-workload-socket\") pod \"e2e-ipp-b-fe0612-openshift-default-884c77ff7-vd5tv\" (UID: \"32211654-2ca8-4305-a4fc-fe06a98bcbd3\") " pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-vd5tv"
Oct 30 18:58:52.538729 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:52.538726    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32211654-2ca8-4305-a4fc-fe06a98bcbd3-istio-envoy\") pod \"e2e-ipp-b-fe0612-openshift-default-884c77ff7-vd5tv\" (UID: \"32211654-2ca8-4305-a4fc-fe06a98bcbd3\") " pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-vd5tv"
Oct 30 18:58:52.539031 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:52.538777    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32211654-2ca8-4305-a4fc-fe06a98bcbd3-workload-certs\") pod \"e2e-ipp-b-fe0612-openshift-default-884c77ff7-vd5tv\" (UID: \"32211654-2ca8-4305-a4fc-fe06a98bcbd3\") " pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-vd5tv"
Oct 30 18:58:52.539031 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:52.538831    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32211654-2ca8-4305-a4fc-fe06a98bcbd3-istiod-ca-cert\") pod \"e2e-ipp-b-fe0612-openshift-default-884c77ff7-vd5tv\" (UID: \"32211654-2ca8-4305-a4fc-fe06a98bcbd3\") " pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-vd5tv"
Oct 30 18:58:52.539031 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:52.538944    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32211654-2ca8-4305-a4fc-fe06a98bcbd3-credential-socket\") pod \"e2e-ipp-b-fe0612-openshift-default-884c77ff7-vd5tv\" (UID: \"32211654-2ca8-4305-a4fc-fe06a98bcbd3\") " pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-vd5tv"
Oct 30 18:58:52.539031 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:52.538960    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32211654-2ca8-4305-a4fc-fe06a98bcbd3-istio-data\") pod \"e2e-ipp-b-fe0612-openshift-default-884c77ff7-vd5tv\" (UID: \"32211654-2ca8-4305-a4fc-fe06a98bcbd3\") " pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-vd5tv"
Oct 30 18:58:52.539202 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:52.539089    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32211654-2ca8-4305-a4fc-fe06a98bcbd3-workload-socket\") pod \"e2e-ipp-b-fe0612-openshift-default-884c77ff7-vd5tv\" (UID: \"32211654-2ca8-4305-a4fc-fe06a98bcbd3\") " pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-vd5tv"
Oct 30 18:58:52.539202 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:52.539113    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32211654-2ca8-4305-a4fc-fe06a98bcbd3-workload-certs\") pod \"e2e-ipp-b-fe0612-openshift-default-884c77ff7-vd5tv\" (UID: \"32211654-2ca8-4305-a4fc-fe06a98bcbd3\") " pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-vd5tv"
Oct 30 18:58:52.539486 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:52.539468    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32211654-2ca8-4305-a4fc-fe06a98bcbd3-istiod-ca-cert\") pod \"e2e-ipp-b-fe0612-openshift-default-884c77ff7-vd5tv\" (UID: \"32211654-2ca8-4305-a4fc-fe06a98bcbd3\") " pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-vd5tv"
Oct 30 18:58:52.541258 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:52.541234    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32211654-2ca8-4305-a4fc-fe06a98bcbd3-istio-envoy\") pod \"e2e-ipp-b-fe0612-openshift-default-884c77ff7-vd5tv\" (UID: \"32211654-2ca8-4305-a4fc-fe06a98bcbd3\") " pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-vd5tv"
Oct 30 18:58:52.541389 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:52.541367    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32211654-2ca8-4305-a4fc-fe06a98bcbd3-istio-podinfo\") pod \"e2e-ipp-b-fe0612-openshift-default-884c77ff7-vd5tv\" (UID: \"32211654-2ca8-4305-a4fc-fe06a98bcbd3\") " pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-vd5tv"
Oct 30 18:58:52.547105 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:52.547080    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32211654-2ca8-4305-a4fc-fe06a98bcbd3-istio-token\") pod \"e2e-ipp-b-fe0612-openshift-default-884c77ff7-vd5tv\" (UID: \"32211654-2ca8-4305-a4fc-fe06a98bcbd3\") " pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-vd5tv"
Oct 30 18:58:52.547220 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:52.547123    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpbqb\" (UniqueName: \"kubernetes.io/projected/32211654-2ca8-4305-a4fc-fe06a98bcbd3-kube-api-access-jpbqb\") pod \"e2e-ipp-b-fe0612-openshift-default-884c77ff7-vd5tv\" (UID: \"32211654-2ca8-4305-a4fc-fe06a98bcbd3\") " pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-vd5tv"
Oct 30 18:58:52.596060 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:52.596022    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-vd5tv"
Oct 30 18:58:53.056661 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:53.056590    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-vd5tv"]
Oct 30 18:58:53.109904 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:53.109848    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:58:53.110174 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:53.110152    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:58:53.110316 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:53.110298    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:58:53.830287 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:53.830245    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-vd5tv" event={"ID":"32211654-2ca8-4305-a4fc-fe06a98bcbd3","Type":"ContainerStarted","Data":"62ef678071b23fd005f0e988bc4df7947f643b5fd1044ca172e36a591bafe1b1"}
Oct 30 18:58:53.830738 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:53.830311    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-vd5tv" event={"ID":"32211654-2ca8-4305-a4fc-fe06a98bcbd3","Type":"ContainerStarted","Data":"6cbb1ba01814984f11394f721ed04979011e8db210856ba35fab55d934acd305"}
Oct 30 18:58:53.850815 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:53.850763    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-vd5tv" podStartSLOduration=1.850749726 podStartE2EDuration="1.850749726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:58:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:58:53.84940283 +0000 UTC m=+1356.475072796" watchObservedRunningTime="2026-10-30 18:58:53.850749726 +0000 UTC m=+1356.476419734"
Oct 30 18:58:54.598897 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:54.597059    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-vd5tv"
Oct 30 18:58:54.600848 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:54.600805    4525 patch_prober.go:29] interesting pod/e2e-ipp-b-fe0612-openshift-default-884c77ff7-vd5tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Oct 30 18:58:54.600988 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:54.600893    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-vd5tv" podUID="32211654-2ca8-4305-a4fc-fe06a98bcbd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Oct 30 18:58:55.015189 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:55.014832    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589"]
Oct 30 18:58:55.022748 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:55.022657    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589"
Oct 30 18:58:55.025615 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:55.025590    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589"]
Oct 30 18:58:55.039762 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:55.038768    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-3d1aa8-openshift-default-dockercfg-7xv94\""
Oct 30 18:58:55.163907 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:55.163861    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af747a06-3fa4-4c7f-a5a3-7e33a6d68746-istiod-ca-cert\") pod \"e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589\" (UID: \"af747a06-3fa4-4c7f-a5a3-7e33a6d68746\") " pod="openshift-ingress/e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589"
Oct 30 18:58:55.193704 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:55.164275    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af747a06-3fa4-4c7f-a5a3-7e33a6d68746-workload-socket\") pod \"e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589\" (UID: \"af747a06-3fa4-4c7f-a5a3-7e33a6d68746\") " pod="openshift-ingress/e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589"
Oct 30 18:58:55.207037 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:55.201860    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af747a06-3fa4-4c7f-a5a3-7e33a6d68746-workload-certs\") pod \"e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589\" (UID: \"af747a06-3fa4-4c7f-a5a3-7e33a6d68746\") " pod="openshift-ingress/e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589"
Oct 30 18:58:55.207037 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:55.201923    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af747a06-3fa4-4c7f-a5a3-7e33a6d68746-istio-data\") pod \"e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589\" (UID: \"af747a06-3fa4-4c7f-a5a3-7e33a6d68746\") " pod="openshift-ingress/e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589"
Oct 30 18:58:55.207037 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:55.201983    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af747a06-3fa4-4c7f-a5a3-7e33a6d68746-istio-podinfo\") pod \"e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589\" (UID: \"af747a06-3fa4-4c7f-a5a3-7e33a6d68746\") " pod="openshift-ingress/e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589"
Oct 30 18:58:55.207037 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:55.202055    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af747a06-3fa4-4c7f-a5a3-7e33a6d68746-istio-token\") pod \"e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589\" (UID: \"af747a06-3fa4-4c7f-a5a3-7e33a6d68746\") " pod="openshift-ingress/e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589"
Oct 30 18:58:55.207037 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:55.202230    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af747a06-3fa4-4c7f-a5a3-7e33a6d68746-credential-socket\") pod \"e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589\" (UID: \"af747a06-3fa4-4c7f-a5a3-7e33a6d68746\") " pod="openshift-ingress/e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589"
Oct 30 18:58:55.207037 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:55.202274    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af747a06-3fa4-4c7f-a5a3-7e33a6d68746-istio-envoy\") pod \"e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589\" (UID: \"af747a06-3fa4-4c7f-a5a3-7e33a6d68746\") " pod="openshift-ingress/e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589"
Oct 30 18:58:55.207037 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:55.202404    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7b9ln\" (UniqueName: \"kubernetes.io/projected/af747a06-3fa4-4c7f-a5a3-7e33a6d68746-kube-api-access-7b9ln\") pod \"e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589\" (UID: \"af747a06-3fa4-4c7f-a5a3-7e33a6d68746\") " pod="openshift-ingress/e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589"
Oct 30 18:58:55.312579 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:55.312487    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af747a06-3fa4-4c7f-a5a3-7e33a6d68746-credential-socket\") pod \"e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589\" (UID: \"af747a06-3fa4-4c7f-a5a3-7e33a6d68746\") " pod="openshift-ingress/e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589"
Oct 30 18:58:55.313255 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:55.313233    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af747a06-3fa4-4c7f-a5a3-7e33a6d68746-istio-envoy\") pod \"e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589\" (UID: \"af747a06-3fa4-4c7f-a5a3-7e33a6d68746\") " pod="openshift-ingress/e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589"
Oct 30 18:58:55.313511 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:55.313494    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b9ln\" (UniqueName: \"kubernetes.io/projected/af747a06-3fa4-4c7f-a5a3-7e33a6d68746-kube-api-access-7b9ln\") pod \"e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589\" (UID: \"af747a06-3fa4-4c7f-a5a3-7e33a6d68746\") " pod="openshift-ingress/e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589"
Oct 30 18:58:55.313643 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:55.313628    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af747a06-3fa4-4c7f-a5a3-7e33a6d68746-istiod-ca-cert\") pod \"e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589\" (UID: \"af747a06-3fa4-4c7f-a5a3-7e33a6d68746\") " pod="openshift-ingress/e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589"
Oct 30 18:58:55.313776 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:55.313761    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af747a06-3fa4-4c7f-a5a3-7e33a6d68746-workload-socket\") pod \"e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589\" (UID: \"af747a06-3fa4-4c7f-a5a3-7e33a6d68746\") " pod="openshift-ingress/e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589"
Oct 30 18:58:55.313965 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:55.313951    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af747a06-3fa4-4c7f-a5a3-7e33a6d68746-workload-certs\") pod \"e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589\" (UID: \"af747a06-3fa4-4c7f-a5a3-7e33a6d68746\") " pod="openshift-ingress/e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589"
Oct 30 18:58:55.314071 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:55.314058    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af747a06-3fa4-4c7f-a5a3-7e33a6d68746-istio-data\") pod \"e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589\" (UID: \"af747a06-3fa4-4c7f-a5a3-7e33a6d68746\") " pod="openshift-ingress/e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589"
Oct 30 18:58:55.314216 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:55.314202    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af747a06-3fa4-4c7f-a5a3-7e33a6d68746-istio-podinfo\") pod \"e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589\" (UID: \"af747a06-3fa4-4c7f-a5a3-7e33a6d68746\") " pod="openshift-ingress/e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589"
Oct 30 18:58:55.315335 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:55.315314    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af747a06-3fa4-4c7f-a5a3-7e33a6d68746-istio-token\") pod \"e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589\" (UID: \"af747a06-3fa4-4c7f-a5a3-7e33a6d68746\") " pod="openshift-ingress/e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589"
Oct 30 18:58:55.315608 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:55.313155    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af747a06-3fa4-4c7f-a5a3-7e33a6d68746-credential-socket\") pod \"e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589\" (UID: \"af747a06-3fa4-4c7f-a5a3-7e33a6d68746\") " pod="openshift-ingress/e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589"
Oct 30 18:58:55.316775 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:55.315946    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af747a06-3fa4-4c7f-a5a3-7e33a6d68746-workload-socket\") pod \"e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589\" (UID: \"af747a06-3fa4-4c7f-a5a3-7e33a6d68746\") " pod="openshift-ingress/e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589"
Oct 30 18:58:55.316910 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:55.316165    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af747a06-3fa4-4c7f-a5a3-7e33a6d68746-workload-certs\") pod \"e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589\" (UID: \"af747a06-3fa4-4c7f-a5a3-7e33a6d68746\") " pod="openshift-ingress/e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589"
Oct 30 18:58:55.317021 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:55.316376    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af747a06-3fa4-4c7f-a5a3-7e33a6d68746-istio-data\") pod \"e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589\" (UID: \"af747a06-3fa4-4c7f-a5a3-7e33a6d68746\") " pod="openshift-ingress/e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589"
Oct 30 18:58:55.352707 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:55.314961    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af747a06-3fa4-4c7f-a5a3-7e33a6d68746-istiod-ca-cert\") pod \"e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589\" (UID: \"af747a06-3fa4-4c7f-a5a3-7e33a6d68746\") " pod="openshift-ingress/e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589"
Oct 30 18:58:55.410472 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:55.397988    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af747a06-3fa4-4c7f-a5a3-7e33a6d68746-istio-podinfo\") pod \"e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589\" (UID: \"af747a06-3fa4-4c7f-a5a3-7e33a6d68746\") " pod="openshift-ingress/e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589"
Oct 30 18:58:55.515128 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:55.515075    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af747a06-3fa4-4c7f-a5a3-7e33a6d68746-istio-token\") pod \"e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589\" (UID: \"af747a06-3fa4-4c7f-a5a3-7e33a6d68746\") " pod="openshift-ingress/e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589"
Oct 30 18:58:55.531255 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:55.531212    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af747a06-3fa4-4c7f-a5a3-7e33a6d68746-istio-envoy\") pod \"e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589\" (UID: \"af747a06-3fa4-4c7f-a5a3-7e33a6d68746\") " pod="openshift-ingress/e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589"
Oct 30 18:58:55.532238 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:55.531597    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b9ln\" (UniqueName: \"kubernetes.io/projected/af747a06-3fa4-4c7f-a5a3-7e33a6d68746-kube-api-access-7b9ln\") pod \"e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589\" (UID: \"af747a06-3fa4-4c7f-a5a3-7e33a6d68746\") " pod="openshift-ingress/e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589"
Oct 30 18:58:55.603604 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:55.603515    4525 patch_prober.go:29] interesting pod/e2e-ipp-b-fe0612-openshift-default-884c77ff7-vd5tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Oct 30 18:58:55.603943 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:55.603911    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-vd5tv" podUID="32211654-2ca8-4305-a4fc-fe06a98bcbd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Oct 30 18:58:55.652385 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:55.651965    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589"
Oct 30 18:58:56.011571 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:58:56.011540    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf747a06_3fa4_4c7f_a5a3_7e33a6d68746.slice/crio-d1036e7fdeab2598026bbc44cb98f4554467cb1f6040b17e4eb979991a7d330b WatchSource:0}: Error finding container d1036e7fdeab2598026bbc44cb98f4554467cb1f6040b17e4eb979991a7d330b: Status 404 returned error can't find the container with id d1036e7fdeab2598026bbc44cb98f4554467cb1f6040b17e4eb979991a7d330b
Oct 30 18:58:56.012138 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:56.012099    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589"]
Oct 30 18:58:56.014981 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:56.014950    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:58:56.015067 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:56.015016    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:58:56.015067 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:56.015048    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:58:56.604793 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:56.604712    4525 patch_prober.go:29] interesting pod/e2e-ipp-b-fe0612-openshift-default-884c77ff7-vd5tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Oct 30 18:58:56.605299 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:56.604802    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-vd5tv" podUID="32211654-2ca8-4305-a4fc-fe06a98bcbd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Oct 30 18:58:56.860125 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:56.860036    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589" event={"ID":"af747a06-3fa4-4c7f-a5a3-7e33a6d68746","Type":"ContainerStarted","Data":"4cf262d2a7d076ae846899bee2c71fb71c18aab346e48f1914fb85cc3249adf0"}
Oct 30 18:58:56.860125 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:56.860103    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589" event={"ID":"af747a06-3fa4-4c7f-a5a3-7e33a6d68746","Type":"ContainerStarted","Data":"d1036e7fdeab2598026bbc44cb98f4554467cb1f6040b17e4eb979991a7d330b"}
Oct 30 18:58:56.905705 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:56.905612    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589" podStartSLOduration=2.905590873 podStartE2EDuration="2.905590873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:58:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:58:56.890015171 +0000 UTC m=+1359.515685132" watchObservedRunningTime="2026-10-30 18:58:56.905590873 +0000 UTC m=+1359.531260838"
Oct 30 18:58:57.600862 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:57.600825    4525 patch_prober.go:29] interesting pod/e2e-ipp-b-fe0612-openshift-default-884c77ff7-vd5tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Oct 30 18:58:57.601144 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:57.601112    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-vd5tv" podUID="32211654-2ca8-4305-a4fc-fe06a98bcbd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Oct 30 18:58:57.653504 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:57.653464    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589"
Oct 30 18:58:57.659264 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:57.659236    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589"
Oct 30 18:58:57.868036 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:57.868009    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589"
Oct 30 18:58:57.868979 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:57.868961    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589"
Oct 30 18:58:58.601200 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:58.601156    4525 patch_prober.go:29] interesting pod/e2e-ipp-b-fe0612-openshift-default-884c77ff7-vd5tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Oct 30 18:58:58.601414 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:58.601227    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-vd5tv" podUID="32211654-2ca8-4305-a4fc-fe06a98bcbd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Oct 30 18:58:59.597491 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:59.597455    4525 patch_prober.go:29] interesting pod/e2e-ipp-b-fe0612-openshift-default-884c77ff7-vd5tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Oct 30 18:58:59.597899 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:58:59.597510    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-vd5tv" podUID="32211654-2ca8-4305-a4fc-fe06a98bcbd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Oct 30 18:59:00.270621 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:00.270586    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-praxis-a-6a327a/praxis-route-6a327a-kserve-5cb7997949-4vzjq"
Oct 30 18:59:00.282882 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:00.282852    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-praxis-a-6a327a/praxis-route-6a327a-kserve-5cb7997949-4vzjq"
Oct 30 18:59:00.602474 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:00.602402    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-vd5tv"
Oct 30 18:59:00.603359 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:00.603334    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-vd5tv"
Oct 30 18:59:00.604332 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:00.604312    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-vd5tv"
Oct 30 18:59:00.919878 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:00.919842    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-6a327a/praxis-route-6a327a-kserve-5cb7997949-4vzjq"
Oct 30 18:59:09.527802 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:09.527761    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-54bc2b-2cnsr"]
Oct 30 18:59:09.531944 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:09.531924    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-54bc2b-2cnsr"
Oct 30 18:59:09.535633 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:09.535606    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:59:09.535633 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:09.535623    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94hcn\""
Oct 30 18:59:09.577004 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:09.576975    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-54bc2b-2cnsr"]
Oct 30 18:59:09.604769 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:09.604726    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8xgh\" (UniqueName: \"kubernetes.io/projected/60b998b2-6444-4b83-aa28-2562a66e54fe-kube-api-access-c8xgh\") pod \"maas-api-revoke-keys-e2e-ipp-a-54bc2b-2cnsr\" (UID: \"60b998b2-6444-4b83-aa28-2562a66e54fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-54bc2b-2cnsr"
Oct 30 18:59:09.604924 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:09.604843    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60b998b2-6444-4b83-aa28-2562a66e54fe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-54bc2b-2cnsr\" (UID: \"60b998b2-6444-4b83-aa28-2562a66e54fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-54bc2b-2cnsr"
Oct 30 18:59:09.604987 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:09.604968    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60b998b2-6444-4b83-aa28-2562a66e54fe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-54bc2b-2cnsr\" (UID: \"60b998b2-6444-4b83-aa28-2562a66e54fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-54bc2b-2cnsr"
Oct 30 18:59:09.706369 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:09.706332    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60b998b2-6444-4b83-aa28-2562a66e54fe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-54bc2b-2cnsr\" (UID: \"60b998b2-6444-4b83-aa28-2562a66e54fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-54bc2b-2cnsr"
Oct 30 18:59:09.706546 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:09.706426    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60b998b2-6444-4b83-aa28-2562a66e54fe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-54bc2b-2cnsr\" (UID: \"60b998b2-6444-4b83-aa28-2562a66e54fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-54bc2b-2cnsr"
Oct 30 18:59:09.706707 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:09.706659    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8xgh\" (UniqueName: \"kubernetes.io/projected/60b998b2-6444-4b83-aa28-2562a66e54fe-kube-api-access-c8xgh\") pod \"maas-api-revoke-keys-e2e-ipp-a-54bc2b-2cnsr\" (UID: \"60b998b2-6444-4b83-aa28-2562a66e54fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-54bc2b-2cnsr"
Oct 30 18:59:09.706993 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:09.706964    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60b998b2-6444-4b83-aa28-2562a66e54fe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-54bc2b-2cnsr\" (UID: \"60b998b2-6444-4b83-aa28-2562a66e54fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-54bc2b-2cnsr"
Oct 30 18:59:09.708987 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:09.708962    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60b998b2-6444-4b83-aa28-2562a66e54fe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-54bc2b-2cnsr\" (UID: \"60b998b2-6444-4b83-aa28-2562a66e54fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-54bc2b-2cnsr"
Oct 30 18:59:09.714525 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:09.714504    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8xgh\" (UniqueName: \"kubernetes.io/projected/60b998b2-6444-4b83-aa28-2562a66e54fe-kube-api-access-c8xgh\") pod \"maas-api-revoke-keys-e2e-ipp-a-54bc2b-2cnsr\" (UID: \"60b998b2-6444-4b83-aa28-2562a66e54fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-54bc2b-2cnsr"
Oct 30 18:59:09.876309 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:09.876278    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-54bc2b-2cnsr"
Oct 30 18:59:10.165210 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:59:10.165177    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60b998b2_6444_4b83_aa28_2562a66e54fe.slice/crio-c0c24b9b1f63d1cb338941c67f45081b416aa8991138bf7b037ebba3cababc68 WatchSource:0}: Error finding container c0c24b9b1f63d1cb338941c67f45081b416aa8991138bf7b037ebba3cababc68: Status 404 returned error can't find the container with id c0c24b9b1f63d1cb338941c67f45081b416aa8991138bf7b037ebba3cababc68
Oct 30 18:59:10.165693 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:10.165654    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-54bc2b-2cnsr"]
Oct 30 18:59:11.079729 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:11.079653    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-54bc2b-2cnsr" event={"ID":"60b998b2-6444-4b83-aa28-2562a66e54fe","Type":"ContainerStarted","Data":"db9d1e2f9b14d90a3167a321cea61b8c5f0b7426b85d0cc2a1676977230054f3"}
Oct 30 18:59:11.080142 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:11.079785    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-54bc2b-2cnsr" event={"ID":"60b998b2-6444-4b83-aa28-2562a66e54fe","Type":"ContainerStarted","Data":"c0c24b9b1f63d1cb338941c67f45081b416aa8991138bf7b037ebba3cababc68"}
Oct 30 18:59:11.100215 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:11.100166    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-54bc2b-2cnsr" podStartSLOduration=2.100150955 podStartE2EDuration="2.100150955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:59:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:59:11.095511913 +0000 UTC m=+1373.721181878" watchObservedRunningTime="2026-10-30 18:59:11.100150955 +0000 UTC m=+1373.725820963"
Oct 30 18:59:12.107425 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:12.107387    4525 generic.go:356] "Generic (PLEG): container finished" podID="60b998b2-6444-4b83-aa28-2562a66e54fe" containerID="db9d1e2f9b14d90a3167a321cea61b8c5f0b7426b85d0cc2a1676977230054f3" exitCode=0
Oct 30 18:59:12.108008 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:12.107495    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-54bc2b-2cnsr" event={"ID":"60b998b2-6444-4b83-aa28-2562a66e54fe","Type":"ContainerDied","Data":"db9d1e2f9b14d90a3167a321cea61b8c5f0b7426b85d0cc2a1676977230054f3"}
Oct 30 18:59:13.279386 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:13.279357    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-54bc2b-2cnsr"
Oct 30 18:59:13.408147 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:13.408042    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60b998b2-6444-4b83-aa28-2562a66e54fe-kube-api-access-c8xgh\" (UniqueName: \"kubernetes.io/projected/60b998b2-6444-4b83-aa28-2562a66e54fe-kube-api-access-c8xgh\") pod \"60b998b2-6444-4b83-aa28-2562a66e54fe\" (UID: \"60b998b2-6444-4b83-aa28-2562a66e54fe\") "
Oct 30 18:59:13.408147 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:13.408090    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60b998b2-6444-4b83-aa28-2562a66e54fe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60b998b2-6444-4b83-aa28-2562a66e54fe-maas-api-xks-ca\") pod \"60b998b2-6444-4b83-aa28-2562a66e54fe\" (UID: \"60b998b2-6444-4b83-aa28-2562a66e54fe\") "
Oct 30 18:59:13.408398 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:13.408191    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60b998b2-6444-4b83-aa28-2562a66e54fe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60b998b2-6444-4b83-aa28-2562a66e54fe-maas-api-openshift-ca\") pod \"60b998b2-6444-4b83-aa28-2562a66e54fe\" (UID: \"60b998b2-6444-4b83-aa28-2562a66e54fe\") "
Oct 30 18:59:13.408593 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:13.408554    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60b998b2-6444-4b83-aa28-2562a66e54fe-maas-api-openshift-ca" pod "60b998b2-6444-4b83-aa28-2562a66e54fe" (UID: "60b998b2-6444-4b83-aa28-2562a66e54fe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:59:13.408914 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:13.408891    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60b998b2-6444-4b83-aa28-2562a66e54fe-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:13.410460 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:13.410434    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60b998b2-6444-4b83-aa28-2562a66e54fe-maas-api-xks-ca" pod "60b998b2-6444-4b83-aa28-2562a66e54fe" (UID: "60b998b2-6444-4b83-aa28-2562a66e54fe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:59:13.410549 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:13.410533    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60b998b2-6444-4b83-aa28-2562a66e54fe-kube-api-access-c8xgh" pod "60b998b2-6444-4b83-aa28-2562a66e54fe" (UID: "60b998b2-6444-4b83-aa28-2562a66e54fe"). InnerVolumeSpecName "kube-api-access-c8xgh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:59:13.510026 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:13.509988    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8xgh\" (UniqueName: \"kubernetes.io/projected/60b998b2-6444-4b83-aa28-2562a66e54fe-kube-api-access-c8xgh\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:13.510026 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:13.510022    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60b998b2-6444-4b83-aa28-2562a66e54fe-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:14.126130 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:14.126095    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-54bc2b-2cnsr"
Oct 30 18:59:14.126327 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:14.126092    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-54bc2b-2cnsr" event={"ID":"60b998b2-6444-4b83-aa28-2562a66e54fe","Type":"ContainerDied","Data":"c0c24b9b1f63d1cb338941c67f45081b416aa8991138bf7b037ebba3cababc68"}
Oct 30 18:59:14.126327 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:14.126258    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c0c24b9b1f63d1cb338941c67f45081b416aa8991138bf7b037ebba3cababc68"
Oct 30 18:59:17.400273 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:59:17.400211    4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d49l" containerName="extractor"
Oct 30 18:59:18.851139 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:18.851106    4525 scope.go:122] "RemoveContainer" containerID="ad786fa88c15e58611c19ae6a84a8025eedc972c43c4fd1bea5a05a92424661f"
Oct 30 18:59:18.862385 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:18.862361    4525 scope.go:122] "RemoveContainer" containerID="589f258e5d1defd5a31215f460fe4236574cc45040b86ffcec0d76eecb6d80b7"
Oct 30 18:59:18.872748 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:18.872717    4525 scope.go:122] "RemoveContainer" containerID="c7f26c3ac62fc10ff8dc1446b88fd4f7e3d948f7c66e5b9c2f5c72d11b2cb720"
Oct 30 18:59:18.883152 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:18.883125    4525 scope.go:122] "RemoveContainer" containerID="53518b6cc290c11fb287151812d2643097bcda6b78886d17ffd452caa2028415"
Oct 30 18:59:18.894114 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:18.894078    4525 scope.go:122] "RemoveContainer" containerID="6bbdef8a51e66f03eceac71630ca234f33ced4c0e637e1c41f5177ab01e5ba93"
Oct 30 18:59:18.904497 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:18.904476    4525 scope.go:122] "RemoveContainer" containerID="e965e38955c96400cbd2ab3af5ee488d9ca95d19f24e3b9bc585667d04b75b07"
Oct 30 18:59:18.926119 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:18.926093    4525 scope.go:122] "RemoveContainer" containerID="b68366aeb4e054341958f0f94410c05cdd75bef33f89982d47e30f3bfdf350a3"
Oct 30 18:59:18.936716 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:18.936694    4525 scope.go:122] "RemoveContainer" containerID="dcd1416cb55f1679e4324af778c237c54e4f8702c3de01040cc2da7ea7497767"
Oct 30 18:59:18.947117 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:18.947097    4525 scope.go:122] "RemoveContainer" containerID="c1a5039fa72dd937ad5cdb1b2bd7c27991e5e5255a1cbf9ef0d5c73fa75e527f"
Oct 30 18:59:20.394314 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:20.394243    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-54bc2b-2cnsr"]
Oct 30 18:59:20.435839 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:20.435799    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-54bc2b-2cnsr"]
Oct 30 18:59:21.014077 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:21.014051    4525 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-93574a-ccbc886cb-kq9wb_33de14c3-409a-4c51-b4c4-74a849da5b63/payload-processing/0.log"
Oct 30 18:59:21.546383 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:21.546348    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60b998b2-6444-4b83-aa28-2562a66e54fe" path="/var/lib/kubelet/pods/60b998b2-6444-4b83-aa28-2562a66e54fe/volumes"
Oct 30 18:59:22.042429 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:22.042386    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70bb645b-wfhgh"]
Oct 30 18:59:22.048089 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:22.048061    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70bb645b-wfhgh"]
Oct 30 18:59:22.581353 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:22.581316    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-93574a-f2vcw"]
Oct 30 18:59:22.581961 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:22.581944    4525 cpu_manager.go:414] "containerMap: removing container" podUID="60b998b2-6444-4b83-aa28-2562a66e54fe" containerName="revoke-keys"
Oct 30 18:59:22.582009 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:22.581963    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60b998b2-6444-4b83-aa28-2562a66e54fe" containerName="revoke-keys"
Oct 30 18:59:22.582099 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:22.582081    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="60b998b2-6444-4b83-aa28-2562a66e54fe" containerName="revoke-keys"
Oct 30 18:59:22.587820 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:22.587795    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-93574a-f2vcw"
Oct 30 18:59:22.592480 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:22.592454    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:59:22.593137 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:22.593112    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-93574a-f2vcw"]
Oct 30 18:59:22.593484 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:22.593461    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94hcn\""
Oct 30 18:59:22.656548 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:22.656517    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19602b5c-399d-4a0b-809e-5c46dd300f1d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-93574a-f2vcw\" (UID: \"19602b5c-399d-4a0b-809e-5c46dd300f1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-93574a-f2vcw"
Oct 30 18:59:22.656774 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:22.656657    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19602b5c-399d-4a0b-809e-5c46dd300f1d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-93574a-f2vcw\" (UID: \"19602b5c-399d-4a0b-809e-5c46dd300f1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-93574a-f2vcw"
Oct 30 18:59:22.656846 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:22.656833    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2stxp\" (UniqueName: \"kubernetes.io/projected/19602b5c-399d-4a0b-809e-5c46dd300f1d-kube-api-access-2stxp\") pod \"maas-api-revoke-keys-e2e-ipp-a-93574a-f2vcw\" (UID: \"19602b5c-399d-4a0b-809e-5c46dd300f1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-93574a-f2vcw"
Oct 30 18:59:22.765144 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:22.765095    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19602b5c-399d-4a0b-809e-5c46dd300f1d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-93574a-f2vcw\" (UID: \"19602b5c-399d-4a0b-809e-5c46dd300f1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-93574a-f2vcw"
Oct 30 18:59:22.765360 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:22.765274    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2stxp\" (UniqueName: \"kubernetes.io/projected/19602b5c-399d-4a0b-809e-5c46dd300f1d-kube-api-access-2stxp\") pod \"maas-api-revoke-keys-e2e-ipp-a-93574a-f2vcw\" (UID: \"19602b5c-399d-4a0b-809e-5c46dd300f1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-93574a-f2vcw"
Oct 30 18:59:22.765484 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:22.765465    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19602b5c-399d-4a0b-809e-5c46dd300f1d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-93574a-f2vcw\" (UID: \"19602b5c-399d-4a0b-809e-5c46dd300f1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-93574a-f2vcw"
Oct 30 18:59:22.766396 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:22.766367    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19602b5c-399d-4a0b-809e-5c46dd300f1d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-93574a-f2vcw\" (UID: \"19602b5c-399d-4a0b-809e-5c46dd300f1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-93574a-f2vcw"
Oct 30 18:59:22.769880 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:22.769851    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19602b5c-399d-4a0b-809e-5c46dd300f1d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-93574a-f2vcw\" (UID: \"19602b5c-399d-4a0b-809e-5c46dd300f1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-93574a-f2vcw"
Oct 30 18:59:22.776946 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:22.776921    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2stxp\" (UniqueName: \"kubernetes.io/projected/19602b5c-399d-4a0b-809e-5c46dd300f1d-kube-api-access-2stxp\") pod \"maas-api-revoke-keys-e2e-ipp-a-93574a-f2vcw\" (UID: \"19602b5c-399d-4a0b-809e-5c46dd300f1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-93574a-f2vcw"
Oct 30 18:59:22.907516 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:22.907481    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-93574a-f2vcw"
Oct 30 18:59:23.194624 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:59:23.194595    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19602b5c_399d_4a0b_809e_5c46dd300f1d.slice/crio-06e106c2284bceee07548a9a7719f7785c3da029246a4a7721500776804a1957 WatchSource:0}: Error finding container 06e106c2284bceee07548a9a7719f7785c3da029246a4a7721500776804a1957: Status 404 returned error can't find the container with id 06e106c2284bceee07548a9a7719f7785c3da029246a4a7721500776804a1957
Oct 30 18:59:23.196782 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:23.196757    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-93574a-f2vcw"]
Oct 30 18:59:23.312299 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:23.312260    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-93574a-f2vcw" event={"ID":"19602b5c-399d-4a0b-809e-5c46dd300f1d","Type":"ContainerStarted","Data":"06e106c2284bceee07548a9a7719f7785c3da029246a4a7721500776804a1957"}
Oct 30 18:59:23.546812 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:23.546776    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c983c51-cb90-47f2-8bc2-e2b7806df111" path="/var/lib/kubelet/pods/0c983c51-cb90-47f2-8bc2-e2b7806df111/volumes"
Oct 30 18:59:23.603790 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:23.603756    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-655193-zvxsm"]
Oct 30 18:59:23.624599 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:23.621161    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-655193-zvxsm"
Oct 30 18:59:23.677519 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:23.655440    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-655193-zvxsm"]
Oct 30 18:59:23.679833 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:23.679786    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4dc3e22-d7a2-45a9-891c-64857fa3188f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-655193-zvxsm\" (UID: \"d4dc3e22-d7a2-45a9-891c-64857fa3188f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-655193-zvxsm"
Oct 30 18:59:23.680064 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:23.680040    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4dc3e22-d7a2-45a9-891c-64857fa3188f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-655193-zvxsm\" (UID: \"d4dc3e22-d7a2-45a9-891c-64857fa3188f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-655193-zvxsm"
Oct 30 18:59:23.680662 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:23.680637    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpzkw\" (UniqueName: \"kubernetes.io/projected/d4dc3e22-d7a2-45a9-891c-64857fa3188f-kube-api-access-lpzkw\") pod \"maas-api-revoke-keys-e2e-praxis-a-655193-zvxsm\" (UID: \"d4dc3e22-d7a2-45a9-891c-64857fa3188f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-655193-zvxsm"
Oct 30 18:59:23.782195 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:23.782105    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpzkw\" (UniqueName: \"kubernetes.io/projected/d4dc3e22-d7a2-45a9-891c-64857fa3188f-kube-api-access-lpzkw\") pod \"maas-api-revoke-keys-e2e-praxis-a-655193-zvxsm\" (UID: \"d4dc3e22-d7a2-45a9-891c-64857fa3188f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-655193-zvxsm"
Oct 30 18:59:23.782195 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:23.782197    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4dc3e22-d7a2-45a9-891c-64857fa3188f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-655193-zvxsm\" (UID: \"d4dc3e22-d7a2-45a9-891c-64857fa3188f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-655193-zvxsm"
Oct 30 18:59:23.782429 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:23.782222    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4dc3e22-d7a2-45a9-891c-64857fa3188f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-655193-zvxsm\" (UID: \"d4dc3e22-d7a2-45a9-891c-64857fa3188f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-655193-zvxsm"
Oct 30 18:59:23.782991 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:23.782966    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4dc3e22-d7a2-45a9-891c-64857fa3188f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-655193-zvxsm\" (UID: \"d4dc3e22-d7a2-45a9-891c-64857fa3188f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-655193-zvxsm"
Oct 30 18:59:23.784876 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:23.784853    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4dc3e22-d7a2-45a9-891c-64857fa3188f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-655193-zvxsm\" (UID: \"d4dc3e22-d7a2-45a9-891c-64857fa3188f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-655193-zvxsm"
Oct 30 18:59:23.791544 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:23.791522    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpzkw\" (UniqueName: \"kubernetes.io/projected/d4dc3e22-d7a2-45a9-891c-64857fa3188f-kube-api-access-lpzkw\") pod \"maas-api-revoke-keys-e2e-praxis-a-655193-zvxsm\" (UID: \"d4dc3e22-d7a2-45a9-891c-64857fa3188f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-655193-zvxsm"
Oct 30 18:59:23.974889 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:23.974854    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-655193-zvxsm"
Oct 30 18:59:24.260848 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:59:24.260816    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4dc3e22_d7a2_45a9_891c_64857fa3188f.slice/crio-66ef6ecb6009b1539fd0032a09cb7af99a69937b02bdf3730a652471f0bb3338 WatchSource:0}: Error finding container 66ef6ecb6009b1539fd0032a09cb7af99a69937b02bdf3730a652471f0bb3338: Status 404 returned error can't find the container with id 66ef6ecb6009b1539fd0032a09cb7af99a69937b02bdf3730a652471f0bb3338
Oct 30 18:59:24.261256 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:24.261234    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-655193-zvxsm"]
Oct 30 18:59:24.321189 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:24.321149    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-655193-zvxsm" event={"ID":"d4dc3e22-d7a2-45a9-891c-64857fa3188f","Type":"ContainerStarted","Data":"66ef6ecb6009b1539fd0032a09cb7af99a69937b02bdf3730a652471f0bb3338"}
Oct 30 18:59:24.323145 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:24.323116    4525 generic.go:356] "Generic (PLEG): container finished" podID="19602b5c-399d-4a0b-809e-5c46dd300f1d" containerID="2620ccbfb4d896807bc5ced5bafea6d411ac5b4fd080007a7cd8cafdbea9cbdf" exitCode=0
Oct 30 18:59:24.323305 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:24.323162    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-93574a-f2vcw" event={"ID":"19602b5c-399d-4a0b-809e-5c46dd300f1d","Type":"ContainerDied","Data":"2620ccbfb4d896807bc5ced5bafea6d411ac5b4fd080007a7cd8cafdbea9cbdf"}
Oct 30 18:59:25.375000 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:25.374964    4525 generic.go:356] "Generic (PLEG): container finished" podID="d4dc3e22-d7a2-45a9-891c-64857fa3188f" containerID="c4106c7b4c2dde0616ce890ec4819ceb3a4da192cc5f73e947839648e05ff770" exitCode=0
Oct 30 18:59:25.375421 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:25.375044    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-655193-zvxsm" event={"ID":"d4dc3e22-d7a2-45a9-891c-64857fa3188f","Type":"ContainerDied","Data":"c4106c7b4c2dde0616ce890ec4819ceb3a4da192cc5f73e947839648e05ff770"}
Oct 30 18:59:25.567448 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:25.567411    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-93574a-f2vcw"
Oct 30 18:59:25.715504 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:25.715412    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19602b5c-399d-4a0b-809e-5c46dd300f1d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19602b5c-399d-4a0b-809e-5c46dd300f1d-maas-api-openshift-ca\") pod \"19602b5c-399d-4a0b-809e-5c46dd300f1d\" (UID: \"19602b5c-399d-4a0b-809e-5c46dd300f1d\") "
Oct 30 18:59:25.715504 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:25.715465    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19602b5c-399d-4a0b-809e-5c46dd300f1d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19602b5c-399d-4a0b-809e-5c46dd300f1d-maas-api-xks-ca\") pod \"19602b5c-399d-4a0b-809e-5c46dd300f1d\" (UID: \"19602b5c-399d-4a0b-809e-5c46dd300f1d\") "
Oct 30 18:59:25.715787 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:25.715654    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19602b5c-399d-4a0b-809e-5c46dd300f1d-kube-api-access-2stxp\" (UniqueName: \"kubernetes.io/projected/19602b5c-399d-4a0b-809e-5c46dd300f1d-kube-api-access-2stxp\") pod \"19602b5c-399d-4a0b-809e-5c46dd300f1d\" (UID: \"19602b5c-399d-4a0b-809e-5c46dd300f1d\") "
Oct 30 18:59:25.715907 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:25.715873    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19602b5c-399d-4a0b-809e-5c46dd300f1d-maas-api-openshift-ca" pod "19602b5c-399d-4a0b-809e-5c46dd300f1d" (UID: "19602b5c-399d-4a0b-809e-5c46dd300f1d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:59:25.716395 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:25.716374    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19602b5c-399d-4a0b-809e-5c46dd300f1d-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:25.717851 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:25.717831    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19602b5c-399d-4a0b-809e-5c46dd300f1d-maas-api-xks-ca" pod "19602b5c-399d-4a0b-809e-5c46dd300f1d" (UID: "19602b5c-399d-4a0b-809e-5c46dd300f1d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:59:25.717953 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:25.717933    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19602b5c-399d-4a0b-809e-5c46dd300f1d-kube-api-access-2stxp" pod "19602b5c-399d-4a0b-809e-5c46dd300f1d" (UID: "19602b5c-399d-4a0b-809e-5c46dd300f1d"). InnerVolumeSpecName "kube-api-access-2stxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:59:25.817930 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:25.817882    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19602b5c-399d-4a0b-809e-5c46dd300f1d-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:25.817930 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:25.817923    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2stxp\" (UniqueName: \"kubernetes.io/projected/19602b5c-399d-4a0b-809e-5c46dd300f1d-kube-api-access-2stxp\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:26.387500 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:26.387459    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-93574a-f2vcw"
Oct 30 18:59:26.391724 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:26.387459    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-93574a-f2vcw" event={"ID":"19602b5c-399d-4a0b-809e-5c46dd300f1d","Type":"ContainerDied","Data":"06e106c2284bceee07548a9a7719f7785c3da029246a4a7721500776804a1957"}
Oct 30 18:59:26.391894 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:26.391798    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06e106c2284bceee07548a9a7719f7785c3da029246a4a7721500776804a1957"
Oct 30 18:59:26.557785 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:26.557758    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-655193-zvxsm"
Oct 30 18:59:26.629389 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:26.629351    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4dc3e22-d7a2-45a9-891c-64857fa3188f-kube-api-access-lpzkw\" (UniqueName: \"kubernetes.io/projected/d4dc3e22-d7a2-45a9-891c-64857fa3188f-kube-api-access-lpzkw\") pod \"d4dc3e22-d7a2-45a9-891c-64857fa3188f\" (UID: \"d4dc3e22-d7a2-45a9-891c-64857fa3188f\") "
Oct 30 18:59:26.629593 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:26.629495    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4dc3e22-d7a2-45a9-891c-64857fa3188f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4dc3e22-d7a2-45a9-891c-64857fa3188f-maas-api-openshift-ca\") pod \"d4dc3e22-d7a2-45a9-891c-64857fa3188f\" (UID: \"d4dc3e22-d7a2-45a9-891c-64857fa3188f\") "
Oct 30 18:59:26.629927 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:26.629897    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4dc3e22-d7a2-45a9-891c-64857fa3188f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4dc3e22-d7a2-45a9-891c-64857fa3188f-maas-api-xks-ca\") pod \"d4dc3e22-d7a2-45a9-891c-64857fa3188f\" (UID: \"d4dc3e22-d7a2-45a9-891c-64857fa3188f\") "
Oct 30 18:59:26.630068 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:26.630036    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4dc3e22-d7a2-45a9-891c-64857fa3188f-maas-api-openshift-ca" pod "d4dc3e22-d7a2-45a9-891c-64857fa3188f" (UID: "d4dc3e22-d7a2-45a9-891c-64857fa3188f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:59:26.630646 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:26.630618    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4dc3e22-d7a2-45a9-891c-64857fa3188f-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:26.632825 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:26.632798    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4dc3e22-d7a2-45a9-891c-64857fa3188f-maas-api-xks-ca" pod "d4dc3e22-d7a2-45a9-891c-64857fa3188f" (UID: "d4dc3e22-d7a2-45a9-891c-64857fa3188f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:59:26.632950 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:26.632822    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4dc3e22-d7a2-45a9-891c-64857fa3188f-kube-api-access-lpzkw" pod "d4dc3e22-d7a2-45a9-891c-64857fa3188f" (UID: "d4dc3e22-d7a2-45a9-891c-64857fa3188f"). InnerVolumeSpecName "kube-api-access-lpzkw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:59:26.732042 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:26.732008    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpzkw\" (UniqueName: \"kubernetes.io/projected/d4dc3e22-d7a2-45a9-891c-64857fa3188f-kube-api-access-lpzkw\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:26.732042 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:26.732038    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4dc3e22-d7a2-45a9-891c-64857fa3188f-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:27.396011 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:27.395984    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-655193-zvxsm"
Oct 30 18:59:27.396518 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:27.396005    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-655193-zvxsm" event={"ID":"d4dc3e22-d7a2-45a9-891c-64857fa3188f","Type":"ContainerDied","Data":"66ef6ecb6009b1539fd0032a09cb7af99a69937b02bdf3730a652471f0bb3338"}
Oct 30 18:59:27.396518 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:27.396060    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66ef6ecb6009b1539fd0032a09cb7af99a69937b02bdf3730a652471f0bb3338"
Oct 30 18:59:27.926701 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:27.926654    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-54bc2b-openshift-default-bf84f77dc-jqfxp"]
Oct 30 18:59:27.927273 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:27.927231    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-54bc2b-openshift-default-bf84f77dc-jqfxp" podUID="9aca1683-2018-4248-8c22-bd7a46096df3" containerName="istio-proxy" containerID="cri-o://075e22b9ed3d43cbd03af910a58084d3da7f6b091dbe8634deb7064d2e4a0d87" gracePeriod=30
Oct 30 18:59:28.388406 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:28.388378    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fe0612-xwq59"]
Oct 30 18:59:28.389108 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:28.389089    4525 cpu_manager.go:414] "containerMap: removing container" podUID="19602b5c-399d-4a0b-809e-5c46dd300f1d" containerName="revoke-keys"
Oct 30 18:59:28.389108 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:28.389109    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19602b5c-399d-4a0b-809e-5c46dd300f1d" containerName="revoke-keys"
Oct 30 18:59:28.389303 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:28.389165    4525 cpu_manager.go:414] "containerMap: removing container" podUID="d4dc3e22-d7a2-45a9-891c-64857fa3188f" containerName="revoke-keys"
Oct 30 18:59:28.389303 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:28.389174    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4dc3e22-d7a2-45a9-891c-64857fa3188f" containerName="revoke-keys"
Oct 30 18:59:28.389422 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:28.389319    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="19602b5c-399d-4a0b-809e-5c46dd300f1d" containerName="revoke-keys"
Oct 30 18:59:28.389422 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:28.389343    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4dc3e22-d7a2-45a9-891c-64857fa3188f" containerName="revoke-keys"
Oct 30 18:59:28.393171 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:28.393130    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fe0612-xwq59"
Oct 30 18:59:28.397069 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:28.397030    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94hcn\""
Oct 30 18:59:28.397628 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:28.397603    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fh5f\" (UniqueName: \"kubernetes.io/projected/c22b56a5-f8a3-418c-b6b3-20d5e05b1f8b-kube-api-access-2fh5f\") pod \"maas-api-revoke-keys-e2e-ipp-b-fe0612-xwq59\" (UID: \"c22b56a5-f8a3-418c-b6b3-20d5e05b1f8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fe0612-xwq59"
Oct 30 18:59:28.397782 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:28.397661    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:59:28.397782 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:28.397749    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c22b56a5-f8a3-418c-b6b3-20d5e05b1f8b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-fe0612-xwq59\" (UID: \"c22b56a5-f8a3-418c-b6b3-20d5e05b1f8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fe0612-xwq59"
Oct 30 18:59:28.397915 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:28.397891    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c22b56a5-f8a3-418c-b6b3-20d5e05b1f8b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-fe0612-xwq59\" (UID: \"c22b56a5-f8a3-418c-b6b3-20d5e05b1f8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fe0612-xwq59"
Oct 30 18:59:28.399531 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:28.399508    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fe0612-xwq59"]
Oct 30 18:59:28.498910 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:28.498874    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fh5f\" (UniqueName: \"kubernetes.io/projected/c22b56a5-f8a3-418c-b6b3-20d5e05b1f8b-kube-api-access-2fh5f\") pod \"maas-api-revoke-keys-e2e-ipp-b-fe0612-xwq59\" (UID: \"c22b56a5-f8a3-418c-b6b3-20d5e05b1f8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fe0612-xwq59"
Oct 30 18:59:28.499121 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:28.498957    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c22b56a5-f8a3-418c-b6b3-20d5e05b1f8b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-fe0612-xwq59\" (UID: \"c22b56a5-f8a3-418c-b6b3-20d5e05b1f8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fe0612-xwq59"
Oct 30 18:59:28.499121 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:28.499015    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c22b56a5-f8a3-418c-b6b3-20d5e05b1f8b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-fe0612-xwq59\" (UID: \"c22b56a5-f8a3-418c-b6b3-20d5e05b1f8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fe0612-xwq59"
Oct 30 18:59:28.499614 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:28.499583    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c22b56a5-f8a3-418c-b6b3-20d5e05b1f8b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-fe0612-xwq59\" (UID: \"c22b56a5-f8a3-418c-b6b3-20d5e05b1f8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fe0612-xwq59"
Oct 30 18:59:28.501518 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:28.501498    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c22b56a5-f8a3-418c-b6b3-20d5e05b1f8b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-fe0612-xwq59\" (UID: \"c22b56a5-f8a3-418c-b6b3-20d5e05b1f8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fe0612-xwq59"
Oct 30 18:59:28.506260 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:28.506236    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fh5f\" (UniqueName: \"kubernetes.io/projected/c22b56a5-f8a3-418c-b6b3-20d5e05b1f8b-kube-api-access-2fh5f\") pod \"maas-api-revoke-keys-e2e-ipp-b-fe0612-xwq59\" (UID: \"c22b56a5-f8a3-418c-b6b3-20d5e05b1f8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fe0612-xwq59"
Oct 30 18:59:28.714397 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:28.714298    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fe0612-xwq59"
Oct 30 18:59:29.009159 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:29.008987    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fe0612-xwq59"]
Oct 30 18:59:29.419497 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:29.419455    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fe0612-xwq59" event={"ID":"c22b56a5-f8a3-418c-b6b3-20d5e05b1f8b","Type":"ContainerStarted","Data":"a0e5783c4a80866da15737955cb0ab2f55553649def9edeababbf22b46766061"}
Oct 30 18:59:29.420016 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:29.419514    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fe0612-xwq59" event={"ID":"c22b56a5-f8a3-418c-b6b3-20d5e05b1f8b","Type":"ContainerStarted","Data":"e663a2020c5d4c54cd11486e3bceb05266950295b86a0f6bd03e5b020fb2d7fb"}
Oct 30 18:59:29.441820 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:29.441759    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fe0612-xwq59" podStartSLOduration=1.441745631 podStartE2EDuration="1.441745631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:59:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:59:29.439778926 +0000 UTC m=+1392.065448890" watchObservedRunningTime="2026-10-30 18:59:29.441745631 +0000 UTC m=+1392.067415596"
Oct 30 18:59:30.429214 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:30.429172    4525 generic.go:356] "Generic (PLEG): container finished" podID="c22b56a5-f8a3-418c-b6b3-20d5e05b1f8b" containerID="a0e5783c4a80866da15737955cb0ab2f55553649def9edeababbf22b46766061" exitCode=0
Oct 30 18:59:30.429615 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:30.429263    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fe0612-xwq59" event={"ID":"c22b56a5-f8a3-418c-b6b3-20d5e05b1f8b","Type":"ContainerDied","Data":"a0e5783c4a80866da15737955cb0ab2f55553649def9edeababbf22b46766061"}
Oct 30 18:59:31.610561 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:31.610537    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fe0612-xwq59"
Oct 30 18:59:31.652626 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:31.652593    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c22b56a5-f8a3-418c-b6b3-20d5e05b1f8b-kube-api-access-2fh5f\" (UniqueName: \"kubernetes.io/projected/c22b56a5-f8a3-418c-b6b3-20d5e05b1f8b-kube-api-access-2fh5f\") pod \"c22b56a5-f8a3-418c-b6b3-20d5e05b1f8b\" (UID: \"c22b56a5-f8a3-418c-b6b3-20d5e05b1f8b\") "
Oct 30 18:59:31.652877 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:31.652764    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c22b56a5-f8a3-418c-b6b3-20d5e05b1f8b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c22b56a5-f8a3-418c-b6b3-20d5e05b1f8b-maas-api-openshift-ca\") pod \"c22b56a5-f8a3-418c-b6b3-20d5e05b1f8b\" (UID: \"c22b56a5-f8a3-418c-b6b3-20d5e05b1f8b\") "
Oct 30 18:59:31.652877 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:31.652812    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c22b56a5-f8a3-418c-b6b3-20d5e05b1f8b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c22b56a5-f8a3-418c-b6b3-20d5e05b1f8b-maas-api-xks-ca\") pod \"c22b56a5-f8a3-418c-b6b3-20d5e05b1f8b\" (UID: \"c22b56a5-f8a3-418c-b6b3-20d5e05b1f8b\") "
Oct 30 18:59:31.653210 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:31.653159    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c22b56a5-f8a3-418c-b6b3-20d5e05b1f8b-maas-api-openshift-ca" pod "c22b56a5-f8a3-418c-b6b3-20d5e05b1f8b" (UID: "c22b56a5-f8a3-418c-b6b3-20d5e05b1f8b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:59:31.653374 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:31.653351    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c22b56a5-f8a3-418c-b6b3-20d5e05b1f8b-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:31.658723 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:31.655490    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c22b56a5-f8a3-418c-b6b3-20d5e05b1f8b-maas-api-xks-ca" pod "c22b56a5-f8a3-418c-b6b3-20d5e05b1f8b" (UID: "c22b56a5-f8a3-418c-b6b3-20d5e05b1f8b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:59:31.658723 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:31.655695    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c22b56a5-f8a3-418c-b6b3-20d5e05b1f8b-kube-api-access-2fh5f" pod "c22b56a5-f8a3-418c-b6b3-20d5e05b1f8b" (UID: "c22b56a5-f8a3-418c-b6b3-20d5e05b1f8b"). InnerVolumeSpecName "kube-api-access-2fh5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:59:31.756767 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:31.756575    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c22b56a5-f8a3-418c-b6b3-20d5e05b1f8b-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:31.756767 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:31.756614    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fh5f\" (UniqueName: \"kubernetes.io/projected/c22b56a5-f8a3-418c-b6b3-20d5e05b1f8b-kube-api-access-2fh5f\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:32.446977 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:32.446949    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fe0612-xwq59"
Oct 30 18:59:32.447125 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:32.446944    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fe0612-xwq59" event={"ID":"c22b56a5-f8a3-418c-b6b3-20d5e05b1f8b","Type":"ContainerDied","Data":"e663a2020c5d4c54cd11486e3bceb05266950295b86a0f6bd03e5b020fb2d7fb"}
Oct 30 18:59:32.447125 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:32.447074    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e663a2020c5d4c54cd11486e3bceb05266950295b86a0f6bd03e5b020fb2d7fb"
Oct 30 18:59:33.460925 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:33.460890    4525 generic.go:356] "Generic (PLEG): container finished" podID="9aca1683-2018-4248-8c22-bd7a46096df3" containerID="075e22b9ed3d43cbd03af910a58084d3da7f6b091dbe8634deb7064d2e4a0d87" exitCode=0
Oct 30 18:59:33.461331 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:33.461188    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-54bc2b-openshift-default-bf84f77dc-jqfxp" event={"ID":"9aca1683-2018-4248-8c22-bd7a46096df3","Type":"ContainerDied","Data":"075e22b9ed3d43cbd03af910a58084d3da7f6b091dbe8634deb7064d2e4a0d87"}
Oct 30 18:59:33.461331 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:33.461259    4525 scope.go:122] "RemoveContainer" containerID="2173c9aa51f156466dcd6a03ba7bb121bb8e74f6380583fcb039eb2bcd650caf"
Oct 30 18:59:33.578058 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:33.578031    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-54bc2b-openshift-default-bf84f77dc-jqfxp"
Oct 30 18:59:33.591226 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:33.591198    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29e4bbf3-rptrl"]
Oct 30 18:59:33.591835 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:33.591818    4525 cpu_manager.go:414] "containerMap: removing container" podUID="9aca1683-2018-4248-8c22-bd7a46096df3" containerName="istio-proxy"
Oct 30 18:59:33.591906 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:33.591837    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9aca1683-2018-4248-8c22-bd7a46096df3" containerName="istio-proxy"
Oct 30 18:59:33.591906 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:33.591851    4525 cpu_manager.go:414] "containerMap: removing container" podUID="9aca1683-2018-4248-8c22-bd7a46096df3" containerName="istio-proxy"
Oct 30 18:59:33.591906 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:33.591857    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9aca1683-2018-4248-8c22-bd7a46096df3" containerName="istio-proxy"
Oct 30 18:59:33.591906 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:33.591878    4525 cpu_manager.go:414] "containerMap: removing container" podUID="c22b56a5-f8a3-418c-b6b3-20d5e05b1f8b" containerName="revoke-keys"
Oct 30 18:59:33.591906 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:33.591883    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c22b56a5-f8a3-418c-b6b3-20d5e05b1f8b" containerName="revoke-keys"
Oct 30 18:59:33.592056 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:33.591972    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="9aca1683-2018-4248-8c22-bd7a46096df3" containerName="istio-proxy"
Oct 30 18:59:33.592056 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:33.591981    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="9aca1683-2018-4248-8c22-bd7a46096df3" containerName="istio-proxy"
Oct 30 18:59:33.592056 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:33.591990    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="c22b56a5-f8a3-418c-b6b3-20d5e05b1f8b" containerName="revoke-keys"
Oct 30 18:59:33.595350 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:33.595332    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29e4bbf3-rptrl"
Oct 30 18:59:33.603220 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:33.603192    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94hcn\""
Oct 30 18:59:33.603401 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:33.603201    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:59:33.605964 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:33.605937    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29e4bbf3-rptrl"]
Oct 30 18:59:33.689442 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:33.689413    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9aca1683-2018-4248-8c22-bd7a46096df3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aca1683-2018-4248-8c22-bd7a46096df3-workload-socket\") pod \"9aca1683-2018-4248-8c22-bd7a46096df3\" (UID: \"9aca1683-2018-4248-8c22-bd7a46096df3\") "
Oct 30 18:59:33.689617 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:33.689458    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9aca1683-2018-4248-8c22-bd7a46096df3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aca1683-2018-4248-8c22-bd7a46096df3-credential-socket\") pod \"9aca1683-2018-4248-8c22-bd7a46096df3\" (UID: \"9aca1683-2018-4248-8c22-bd7a46096df3\") "
Oct 30 18:59:33.689617 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:33.689541    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9aca1683-2018-4248-8c22-bd7a46096df3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9aca1683-2018-4248-8c22-bd7a46096df3-istio-data\") pod \"9aca1683-2018-4248-8c22-bd7a46096df3\" (UID: \"9aca1683-2018-4248-8c22-bd7a46096df3\") "
Oct 30 18:59:33.689617 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:33.689595    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9aca1683-2018-4248-8c22-bd7a46096df3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9aca1683-2018-4248-8c22-bd7a46096df3-workload-certs\") pod \"9aca1683-2018-4248-8c22-bd7a46096df3\" (UID: \"9aca1683-2018-4248-8c22-bd7a46096df3\") "
Oct 30 18:59:33.689775 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:33.689671    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9aca1683-2018-4248-8c22-bd7a46096df3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9aca1683-2018-4248-8c22-bd7a46096df3-istiod-ca-cert\") pod \"9aca1683-2018-4248-8c22-bd7a46096df3\" (UID: \"9aca1683-2018-4248-8c22-bd7a46096df3\") "
Oct 30 18:59:33.689775 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:33.689745    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9aca1683-2018-4248-8c22-bd7a46096df3-kube-api-access-km8bm\" (UniqueName: \"kubernetes.io/projected/9aca1683-2018-4248-8c22-bd7a46096df3-kube-api-access-km8bm\") pod \"9aca1683-2018-4248-8c22-bd7a46096df3\" (UID: \"9aca1683-2018-4248-8c22-bd7a46096df3\") "
Oct 30 18:59:33.689775 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:33.689748    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9aca1683-2018-4248-8c22-bd7a46096df3-workload-socket" pod "9aca1683-2018-4248-8c22-bd7a46096df3" (UID: "9aca1683-2018-4248-8c22-bd7a46096df3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:59:33.689946 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:33.689825    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9aca1683-2018-4248-8c22-bd7a46096df3-istio-token\" (UniqueName: \"kubernetes.io/projected/9aca1683-2018-4248-8c22-bd7a46096df3-istio-token\") pod \"9aca1683-2018-4248-8c22-bd7a46096df3\" (UID: \"9aca1683-2018-4248-8c22-bd7a46096df3\") "
Oct 30 18:59:33.689946 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:33.689857    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9aca1683-2018-4248-8c22-bd7a46096df3-istio-data" pod "9aca1683-2018-4248-8c22-bd7a46096df3" (UID: "9aca1683-2018-4248-8c22-bd7a46096df3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:59:33.689946 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:33.689868    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9aca1683-2018-4248-8c22-bd7a46096df3-credential-socket" pod "9aca1683-2018-4248-8c22-bd7a46096df3" (UID: "9aca1683-2018-4248-8c22-bd7a46096df3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:59:33.689946 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:33.689895    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9aca1683-2018-4248-8c22-bd7a46096df3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9aca1683-2018-4248-8c22-bd7a46096df3-istio-podinfo\") pod \"9aca1683-2018-4248-8c22-bd7a46096df3\" (UID: \"9aca1683-2018-4248-8c22-bd7a46096df3\") "
Oct 30 18:59:33.689946 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:33.689917    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9aca1683-2018-4248-8c22-bd7a46096df3-workload-certs" pod "9aca1683-2018-4248-8c22-bd7a46096df3" (UID: "9aca1683-2018-4248-8c22-bd7a46096df3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:59:33.689946 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:33.689943    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9aca1683-2018-4248-8c22-bd7a46096df3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9aca1683-2018-4248-8c22-bd7a46096df3-istio-envoy\") pod \"9aca1683-2018-4248-8c22-bd7a46096df3\" (UID: \"9aca1683-2018-4248-8c22-bd7a46096df3\") "
Oct 30 18:59:33.690246 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:33.690145    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9aca1683-2018-4248-8c22-bd7a46096df3-istiod-ca-cert" pod "9aca1683-2018-4248-8c22-bd7a46096df3" (UID: "9aca1683-2018-4248-8c22-bd7a46096df3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:59:33.690359 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:33.690334    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b712abe-9c69-4d7f-b4d3-c172d6c0298a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29e4bbf3-rptrl\" (UID: \"3b712abe-9c69-4d7f-b4d3-c172d6c0298a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29e4bbf3-rptrl"
Oct 30 18:59:33.690585 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:33.690565    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b712abe-9c69-4d7f-b4d3-c172d6c0298a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29e4bbf3-rptrl\" (UID: \"3b712abe-9c69-4d7f-b4d3-c172d6c0298a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29e4bbf3-rptrl"
Oct 30 18:59:33.690726 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:33.690710    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wzqs\" (UniqueName: \"kubernetes.io/projected/3b712abe-9c69-4d7f-b4d3-c172d6c0298a-kube-api-access-6wzqs\") pod \"maas-api-revoke-sub-29e4bbf3-rptrl\" (UID: \"3b712abe-9c69-4d7f-b4d3-c172d6c0298a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29e4bbf3-rptrl"
Oct 30 18:59:33.690822 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:33.690801    4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9aca1683-2018-4248-8c22-bd7a46096df3-istiod-ca-cert\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:33.690958 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:33.690853    4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aca1683-2018-4248-8c22-bd7a46096df3-workload-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:33.690958 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:33.690870    4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aca1683-2018-4248-8c22-bd7a46096df3-credential-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:33.690958 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:33.690879    4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9aca1683-2018-4248-8c22-bd7a46096df3-istio-data\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:33.690958 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:33.690888    4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9aca1683-2018-4248-8c22-bd7a46096df3-workload-certs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:33.692641 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:33.692617    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9aca1683-2018-4248-8c22-bd7a46096df3-istio-envoy" pod "9aca1683-2018-4248-8c22-bd7a46096df3" (UID: "9aca1683-2018-4248-8c22-bd7a46096df3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:59:33.692896 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:33.692864    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9aca1683-2018-4248-8c22-bd7a46096df3-istio-podinfo" pod "9aca1683-2018-4248-8c22-bd7a46096df3" (UID: "9aca1683-2018-4248-8c22-bd7a46096df3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 18:59:33.693060 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:33.693042    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9aca1683-2018-4248-8c22-bd7a46096df3-kube-api-access-km8bm" pod "9aca1683-2018-4248-8c22-bd7a46096df3" (UID: "9aca1683-2018-4248-8c22-bd7a46096df3"). InnerVolumeSpecName "kube-api-access-km8bm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:59:33.693178 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:33.693155    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9aca1683-2018-4248-8c22-bd7a46096df3-istio-token" pod "9aca1683-2018-4248-8c22-bd7a46096df3" (UID: "9aca1683-2018-4248-8c22-bd7a46096df3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:59:33.792716 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:33.792641    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wzqs\" (UniqueName: \"kubernetes.io/projected/3b712abe-9c69-4d7f-b4d3-c172d6c0298a-kube-api-access-6wzqs\") pod \"maas-api-revoke-sub-29e4bbf3-rptrl\" (UID: \"3b712abe-9c69-4d7f-b4d3-c172d6c0298a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29e4bbf3-rptrl"
Oct 30 18:59:33.792919 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:33.792901    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b712abe-9c69-4d7f-b4d3-c172d6c0298a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29e4bbf3-rptrl\" (UID: \"3b712abe-9c69-4d7f-b4d3-c172d6c0298a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29e4bbf3-rptrl"
Oct 30 18:59:33.793089 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:33.793072    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b712abe-9c69-4d7f-b4d3-c172d6c0298a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29e4bbf3-rptrl\" (UID: \"3b712abe-9c69-4d7f-b4d3-c172d6c0298a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29e4bbf3-rptrl"
Oct 30 18:59:33.793217 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:33.793201    4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9aca1683-2018-4248-8c22-bd7a46096df3-istio-token\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:33.793290 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:33.793223    4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9aca1683-2018-4248-8c22-bd7a46096df3-istio-podinfo\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:33.793290 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:33.793238    4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9aca1683-2018-4248-8c22-bd7a46096df3-istio-envoy\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:33.793290 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:33.793254    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-km8bm\" (UniqueName: \"kubernetes.io/projected/9aca1683-2018-4248-8c22-bd7a46096df3-kube-api-access-km8bm\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:33.793706 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:33.793639    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b712abe-9c69-4d7f-b4d3-c172d6c0298a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29e4bbf3-rptrl\" (UID: \"3b712abe-9c69-4d7f-b4d3-c172d6c0298a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29e4bbf3-rptrl"
Oct 30 18:59:33.795875 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:33.795847    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b712abe-9c69-4d7f-b4d3-c172d6c0298a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29e4bbf3-rptrl\" (UID: \"3b712abe-9c69-4d7f-b4d3-c172d6c0298a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29e4bbf3-rptrl"
Oct 30 18:59:33.806101 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:33.806065    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wzqs\" (UniqueName: \"kubernetes.io/projected/3b712abe-9c69-4d7f-b4d3-c172d6c0298a-kube-api-access-6wzqs\") pod \"maas-api-revoke-sub-29e4bbf3-rptrl\" (UID: \"3b712abe-9c69-4d7f-b4d3-c172d6c0298a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29e4bbf3-rptrl"
Oct 30 18:59:33.924135 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:33.924100    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29e4bbf3-rptrl"
Oct 30 18:59:34.298673 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:34.298644    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29e4bbf3-rptrl"]
Oct 30 18:59:34.470166 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:34.470137    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-54bc2b-openshift-default-bf84f77dc-jqfxp"
Oct 30 18:59:34.470642 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:34.470167    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-54bc2b-openshift-default-bf84f77dc-jqfxp" event={"ID":"9aca1683-2018-4248-8c22-bd7a46096df3","Type":"ContainerDied","Data":"ea2e05ea2c06ad2ee09ef69230f2b32fefa54ee0bda7335cc38a6d8d2a374d79"}
Oct 30 18:59:34.470642 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:34.470233    4525 scope.go:122] "RemoveContainer" containerID="075e22b9ed3d43cbd03af910a58084d3da7f6b091dbe8634deb7064d2e4a0d87"
Oct 30 18:59:34.471868 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:34.471840    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29e4bbf3-rptrl" event={"ID":"3b712abe-9c69-4d7f-b4d3-c172d6c0298a","Type":"ContainerStarted","Data":"4e983b7aa177778d178222d2ca0cb84ea8b1cfb4b1ac1b3d250775bafe2e3c5a"}
Oct 30 18:59:34.506095 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:34.506064    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-54bc2b-openshift-default-bf84f77dc-jqfxp"]
Oct 30 18:59:34.509920 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:34.509900    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-54bc2b-openshift-default-bf84f77dc-jqfxp"]
Oct 30 18:59:35.481826 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:35.481793    4525 generic.go:356] "Generic (PLEG): container finished" podID="3b712abe-9c69-4d7f-b4d3-c172d6c0298a" containerID="7e1d41d6537275ae19e87c1526a5e553481a55d49fd89ea5a5448b5c86a180f9" exitCode=0
Oct 30 18:59:35.482228 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:35.481888    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29e4bbf3-rptrl" event={"ID":"3b712abe-9c69-4d7f-b4d3-c172d6c0298a","Type":"ContainerDied","Data":"7e1d41d6537275ae19e87c1526a5e553481a55d49fd89ea5a5448b5c86a180f9"}
Oct 30 18:59:35.546557 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:35.546524    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9aca1683-2018-4248-8c22-bd7a46096df3" path="/var/lib/kubelet/pods/9aca1683-2018-4248-8c22-bd7a46096df3/volumes"
Oct 30 18:59:36.647703 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:36.630763    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-655193-zvxsm"]
Oct 30 18:59:36.823749 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:36.823211    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-655193-zvxsm"]
Oct 30 18:59:36.834362 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:36.834330    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f778f5-6v7zn"]
Oct 30 18:59:36.863420 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:36.863357    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f778f5-6v7zn"]
Oct 30 18:59:36.863760 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:36.863494    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f778f5-6v7zn"
Oct 30 18:59:36.976607 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:36.976578    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29e4bbf3-rptrl"
Oct 30 18:59:37.023659 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:37.023621    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecd67510-3824-4b69-bb7f-795a89600d5f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-f778f5-6v7zn\" (UID: \"ecd67510-3824-4b69-bb7f-795a89600d5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f778f5-6v7zn"
Oct 30 18:59:37.023896 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:37.023874    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvhwk\" (UniqueName: \"kubernetes.io/projected/ecd67510-3824-4b69-bb7f-795a89600d5f-kube-api-access-xvhwk\") pod \"maas-api-revoke-keys-e2e-praxis-a-f778f5-6v7zn\" (UID: \"ecd67510-3824-4b69-bb7f-795a89600d5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f778f5-6v7zn"
Oct 30 18:59:37.024180 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:37.024145    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecd67510-3824-4b69-bb7f-795a89600d5f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-f778f5-6v7zn\" (UID: \"ecd67510-3824-4b69-bb7f-795a89600d5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f778f5-6v7zn"
Oct 30 18:59:37.125346 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:37.125313    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b712abe-9c69-4d7f-b4d3-c172d6c0298a-kube-api-access-6wzqs\" (UniqueName: \"kubernetes.io/projected/3b712abe-9c69-4d7f-b4d3-c172d6c0298a-kube-api-access-6wzqs\") pod \"3b712abe-9c69-4d7f-b4d3-c172d6c0298a\" (UID: \"3b712abe-9c69-4d7f-b4d3-c172d6c0298a\") "
Oct 30 18:59:37.125554 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:37.125384    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b712abe-9c69-4d7f-b4d3-c172d6c0298a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b712abe-9c69-4d7f-b4d3-c172d6c0298a-maas-api-xks-ca\") pod \"3b712abe-9c69-4d7f-b4d3-c172d6c0298a\" (UID: \"3b712abe-9c69-4d7f-b4d3-c172d6c0298a\") "
Oct 30 18:59:37.125648 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:37.125635    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b712abe-9c69-4d7f-b4d3-c172d6c0298a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b712abe-9c69-4d7f-b4d3-c172d6c0298a-maas-api-openshift-ca\") pod \"3b712abe-9c69-4d7f-b4d3-c172d6c0298a\" (UID: \"3b712abe-9c69-4d7f-b4d3-c172d6c0298a\") "
Oct 30 18:59:37.125821 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:37.125804    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecd67510-3824-4b69-bb7f-795a89600d5f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-f778f5-6v7zn\" (UID: \"ecd67510-3824-4b69-bb7f-795a89600d5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f778f5-6v7zn"
Oct 30 18:59:37.125982 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:37.125963    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvhwk\" (UniqueName: \"kubernetes.io/projected/ecd67510-3824-4b69-bb7f-795a89600d5f-kube-api-access-xvhwk\") pod \"maas-api-revoke-keys-e2e-praxis-a-f778f5-6v7zn\" (UID: \"ecd67510-3824-4b69-bb7f-795a89600d5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f778f5-6v7zn"
Oct 30 18:59:37.126045 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:37.126007    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b712abe-9c69-4d7f-b4d3-c172d6c0298a-maas-api-openshift-ca" pod "3b712abe-9c69-4d7f-b4d3-c172d6c0298a" (UID: "3b712abe-9c69-4d7f-b4d3-c172d6c0298a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:59:37.126485 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:37.126451    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecd67510-3824-4b69-bb7f-795a89600d5f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-f778f5-6v7zn\" (UID: \"ecd67510-3824-4b69-bb7f-795a89600d5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f778f5-6v7zn"
Oct 30 18:59:37.126617 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:37.126594    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b712abe-9c69-4d7f-b4d3-c172d6c0298a-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:37.127117 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:37.127096    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecd67510-3824-4b69-bb7f-795a89600d5f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-f778f5-6v7zn\" (UID: \"ecd67510-3824-4b69-bb7f-795a89600d5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f778f5-6v7zn"
Oct 30 18:59:37.127804 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:37.127780    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b712abe-9c69-4d7f-b4d3-c172d6c0298a-maas-api-xks-ca" pod "3b712abe-9c69-4d7f-b4d3-c172d6c0298a" (UID: "3b712abe-9c69-4d7f-b4d3-c172d6c0298a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:59:37.127911 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:37.127849    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b712abe-9c69-4d7f-b4d3-c172d6c0298a-kube-api-access-6wzqs" pod "3b712abe-9c69-4d7f-b4d3-c172d6c0298a" (UID: "3b712abe-9c69-4d7f-b4d3-c172d6c0298a"). InnerVolumeSpecName "kube-api-access-6wzqs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:59:37.128225 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:37.128206    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecd67510-3824-4b69-bb7f-795a89600d5f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-f778f5-6v7zn\" (UID: \"ecd67510-3824-4b69-bb7f-795a89600d5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f778f5-6v7zn"
Oct 30 18:59:37.134480 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:37.134456    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvhwk\" (UniqueName: \"kubernetes.io/projected/ecd67510-3824-4b69-bb7f-795a89600d5f-kube-api-access-xvhwk\") pod \"maas-api-revoke-keys-e2e-praxis-a-f778f5-6v7zn\" (UID: \"ecd67510-3824-4b69-bb7f-795a89600d5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f778f5-6v7zn"
Oct 30 18:59:37.207320 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:37.207287    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f778f5-6v7zn"
Oct 30 18:59:37.227996 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:37.227967    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wzqs\" (UniqueName: \"kubernetes.io/projected/3b712abe-9c69-4d7f-b4d3-c172d6c0298a-kube-api-access-6wzqs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:37.228100 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:37.228003    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b712abe-9c69-4d7f-b4d3-c172d6c0298a-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:37.494629 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:37.494599    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f778f5-6v7zn"]
Oct 30 18:59:37.494786 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:59:37.494657    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecd67510_3824_4b69_bb7f_795a89600d5f.slice/crio-47daf6ffe09040a2e1bd4b0f8ebd4e06da2a50bd1d9846458ac87231d6a26fc9 WatchSource:0}: Error finding container 47daf6ffe09040a2e1bd4b0f8ebd4e06da2a50bd1d9846458ac87231d6a26fc9: Status 404 returned error can't find the container with id 47daf6ffe09040a2e1bd4b0f8ebd4e06da2a50bd1d9846458ac87231d6a26fc9
Oct 30 18:59:37.500215 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:37.500194    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29e4bbf3-rptrl"
Oct 30 18:59:37.500301 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:37.500245    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29e4bbf3-rptrl" event={"ID":"3b712abe-9c69-4d7f-b4d3-c172d6c0298a","Type":"ContainerDied","Data":"4e983b7aa177778d178222d2ca0cb84ea8b1cfb4b1ac1b3d250775bafe2e3c5a"}
Oct 30 18:59:37.500301 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:37.500285    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e983b7aa177778d178222d2ca0cb84ea8b1cfb4b1ac1b3d250775bafe2e3c5a"
Oct 30 18:59:37.547289 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:37.547259    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4dc3e22-d7a2-45a9-891c-64857fa3188f" path="/var/lib/kubelet/pods/d4dc3e22-d7a2-45a9-891c-64857fa3188f/volumes"
Oct 30 18:59:38.510175 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:38.510139    4525 generic.go:356] "Generic (PLEG): container finished" podID="ecd67510-3824-4b69-bb7f-795a89600d5f" containerID="06aca8bce78f98b54b68c884c2164a558e7773c0463ce67f4619ec9cff50a933" exitCode=0
Oct 30 18:59:38.510579 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:38.510221    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f778f5-6v7zn" event={"ID":"ecd67510-3824-4b69-bb7f-795a89600d5f","Type":"ContainerDied","Data":"06aca8bce78f98b54b68c884c2164a558e7773c0463ce67f4619ec9cff50a933"}
Oct 30 18:59:38.510579 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:38.510274    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f778f5-6v7zn" event={"ID":"ecd67510-3824-4b69-bb7f-795a89600d5f","Type":"ContainerStarted","Data":"47daf6ffe09040a2e1bd4b0f8ebd4e06da2a50bd1d9846458ac87231d6a26fc9"}
Oct 30 18:59:39.683484 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:39.683456    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f778f5-6v7zn"
Oct 30 18:59:39.899312 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:39.899281    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ecd67510-3824-4b69-bb7f-795a89600d5f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecd67510-3824-4b69-bb7f-795a89600d5f-maas-api-xks-ca\") pod \"ecd67510-3824-4b69-bb7f-795a89600d5f\" (UID: \"ecd67510-3824-4b69-bb7f-795a89600d5f\") "
Oct 30 18:59:39.899486 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:39.899434    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecd67510-3824-4b69-bb7f-795a89600d5f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecd67510-3824-4b69-bb7f-795a89600d5f-maas-api-openshift-ca\") pod \"ecd67510-3824-4b69-bb7f-795a89600d5f\" (UID: \"ecd67510-3824-4b69-bb7f-795a89600d5f\") "
Oct 30 18:59:39.899486 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:39.899481    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecd67510-3824-4b69-bb7f-795a89600d5f-kube-api-access-xvhwk\" (UniqueName: \"kubernetes.io/projected/ecd67510-3824-4b69-bb7f-795a89600d5f-kube-api-access-xvhwk\") pod \"ecd67510-3824-4b69-bb7f-795a89600d5f\" (UID: \"ecd67510-3824-4b69-bb7f-795a89600d5f\") "
Oct 30 18:59:39.899883 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:39.899851    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecd67510-3824-4b69-bb7f-795a89600d5f-maas-api-openshift-ca" pod "ecd67510-3824-4b69-bb7f-795a89600d5f" (UID: "ecd67510-3824-4b69-bb7f-795a89600d5f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:59:39.900407 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:39.900384    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecd67510-3824-4b69-bb7f-795a89600d5f-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:39.901474 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:39.901445    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecd67510-3824-4b69-bb7f-795a89600d5f-maas-api-xks-ca" pod "ecd67510-3824-4b69-bb7f-795a89600d5f" (UID: "ecd67510-3824-4b69-bb7f-795a89600d5f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:59:39.902164 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:39.902142    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecd67510-3824-4b69-bb7f-795a89600d5f-kube-api-access-xvhwk" pod "ecd67510-3824-4b69-bb7f-795a89600d5f" (UID: "ecd67510-3824-4b69-bb7f-795a89600d5f"). InnerVolumeSpecName "kube-api-access-xvhwk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:59:40.001824 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:40.001784    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecd67510-3824-4b69-bb7f-795a89600d5f-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:40.001824 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:40.001822    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvhwk\" (UniqueName: \"kubernetes.io/projected/ecd67510-3824-4b69-bb7f-795a89600d5f-kube-api-access-xvhwk\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:40.532985 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:40.532952    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f778f5-6v7zn" event={"ID":"ecd67510-3824-4b69-bb7f-795a89600d5f","Type":"ContainerDied","Data":"47daf6ffe09040a2e1bd4b0f8ebd4e06da2a50bd1d9846458ac87231d6a26fc9"}
Oct 30 18:59:40.532985 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:40.532993    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f778f5-6v7zn"
Oct 30 18:59:40.533241 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:40.533034    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="47daf6ffe09040a2e1bd4b0f8ebd4e06da2a50bd1d9846458ac87231d6a26fc9"
Oct 30 18:59:44.012028 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:44.011995    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-10d367-75pxd"]
Oct 30 18:59:44.012588 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:44.012572    4525 cpu_manager.go:414] "containerMap: removing container" podUID="3b712abe-9c69-4d7f-b4d3-c172d6c0298a" containerName="revoke-keys"
Oct 30 18:59:44.012633 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:44.012590    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b712abe-9c69-4d7f-b4d3-c172d6c0298a" containerName="revoke-keys"
Oct 30 18:59:44.012633 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:44.012624    4525 cpu_manager.go:414] "containerMap: removing container" podUID="ecd67510-3824-4b69-bb7f-795a89600d5f" containerName="revoke-keys"
Oct 30 18:59:44.012633 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:44.012630    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecd67510-3824-4b69-bb7f-795a89600d5f" containerName="revoke-keys"
Oct 30 18:59:44.012765 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:44.012746    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecd67510-3824-4b69-bb7f-795a89600d5f" containerName="revoke-keys"
Oct 30 18:59:44.012765 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:44.012761    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b712abe-9c69-4d7f-b4d3-c172d6c0298a" containerName="revoke-keys"
Oct 30 18:59:44.016624 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:44.016605    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-10d367-75pxd"
Oct 30 18:59:44.026719 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:44.026673    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-10d367-75pxd"]
Oct 30 18:59:44.039016 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:44.038989    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94hcn\""
Oct 30 18:59:44.039205 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:44.039084    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:59:44.086622 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:44.086583    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c43ad6b-98ac-40a2-96e9-d39497988a67-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-10d367-75pxd\" (UID: \"1c43ad6b-98ac-40a2-96e9-d39497988a67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-10d367-75pxd"
Oct 30 18:59:44.086972 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:44.086948    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8rzl\" (UniqueName: \"kubernetes.io/projected/1c43ad6b-98ac-40a2-96e9-d39497988a67-kube-api-access-n8rzl\") pod \"maas-api-revoke-keys-e2e-praxis-b-10d367-75pxd\" (UID: \"1c43ad6b-98ac-40a2-96e9-d39497988a67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-10d367-75pxd"
Oct 30 18:59:44.087125 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:44.087108    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c43ad6b-98ac-40a2-96e9-d39497988a67-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-10d367-75pxd\" (UID: \"1c43ad6b-98ac-40a2-96e9-d39497988a67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-10d367-75pxd"
Oct 30 18:59:44.135439 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:44.135407    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-93574a-ccbc886cb-kq9wb"]
Oct 30 18:59:44.135625 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:44.135605    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-93574a-ccbc886cb-kq9wb" podUID="33de14c3-409a-4c51-b4c4-74a849da5b63" containerName="payload-processing" containerID="cri-o://15adbe12d785bb996e9c7462d61ec8dac5e227b576d82792b93cb56d717c8806" gracePeriod=30
Oct 30 18:59:44.151548 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:44.151519    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-93574a-84684456c9-k4pt5"]
Oct 30 18:59:44.151786 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:44.151765    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-93574a-84684456c9-k4pt5" podUID="4728df97-385f-4c6d-ac79-be8509ddc7ad" containerName="payload-pre-processing" containerID="cri-o://8fb3ffc670470a1aa1dd3fd56a850d37206d5bab833a613db9ac647eee05a61a" gracePeriod=30
Oct 30 18:59:44.190363 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:44.188530    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c43ad6b-98ac-40a2-96e9-d39497988a67-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-10d367-75pxd\" (UID: \"1c43ad6b-98ac-40a2-96e9-d39497988a67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-10d367-75pxd"
Oct 30 18:59:44.190363 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:44.188610    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8rzl\" (UniqueName: \"kubernetes.io/projected/1c43ad6b-98ac-40a2-96e9-d39497988a67-kube-api-access-n8rzl\") pod \"maas-api-revoke-keys-e2e-praxis-b-10d367-75pxd\" (UID: \"1c43ad6b-98ac-40a2-96e9-d39497988a67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-10d367-75pxd"
Oct 30 18:59:44.190363 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:44.188645    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c43ad6b-98ac-40a2-96e9-d39497988a67-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-10d367-75pxd\" (UID: \"1c43ad6b-98ac-40a2-96e9-d39497988a67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-10d367-75pxd"
Oct 30 18:59:44.190363 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:44.189270    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c43ad6b-98ac-40a2-96e9-d39497988a67-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-10d367-75pxd\" (UID: \"1c43ad6b-98ac-40a2-96e9-d39497988a67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-10d367-75pxd"
Oct 30 18:59:44.202113 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:44.202073    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c43ad6b-98ac-40a2-96e9-d39497988a67-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-10d367-75pxd\" (UID: \"1c43ad6b-98ac-40a2-96e9-d39497988a67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-10d367-75pxd"
Oct 30 18:59:44.209484 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:44.209458    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8rzl\" (UniqueName: \"kubernetes.io/projected/1c43ad6b-98ac-40a2-96e9-d39497988a67-kube-api-access-n8rzl\") pod \"maas-api-revoke-keys-e2e-praxis-b-10d367-75pxd\" (UID: \"1c43ad6b-98ac-40a2-96e9-d39497988a67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-10d367-75pxd"
Oct 30 18:59:44.343547 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:44.343517    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-10d367-75pxd"
Oct 30 18:59:44.631587 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:44.631546    4525 generic.go:356] "Generic (PLEG): container finished" podID="33de14c3-409a-4c51-b4c4-74a849da5b63" containerID="15adbe12d785bb996e9c7462d61ec8dac5e227b576d82792b93cb56d717c8806" exitCode=0
Oct 30 18:59:44.631965 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:44.631938    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-93574a-ccbc886cb-kq9wb" event={"ID":"33de14c3-409a-4c51-b4c4-74a849da5b63","Type":"ContainerDied","Data":"15adbe12d785bb996e9c7462d61ec8dac5e227b576d82792b93cb56d717c8806"}
Oct 30 18:59:44.640053 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:44.638776    4525 generic.go:356] "Generic (PLEG): container finished" podID="4728df97-385f-4c6d-ac79-be8509ddc7ad" containerID="8fb3ffc670470a1aa1dd3fd56a850d37206d5bab833a613db9ac647eee05a61a" exitCode=0
Oct 30 18:59:44.640053 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:44.638832    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-93574a-84684456c9-k4pt5" event={"ID":"4728df97-385f-4c6d-ac79-be8509ddc7ad","Type":"ContainerDied","Data":"8fb3ffc670470a1aa1dd3fd56a850d37206d5bab833a613db9ac647eee05a61a"}
Oct 30 18:59:44.656118 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:44.655785    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-93574a-ccbc886cb-kq9wb"
Oct 30 18:59:44.726490 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:44.726454    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-93574a-f2vcw"]
Oct 30 18:59:44.731086 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:44.731049    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-93574a-f2vcw"]
Oct 30 18:59:44.740325 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:44.740300    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-93574a-84684456c9-k4pt5"
Oct 30 18:59:44.753610 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:44.753580    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33de14c3-409a-4c51-b4c4-74a849da5b63-kube-api-access-nllf6\" (UniqueName: \"kubernetes.io/projected/33de14c3-409a-4c51-b4c4-74a849da5b63-kube-api-access-nllf6\") pod \"33de14c3-409a-4c51-b4c4-74a849da5b63\" (UID: \"33de14c3-409a-4c51-b4c4-74a849da5b63\") "
Oct 30 18:59:44.753775 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:44.753646    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33de14c3-409a-4c51-b4c4-74a849da5b63-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33de14c3-409a-4c51-b4c4-74a849da5b63-plugins-config-volume\") pod \"33de14c3-409a-4c51-b4c4-74a849da5b63\" (UID: \"33de14c3-409a-4c51-b4c4-74a849da5b63\") "
Oct 30 18:59:44.754009 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:44.753988    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33de14c3-409a-4c51-b4c4-74a849da5b63-plugins-config-volume" pod "33de14c3-409a-4c51-b4c4-74a849da5b63" (UID: "33de14c3-409a-4c51-b4c4-74a849da5b63"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:59:44.754421 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:44.754357    4525 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33de14c3-409a-4c51-b4c4-74a849da5b63-plugins-config-volume\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:44.755997 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:44.755963    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33de14c3-409a-4c51-b4c4-74a849da5b63-kube-api-access-nllf6" pod "33de14c3-409a-4c51-b4c4-74a849da5b63" (UID: "33de14c3-409a-4c51-b4c4-74a849da5b63"). InnerVolumeSpecName "kube-api-access-nllf6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:59:44.863723 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:44.857524    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4728df97-385f-4c6d-ac79-be8509ddc7ad-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4728df97-385f-4c6d-ac79-be8509ddc7ad-plugins-config-volume\") pod \"4728df97-385f-4c6d-ac79-be8509ddc7ad\" (UID: \"4728df97-385f-4c6d-ac79-be8509ddc7ad\") "
Oct 30 18:59:44.863723 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:44.857811    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4728df97-385f-4c6d-ac79-be8509ddc7ad-kube-api-access-8ltfr\" (UniqueName: \"kubernetes.io/projected/4728df97-385f-4c6d-ac79-be8509ddc7ad-kube-api-access-8ltfr\") pod \"4728df97-385f-4c6d-ac79-be8509ddc7ad\" (UID: \"4728df97-385f-4c6d-ac79-be8509ddc7ad\") "
Oct 30 18:59:44.863723 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:44.858575    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nllf6\" (UniqueName: \"kubernetes.io/projected/33de14c3-409a-4c51-b4c4-74a849da5b63-kube-api-access-nllf6\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:44.863723 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:44.858640    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4728df97-385f-4c6d-ac79-be8509ddc7ad-plugins-config-volume" pod "4728df97-385f-4c6d-ac79-be8509ddc7ad" (UID: "4728df97-385f-4c6d-ac79-be8509ddc7ad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:59:44.875707 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:44.875079    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4728df97-385f-4c6d-ac79-be8509ddc7ad-kube-api-access-8ltfr" pod "4728df97-385f-4c6d-ac79-be8509ddc7ad" (UID: "4728df97-385f-4c6d-ac79-be8509ddc7ad"). InnerVolumeSpecName "kube-api-access-8ltfr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:59:44.921236 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:44.921208    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-10d367-75pxd"]
Oct 30 18:59:44.921380 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:59:44.921207    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c43ad6b_98ac_40a2_96e9_d39497988a67.slice/crio-c17fba96f5051d7a1e1ffa0c32d4392716e04956e2ef3d13a6eede684605b9e3 WatchSource:0}: Error finding container c17fba96f5051d7a1e1ffa0c32d4392716e04956e2ef3d13a6eede684605b9e3: Status 404 returned error can't find the container with id c17fba96f5051d7a1e1ffa0c32d4392716e04956e2ef3d13a6eede684605b9e3
Oct 30 18:59:44.960386 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:44.960361    4525 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4728df97-385f-4c6d-ac79-be8509ddc7ad-plugins-config-volume\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:44.960386 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:44.960388    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ltfr\" (UniqueName: \"kubernetes.io/projected/4728df97-385f-4c6d-ac79-be8509ddc7ad-kube-api-access-8ltfr\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:45.553729 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:45.553616    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19602b5c-399d-4a0b-809e-5c46dd300f1d" path="/var/lib/kubelet/pods/19602b5c-399d-4a0b-809e-5c46dd300f1d/volumes"
Oct 30 18:59:45.650230 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:45.650202    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-93574a-84684456c9-k4pt5"
Oct 30 18:59:45.650838 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:45.650809    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-93574a-84684456c9-k4pt5" event={"ID":"4728df97-385f-4c6d-ac79-be8509ddc7ad","Type":"ContainerDied","Data":"f8e395e1a9d274817c96a583f055bb64ad5629c7ee862156d613dce84e31e4e6"}
Oct 30 18:59:45.651011 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:45.650871    4525 scope.go:122] "RemoveContainer" containerID="8fb3ffc670470a1aa1dd3fd56a850d37206d5bab833a613db9ac647eee05a61a"
Oct 30 18:59:45.653825 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:45.653784    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-93574a-ccbc886cb-kq9wb" event={"ID":"33de14c3-409a-4c51-b4c4-74a849da5b63","Type":"ContainerDied","Data":"1132e3e932c6b8b8ab1bac44df60a578ceeea9b51b33529932d66b76207aedff"}
Oct 30 18:59:45.653985 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:45.653911    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-93574a-ccbc886cb-kq9wb"
Oct 30 18:59:45.656770 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:45.656641    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-10d367-75pxd" event={"ID":"1c43ad6b-98ac-40a2-96e9-d39497988a67","Type":"ContainerStarted","Data":"99f0dfb1bddd00e89c7fa39da1789ef5fe9219f18c2acb5646587066e3c0c674"}
Oct 30 18:59:45.656770 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:45.656737    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-10d367-75pxd" event={"ID":"1c43ad6b-98ac-40a2-96e9-d39497988a67","Type":"ContainerStarted","Data":"c17fba96f5051d7a1e1ffa0c32d4392716e04956e2ef3d13a6eede684605b9e3"}
Oct 30 18:59:45.663740 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:45.663715    4525 scope.go:122] "RemoveContainer" containerID="15adbe12d785bb996e9c7462d61ec8dac5e227b576d82792b93cb56d717c8806"
Oct 30 18:59:45.679482 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:45.679456    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-93574a-84684456c9-k4pt5"]
Oct 30 18:59:45.683329 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:45.683299    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-93574a-84684456c9-k4pt5"]
Oct 30 18:59:45.715576 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:45.712506    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-10d367-75pxd" podStartSLOduration=2.712485031 podStartE2EDuration="2.712485031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:59:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:59:45.710398288 +0000 UTC m=+1408.336068253" watchObservedRunningTime="2026-10-30 18:59:45.712485031 +0000 UTC m=+1408.338154996"
Oct 30 18:59:45.801556 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:45.799737    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-93574a-ccbc886cb-kq9wb"]
Oct 30 18:59:45.809764 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:45.809433    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-93574a-ccbc886cb-kq9wb"]
Oct 30 18:59:46.049207 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:46.049141    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fe0612-xwq59"]
Oct 30 18:59:46.074864 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:46.074771    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fe0612-xwq59"]
Oct 30 18:59:46.732336 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:46.724507    4525 generic.go:356] "Generic (PLEG): container finished" podID="1c43ad6b-98ac-40a2-96e9-d39497988a67" containerID="99f0dfb1bddd00e89c7fa39da1789ef5fe9219f18c2acb5646587066e3c0c674" exitCode=0
Oct 30 18:59:46.735225 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:46.734936    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-10d367-75pxd" event={"ID":"1c43ad6b-98ac-40a2-96e9-d39497988a67","Type":"ContainerDied","Data":"99f0dfb1bddd00e89c7fa39da1789ef5fe9219f18c2acb5646587066e3c0c674"}
Oct 30 18:59:47.435343 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:59:47.435309    4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d49l" containerName="extractor"
Oct 30 18:59:47.548122 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:47.548087    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33de14c3-409a-4c51-b4c4-74a849da5b63" path="/var/lib/kubelet/pods/33de14c3-409a-4c51-b4c4-74a849da5b63/volumes"
Oct 30 18:59:47.548479 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:47.548465    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4728df97-385f-4c6d-ac79-be8509ddc7ad" path="/var/lib/kubelet/pods/4728df97-385f-4c6d-ac79-be8509ddc7ad/volumes"
Oct 30 18:59:47.548883 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:47.548869    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c22b56a5-f8a3-418c-b6b3-20d5e05b1f8b" path="/var/lib/kubelet/pods/c22b56a5-f8a3-418c-b6b3-20d5e05b1f8b/volumes"
Oct 30 18:59:47.952009 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:47.951979    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-10d367-75pxd"
Oct 30 18:59:47.971016 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:47.970984    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c43ad6b-98ac-40a2-96e9-d39497988a67-kube-api-access-n8rzl\" (UniqueName: \"kubernetes.io/projected/1c43ad6b-98ac-40a2-96e9-d39497988a67-kube-api-access-n8rzl\") pod \"1c43ad6b-98ac-40a2-96e9-d39497988a67\" (UID: \"1c43ad6b-98ac-40a2-96e9-d39497988a67\") "
Oct 30 18:59:47.971222 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:47.971055    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c43ad6b-98ac-40a2-96e9-d39497988a67-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c43ad6b-98ac-40a2-96e9-d39497988a67-maas-api-xks-ca\") pod \"1c43ad6b-98ac-40a2-96e9-d39497988a67\" (UID: \"1c43ad6b-98ac-40a2-96e9-d39497988a67\") "
Oct 30 18:59:47.971302 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:47.971267    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c43ad6b-98ac-40a2-96e9-d39497988a67-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c43ad6b-98ac-40a2-96e9-d39497988a67-maas-api-openshift-ca\") pod \"1c43ad6b-98ac-40a2-96e9-d39497988a67\" (UID: \"1c43ad6b-98ac-40a2-96e9-d39497988a67\") "
Oct 30 18:59:47.971937 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:47.971904    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c43ad6b-98ac-40a2-96e9-d39497988a67-maas-api-openshift-ca" pod "1c43ad6b-98ac-40a2-96e9-d39497988a67" (UID: "1c43ad6b-98ac-40a2-96e9-d39497988a67"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:59:47.973719 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:47.973657    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c43ad6b-98ac-40a2-96e9-d39497988a67-kube-api-access-n8rzl" pod "1c43ad6b-98ac-40a2-96e9-d39497988a67" (UID: "1c43ad6b-98ac-40a2-96e9-d39497988a67"). InnerVolumeSpecName "kube-api-access-n8rzl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:59:47.973857 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:47.973833    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c43ad6b-98ac-40a2-96e9-d39497988a67-maas-api-xks-ca" pod "1c43ad6b-98ac-40a2-96e9-d39497988a67" (UID: "1c43ad6b-98ac-40a2-96e9-d39497988a67"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:59:48.073226 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:48.073151    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8rzl\" (UniqueName: \"kubernetes.io/projected/1c43ad6b-98ac-40a2-96e9-d39497988a67-kube-api-access-n8rzl\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:48.073226 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:48.073179    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c43ad6b-98ac-40a2-96e9-d39497988a67-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:48.073226 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:48.073189    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c43ad6b-98ac-40a2-96e9-d39497988a67-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:48.767556 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:48.767516    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-10d367-75pxd" event={"ID":"1c43ad6b-98ac-40a2-96e9-d39497988a67","Type":"ContainerDied","Data":"c17fba96f5051d7a1e1ffa0c32d4392716e04956e2ef3d13a6eede684605b9e3"}
Oct 30 18:59:48.767861 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:48.767842    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c17fba96f5051d7a1e1ffa0c32d4392716e04956e2ef3d13a6eede684605b9e3"
Oct 30 18:59:48.768046 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:48.768019    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-10d367-75pxd"
Oct 30 18:59:51.345757 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:51.344813    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f778f5-6v7zn"]
Oct 30 18:59:51.394703 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:51.394582    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f778f5-6v7zn"]
Oct 30 18:59:51.595160 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:51.594967    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecd67510-3824-4b69-bb7f-795a89600d5f" path="/var/lib/kubelet/pods/ecd67510-3824-4b69-bb7f-795a89600d5f/volumes"
Oct 30 18:59:52.916553 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:52.916521    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-vd5tv"]
Oct 30 18:59:52.917066 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:52.916779    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-vd5tv" podUID="32211654-2ca8-4305-a4fc-fe06a98bcbd3" containerName="istio-proxy" containerID="cri-o://62ef678071b23fd005f0e988bc4df7947f643b5fd1044ca172e36a591bafe1b1" gracePeriod=30
Oct 30 18:59:56.295557 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:56.295522    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfa695b1-kmg2l"]
Oct 30 18:59:56.296125 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:56.296109    4525 cpu_manager.go:414] "containerMap: removing container" podUID="1c43ad6b-98ac-40a2-96e9-d39497988a67" containerName="revoke-keys"
Oct 30 18:59:56.296170 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:56.296126    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c43ad6b-98ac-40a2-96e9-d39497988a67" containerName="revoke-keys"
Oct 30 18:59:56.296170 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:56.296141    4525 cpu_manager.go:414] "containerMap: removing container" podUID="33de14c3-409a-4c51-b4c4-74a849da5b63" containerName="payload-processing"
Oct 30 18:59:56.296170 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:56.296146    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33de14c3-409a-4c51-b4c4-74a849da5b63" containerName="payload-processing"
Oct 30 18:59:56.296170 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:56.296169    4525 cpu_manager.go:414] "containerMap: removing container" podUID="4728df97-385f-4c6d-ac79-be8509ddc7ad" containerName="payload-pre-processing"
Oct 30 18:59:56.296293 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:56.296174    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4728df97-385f-4c6d-ac79-be8509ddc7ad" containerName="payload-pre-processing"
Oct 30 18:59:56.296293 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:56.296268    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c43ad6b-98ac-40a2-96e9-d39497988a67" containerName="revoke-keys"
Oct 30 18:59:56.296293 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:56.296275    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="33de14c3-409a-4c51-b4c4-74a849da5b63" containerName="payload-processing"
Oct 30 18:59:56.296293 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:56.296286    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="4728df97-385f-4c6d-ac79-be8509ddc7ad" containerName="payload-pre-processing"
Oct 30 18:59:56.331061 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:56.331031    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfa695b1-kmg2l"]
Oct 30 18:59:56.331298 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:56.331164    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfa695b1-kmg2l"
Oct 30 18:59:56.336853 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:56.336123    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:59:56.336853 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:56.336170    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94hcn\""
Oct 30 18:59:56.466229 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:56.465552    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/340f330d-01ae-46bd-bc7f-98d8984ef925-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dfa695b1-kmg2l\" (UID: \"340f330d-01ae-46bd-bc7f-98d8984ef925\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfa695b1-kmg2l"
Oct 30 18:59:56.466229 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:56.465614    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rp9g\" (UniqueName: \"kubernetes.io/projected/340f330d-01ae-46bd-bc7f-98d8984ef925-kube-api-access-6rp9g\") pod \"maas-api-revoke-sub-dfa695b1-kmg2l\" (UID: \"340f330d-01ae-46bd-bc7f-98d8984ef925\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfa695b1-kmg2l"
Oct 30 18:59:56.466229 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:56.465878    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/340f330d-01ae-46bd-bc7f-98d8984ef925-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dfa695b1-kmg2l\" (UID: \"340f330d-01ae-46bd-bc7f-98d8984ef925\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfa695b1-kmg2l"
Oct 30 18:59:56.567975 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:56.567888    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/340f330d-01ae-46bd-bc7f-98d8984ef925-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dfa695b1-kmg2l\" (UID: \"340f330d-01ae-46bd-bc7f-98d8984ef925\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfa695b1-kmg2l"
Oct 30 18:59:56.568133 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:56.568031    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/340f330d-01ae-46bd-bc7f-98d8984ef925-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dfa695b1-kmg2l\" (UID: \"340f330d-01ae-46bd-bc7f-98d8984ef925\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfa695b1-kmg2l"
Oct 30 18:59:56.568133 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:56.568053    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rp9g\" (UniqueName: \"kubernetes.io/projected/340f330d-01ae-46bd-bc7f-98d8984ef925-kube-api-access-6rp9g\") pod \"maas-api-revoke-sub-dfa695b1-kmg2l\" (UID: \"340f330d-01ae-46bd-bc7f-98d8984ef925\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfa695b1-kmg2l"
Oct 30 18:59:56.568550 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:56.568514    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/340f330d-01ae-46bd-bc7f-98d8984ef925-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dfa695b1-kmg2l\" (UID: \"340f330d-01ae-46bd-bc7f-98d8984ef925\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfa695b1-kmg2l"
Oct 30 18:59:56.570957 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:56.570929    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/340f330d-01ae-46bd-bc7f-98d8984ef925-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dfa695b1-kmg2l\" (UID: \"340f330d-01ae-46bd-bc7f-98d8984ef925\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfa695b1-kmg2l"
Oct 30 18:59:56.576661 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:56.576637    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rp9g\" (UniqueName: \"kubernetes.io/projected/340f330d-01ae-46bd-bc7f-98d8984ef925-kube-api-access-6rp9g\") pod \"maas-api-revoke-sub-dfa695b1-kmg2l\" (UID: \"340f330d-01ae-46bd-bc7f-98d8984ef925\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfa695b1-kmg2l"
Oct 30 18:59:56.662588 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:56.662560    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfa695b1-kmg2l"
Oct 30 18:59:56.974980 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:59:56.974950    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod340f330d_01ae_46bd_bc7f_98d8984ef925.slice/crio-2da78851c752f10a0cc030fe50a154a60b73bafb894af3e62a6e0f70c12cf55a WatchSource:0}: Error finding container 2da78851c752f10a0cc030fe50a154a60b73bafb894af3e62a6e0f70c12cf55a: Status 404 returned error can't find the container with id 2da78851c752f10a0cc030fe50a154a60b73bafb894af3e62a6e0f70c12cf55a
Oct 30 18:59:56.977500 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:56.977471    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfa695b1-kmg2l"]
Oct 30 18:59:57.869108 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:57.869043    4525 patch_prober.go:29] interesting pod/e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Oct 30 18:59:57.869507 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:57.869096    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589" podUID="af747a06-3fa4-4c7f-a5a3-7e33a6d68746" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Oct 30 18:59:58.106436 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:58.106390    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfa695b1-kmg2l" event={"ID":"340f330d-01ae-46bd-bc7f-98d8984ef925","Type":"ContainerStarted","Data":"92d0af373283536101e96497ed13431ce734b7d159aa203d01b397382259a142"}
Oct 30 18:59:58.106612 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:58.106461    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfa695b1-kmg2l" event={"ID":"340f330d-01ae-46bd-bc7f-98d8984ef925","Type":"ContainerStarted","Data":"2da78851c752f10a0cc030fe50a154a60b73bafb894af3e62a6e0f70c12cf55a"}
Oct 30 18:59:58.123344 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:58.123247    4525 generic.go:356] "Generic (PLEG): container finished" podID="af747a06-3fa4-4c7f-a5a3-7e33a6d68746" containerID="4cf262d2a7d076ae846899bee2c71fb71c18aab346e48f1914fb85cc3249adf0" exitCode=0
Oct 30 18:59:58.123500 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:58.123358    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589" event={"ID":"af747a06-3fa4-4c7f-a5a3-7e33a6d68746","Type":"ContainerDied","Data":"4cf262d2a7d076ae846899bee2c71fb71c18aab346e48f1914fb85cc3249adf0"}
Oct 30 18:59:58.124741 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:58.123982    4525 scope.go:122] "RemoveContainer" containerID="4cf262d2a7d076ae846899bee2c71fb71c18aab346e48f1914fb85cc3249adf0"
Oct 30 18:59:58.126642 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:58.126598    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:59:58.126964 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:58.126934    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:59:58.127187 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:58.127164    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 18:59:58.132422 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:58.132380    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfa695b1-kmg2l" podStartSLOduration=2.132366885 podStartE2EDuration="2.132366885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:59:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:59:58.129540469 +0000 UTC m=+1420.755210467" watchObservedRunningTime="2026-10-30 18:59:58.132366885 +0000 UTC m=+1420.758036848"
Oct 30 18:59:58.197452 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:59:58.197405    4525 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32211654_2ca8_4305_a4fc_fe06a98bcbd3.slice/crio-conmon-62ef678071b23fd005f0e988bc4df7947f643b5fd1044ca172e36a591bafe1b1.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32211654_2ca8_4305_a4fc_fe06a98bcbd3.slice/crio-62ef678071b23fd005f0e988bc4df7947f643b5fd1044ca172e36a591bafe1b1.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 18:59:58.824449 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:58.824418    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-vd5tv"
Oct 30 18:59:58.880025 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:58.879989    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3d1aa8-2x2zv"]
Oct 30 18:59:58.880584 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:58.880567    4525 cpu_manager.go:414] "containerMap: removing container" podUID="32211654-2ca8-4305-a4fc-fe06a98bcbd3" containerName="istio-proxy"
Oct 30 18:59:58.880584 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:58.880583    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32211654-2ca8-4305-a4fc-fe06a98bcbd3" containerName="istio-proxy"
Oct 30 18:59:58.880750 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:58.880719    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="32211654-2ca8-4305-a4fc-fe06a98bcbd3" containerName="istio-proxy"
Oct 30 18:59:58.895777 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:58.895747    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3d1aa8-2x2zv"]
Oct 30 18:59:58.902807 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:58.895916    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3d1aa8-2x2zv"
Oct 30 18:59:58.928595 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:58.928567    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32211654-2ca8-4305-a4fc-fe06a98bcbd3-istio-token\" (UniqueName: \"kubernetes.io/projected/32211654-2ca8-4305-a4fc-fe06a98bcbd3-istio-token\") pod \"32211654-2ca8-4305-a4fc-fe06a98bcbd3\" (UID: \"32211654-2ca8-4305-a4fc-fe06a98bcbd3\") "
Oct 30 18:59:58.928796 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:58.928664    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32211654-2ca8-4305-a4fc-fe06a98bcbd3-kube-api-access-jpbqb\" (UniqueName: \"kubernetes.io/projected/32211654-2ca8-4305-a4fc-fe06a98bcbd3-kube-api-access-jpbqb\") pod \"32211654-2ca8-4305-a4fc-fe06a98bcbd3\" (UID: \"32211654-2ca8-4305-a4fc-fe06a98bcbd3\") "
Oct 30 18:59:58.928796 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:58.928723    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32211654-2ca8-4305-a4fc-fe06a98bcbd3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32211654-2ca8-4305-a4fc-fe06a98bcbd3-istio-envoy\") pod \"32211654-2ca8-4305-a4fc-fe06a98bcbd3\" (UID: \"32211654-2ca8-4305-a4fc-fe06a98bcbd3\") "
Oct 30 18:59:58.928796 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:58.928772    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/32211654-2ca8-4305-a4fc-fe06a98bcbd3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32211654-2ca8-4305-a4fc-fe06a98bcbd3-istio-podinfo\") pod \"32211654-2ca8-4305-a4fc-fe06a98bcbd3\" (UID: \"32211654-2ca8-4305-a4fc-fe06a98bcbd3\") "
Oct 30 18:59:58.928978 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:58.928802    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32211654-2ca8-4305-a4fc-fe06a98bcbd3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32211654-2ca8-4305-a4fc-fe06a98bcbd3-credential-socket\") pod \"32211654-2ca8-4305-a4fc-fe06a98bcbd3\" (UID: \"32211654-2ca8-4305-a4fc-fe06a98bcbd3\") "
Oct 30 18:59:58.928978 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:58.928847    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32211654-2ca8-4305-a4fc-fe06a98bcbd3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32211654-2ca8-4305-a4fc-fe06a98bcbd3-workload-certs\") pod \"32211654-2ca8-4305-a4fc-fe06a98bcbd3\" (UID: \"32211654-2ca8-4305-a4fc-fe06a98bcbd3\") "
Oct 30 18:59:58.928978 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:58.928970    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32211654-2ca8-4305-a4fc-fe06a98bcbd3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32211654-2ca8-4305-a4fc-fe06a98bcbd3-istiod-ca-cert\") pod \"32211654-2ca8-4305-a4fc-fe06a98bcbd3\" (UID: \"32211654-2ca8-4305-a4fc-fe06a98bcbd3\") "
Oct 30 18:59:58.929131 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:58.929119    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32211654-2ca8-4305-a4fc-fe06a98bcbd3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32211654-2ca8-4305-a4fc-fe06a98bcbd3-istio-data\") pod \"32211654-2ca8-4305-a4fc-fe06a98bcbd3\" (UID: \"32211654-2ca8-4305-a4fc-fe06a98bcbd3\") "
Oct 30 18:59:58.929867 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:58.929221    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32211654-2ca8-4305-a4fc-fe06a98bcbd3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32211654-2ca8-4305-a4fc-fe06a98bcbd3-workload-socket\") pod \"32211654-2ca8-4305-a4fc-fe06a98bcbd3\" (UID: \"32211654-2ca8-4305-a4fc-fe06a98bcbd3\") "
Oct 30 18:59:58.929867 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:58.929834    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32211654-2ca8-4305-a4fc-fe06a98bcbd3-istiod-ca-cert" pod "32211654-2ca8-4305-a4fc-fe06a98bcbd3" (UID: "32211654-2ca8-4305-a4fc-fe06a98bcbd3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:59:58.930233 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:58.930098    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32211654-2ca8-4305-a4fc-fe06a98bcbd3-workload-certs" pod "32211654-2ca8-4305-a4fc-fe06a98bcbd3" (UID: "32211654-2ca8-4305-a4fc-fe06a98bcbd3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:59:58.930315 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:58.930295    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32211654-2ca8-4305-a4fc-fe06a98bcbd3-workload-socket" pod "32211654-2ca8-4305-a4fc-fe06a98bcbd3" (UID: "32211654-2ca8-4305-a4fc-fe06a98bcbd3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:59:58.930546 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:58.930435    4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32211654-2ca8-4305-a4fc-fe06a98bcbd3-workload-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:58.930546 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:58.930456    4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32211654-2ca8-4305-a4fc-fe06a98bcbd3-workload-certs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:58.930546 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:58.930465    4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32211654-2ca8-4305-a4fc-fe06a98bcbd3-istiod-ca-cert\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:58.930546 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:58.930518    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32211654-2ca8-4305-a4fc-fe06a98bcbd3-istio-data" pod "32211654-2ca8-4305-a4fc-fe06a98bcbd3" (UID: "32211654-2ca8-4305-a4fc-fe06a98bcbd3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:59:58.931225 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:58.931162    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32211654-2ca8-4305-a4fc-fe06a98bcbd3-credential-socket" pod "32211654-2ca8-4305-a4fc-fe06a98bcbd3" (UID: "32211654-2ca8-4305-a4fc-fe06a98bcbd3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:59:58.933125 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:58.933098    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/32211654-2ca8-4305-a4fc-fe06a98bcbd3-istio-podinfo" pod "32211654-2ca8-4305-a4fc-fe06a98bcbd3" (UID: "32211654-2ca8-4305-a4fc-fe06a98bcbd3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 18:59:58.933882 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:58.933811    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32211654-2ca8-4305-a4fc-fe06a98bcbd3-istio-token" pod "32211654-2ca8-4305-a4fc-fe06a98bcbd3" (UID: "32211654-2ca8-4305-a4fc-fe06a98bcbd3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:59:58.933882 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:58.933849    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32211654-2ca8-4305-a4fc-fe06a98bcbd3-istio-envoy" pod "32211654-2ca8-4305-a4fc-fe06a98bcbd3" (UID: "32211654-2ca8-4305-a4fc-fe06a98bcbd3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:59:58.935473 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:58.935442    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32211654-2ca8-4305-a4fc-fe06a98bcbd3-kube-api-access-jpbqb" pod "32211654-2ca8-4305-a4fc-fe06a98bcbd3" (UID: "32211654-2ca8-4305-a4fc-fe06a98bcbd3"). InnerVolumeSpecName "kube-api-access-jpbqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:59:59.031694 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:59.031648    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlsks\" (UniqueName: \"kubernetes.io/projected/e2a4a34a-0549-436e-9a18-f73d265b949e-kube-api-access-zlsks\") pod \"maas-api-revoke-keys-e2e-praxis-b-3d1aa8-2x2zv\" (UID: \"e2a4a34a-0549-436e-9a18-f73d265b949e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3d1aa8-2x2zv"
Oct 30 18:59:59.031846 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:59.031826    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2a4a34a-0549-436e-9a18-f73d265b949e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-3d1aa8-2x2zv\" (UID: \"e2a4a34a-0549-436e-9a18-f73d265b949e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3d1aa8-2x2zv"
Oct 30 18:59:59.032007 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:59.031985    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2a4a34a-0549-436e-9a18-f73d265b949e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-3d1aa8-2x2zv\" (UID: \"e2a4a34a-0549-436e-9a18-f73d265b949e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3d1aa8-2x2zv"
Oct 30 18:59:59.032192 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:59.032181    4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32211654-2ca8-4305-a4fc-fe06a98bcbd3-istio-data\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:59.032228 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:59.032195    4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32211654-2ca8-4305-a4fc-fe06a98bcbd3-istio-token\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:59.032228 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:59.032206    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpbqb\" (UniqueName: \"kubernetes.io/projected/32211654-2ca8-4305-a4fc-fe06a98bcbd3-kube-api-access-jpbqb\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:59.032228 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:59.032217    4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32211654-2ca8-4305-a4fc-fe06a98bcbd3-istio-envoy\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:59.032228 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:59.032225    4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32211654-2ca8-4305-a4fc-fe06a98bcbd3-istio-podinfo\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:59.032347 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:59.032234    4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32211654-2ca8-4305-a4fc-fe06a98bcbd3-credential-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:59.133146 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:59.133104    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlsks\" (UniqueName: \"kubernetes.io/projected/e2a4a34a-0549-436e-9a18-f73d265b949e-kube-api-access-zlsks\") pod \"maas-api-revoke-keys-e2e-praxis-b-3d1aa8-2x2zv\" (UID: \"e2a4a34a-0549-436e-9a18-f73d265b949e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3d1aa8-2x2zv"
Oct 30 18:59:59.133353 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:59.133198    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2a4a34a-0549-436e-9a18-f73d265b949e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-3d1aa8-2x2zv\" (UID: \"e2a4a34a-0549-436e-9a18-f73d265b949e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3d1aa8-2x2zv"
Oct 30 18:59:59.133353 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:59.133335    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2a4a34a-0549-436e-9a18-f73d265b949e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-3d1aa8-2x2zv\" (UID: \"e2a4a34a-0549-436e-9a18-f73d265b949e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3d1aa8-2x2zv"
Oct 30 18:59:59.133520 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:59.133496    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589" event={"ID":"af747a06-3fa4-4c7f-a5a3-7e33a6d68746","Type":"ContainerStarted","Data":"7d8b22f61a5ce3938d4ce4476451de0f79008caa8503e488122f8256130a4bfd"}
Oct 30 18:59:59.134077 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:59.134051    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2a4a34a-0549-436e-9a18-f73d265b949e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-3d1aa8-2x2zv\" (UID: \"e2a4a34a-0549-436e-9a18-f73d265b949e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3d1aa8-2x2zv"
Oct 30 18:59:59.135230 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:59.135206    4525 generic.go:356] "Generic (PLEG): container finished" podID="32211654-2ca8-4305-a4fc-fe06a98bcbd3" containerID="62ef678071b23fd005f0e988bc4df7947f643b5fd1044ca172e36a591bafe1b1" exitCode=0
Oct 30 18:59:59.135344 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:59.135273    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-vd5tv"
Oct 30 18:59:59.135344 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:59.135287    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-vd5tv" event={"ID":"32211654-2ca8-4305-a4fc-fe06a98bcbd3","Type":"ContainerDied","Data":"62ef678071b23fd005f0e988bc4df7947f643b5fd1044ca172e36a591bafe1b1"}
Oct 30 18:59:59.135344 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:59.135335    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-vd5tv" event={"ID":"32211654-2ca8-4305-a4fc-fe06a98bcbd3","Type":"ContainerDied","Data":"6cbb1ba01814984f11394f721ed04979011e8db210856ba35fab55d934acd305"}
Oct 30 18:59:59.135477 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:59.135353    4525 scope.go:122] "RemoveContainer" containerID="62ef678071b23fd005f0e988bc4df7947f643b5fd1044ca172e36a591bafe1b1"
Oct 30 18:59:59.137128 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:59.137104    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2a4a34a-0549-436e-9a18-f73d265b949e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-3d1aa8-2x2zv\" (UID: \"e2a4a34a-0549-436e-9a18-f73d265b949e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3d1aa8-2x2zv"
Oct 30 18:59:59.137720 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:59.137661    4525 generic.go:356] "Generic (PLEG): container finished" podID="340f330d-01ae-46bd-bc7f-98d8984ef925" containerID="92d0af373283536101e96497ed13431ce734b7d159aa203d01b397382259a142" exitCode=0
Oct 30 18:59:59.137893 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:59.137754    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfa695b1-kmg2l" event={"ID":"340f330d-01ae-46bd-bc7f-98d8984ef925","Type":"ContainerDied","Data":"92d0af373283536101e96497ed13431ce734b7d159aa203d01b397382259a142"}
Oct 30 18:59:59.143227 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:59.143206    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlsks\" (UniqueName: \"kubernetes.io/projected/e2a4a34a-0549-436e-9a18-f73d265b949e-kube-api-access-zlsks\") pod \"maas-api-revoke-keys-e2e-praxis-b-3d1aa8-2x2zv\" (UID: \"e2a4a34a-0549-436e-9a18-f73d265b949e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3d1aa8-2x2zv"
Oct 30 18:59:59.155562 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:59.155541    4525 scope.go:122] "RemoveContainer" containerID="62ef678071b23fd005f0e988bc4df7947f643b5fd1044ca172e36a591bafe1b1"
Oct 30 18:59:59.156864 ip-10-0-137-135 kubenswrapper[4525]: E1030 18:59:59.156838    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62ef678071b23fd005f0e988bc4df7947f643b5fd1044ca172e36a591bafe1b1\": container with ID starting with 62ef678071b23fd005f0e988bc4df7947f643b5fd1044ca172e36a591bafe1b1 not found: ID does not exist" containerID="62ef678071b23fd005f0e988bc4df7947f643b5fd1044ca172e36a591bafe1b1"
Oct 30 18:59:59.156980 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:59.156874    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62ef678071b23fd005f0e988bc4df7947f643b5fd1044ca172e36a591bafe1b1"} err="failed to get container status \"62ef678071b23fd005f0e988bc4df7947f643b5fd1044ca172e36a591bafe1b1\": rpc error: code = NotFound desc = could not find container \"62ef678071b23fd005f0e988bc4df7947f643b5fd1044ca172e36a591bafe1b1\": container with ID starting with 62ef678071b23fd005f0e988bc4df7947f643b5fd1044ca172e36a591bafe1b1 not found: ID does not exist"
Oct 30 18:59:59.173492 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:59.173467    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-vd5tv"]
Oct 30 18:59:59.200705 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:59.192329    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-fe0612-openshift-default-884c77ff7-vd5tv"]
Oct 30 18:59:59.234577 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:59.234540    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3d1aa8-2x2zv"
Oct 30 18:59:59.544486 ip-10-0-137-135 kubenswrapper[4525]: W1030 18:59:59.544449    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2a4a34a_0549_436e_9a18_f73d265b949e.slice/crio-866b70d54380e4bf6c3f5cd40246103af674270fe607f0e2d1a48ce72ef3792b WatchSource:0}: Error finding container 866b70d54380e4bf6c3f5cd40246103af674270fe607f0e2d1a48ce72ef3792b: Status 404 returned error can't find the container with id 866b70d54380e4bf6c3f5cd40246103af674270fe607f0e2d1a48ce72ef3792b
Oct 30 18:59:59.548830 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:59.548803    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32211654-2ca8-4305-a4fc-fe06a98bcbd3" path="/var/lib/kubelet/pods/32211654-2ca8-4305-a4fc-fe06a98bcbd3/volumes"
Oct 30 18:59:59.549234 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:59.549212    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3d1aa8-2x2zv"]
Oct 30 18:59:59.652756 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:59.652724    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589"
Oct 30 18:59:59.653111 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:59.653091    4525 patch_prober.go:29] interesting pod/e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Oct 30 18:59:59.653176 ip-10-0-137-135 kubenswrapper[4525]: I1030 18:59:59.653136    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589" podUID="af747a06-3fa4-4c7f-a5a3-7e33a6d68746" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Oct 30 19:00:00.142905 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:00.142850    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0458df-29889780-dn85w"]
Oct 30 19:00:00.147545 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:00.147516    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-85cb41-29889780-4gb6k"]
Oct 30 19:00:00.147748 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:00.147724    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0458df-29889780-dn85w"
Oct 30 19:00:00.153022 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:00.153001    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-24fceb-29889780-p99bs"]
Oct 30 19:00:00.153249 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:00.153226    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-85cb41-29889780-4gb6k"
Oct 30 19:00:00.158348 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:00.158319    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3d1aa8-2x2zv" event={"ID":"e2a4a34a-0549-436e-9a18-f73d265b949e","Type":"ContainerStarted","Data":"787ec368354820bad674908a98df2bed483589dba61c803f3e6858a66afc4635"}
Oct 30 19:00:00.158449 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:00.158413    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-24fceb-29889780-p99bs"
Oct 30 19:00:00.158503 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:00.158466    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3d1aa8-2x2zv" event={"ID":"e2a4a34a-0549-436e-9a18-f73d265b949e","Type":"ContainerStarted","Data":"866b70d54380e4bf6c3f5cd40246103af674270fe607f0e2d1a48ce72ef3792b"}
Oct 30 19:00:00.158503 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:00.158499    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39bde3-29889780-jcgz6"]
Oct 30 19:00:00.169644 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:00.168415    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22f70fc7-4efd-423a-b7a1-a0c74be53b60-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-24fceb-29889780-p99bs\" (UID: \"22f70fc7-4efd-423a-b7a1-a0c74be53b60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-24fceb-29889780-p99bs"
Oct 30 19:00:00.169885 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:00.169860    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qg4mq\" (UniqueName: \"kubernetes.io/projected/22f70fc7-4efd-423a-b7a1-a0c74be53b60-kube-api-access-qg4mq\") pod \"maas-api-key-cleanup-e2e-ipp-b-24fceb-29889780-p99bs\" (UID: \"22f70fc7-4efd-423a-b7a1-a0c74be53b60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-24fceb-29889780-p99bs"
Oct 30 19:00:00.173444 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:00.173410    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0458df-29889780-dn85w"]
Oct 30 19:00:00.173897 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:00.173526    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39bde3-29889780-jcgz6"
Oct 30 19:00:00.217226 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:00.217191    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-24fceb-29889780-p99bs"]
Oct 30 19:00:00.224908 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:00.224883    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-85cb41-29889780-4gb6k"]
Oct 30 19:00:00.233558 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:00.233537    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-10d367-29889780-qhjkk"]
Oct 30 19:00:00.239961 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:00.239939    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-10d367-29889780-qhjkk"
Oct 30 19:00:00.249171 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:00.249141    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-3d1aa8-29889780-kkcgg"]
Oct 30 19:00:00.260097 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:00.260074    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-3d1aa8-29889780-kkcgg"
Oct 30 19:00:00.260856 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:00.259978    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39bde3-29889780-jcgz6"]
Oct 30 19:00:00.263945 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:00.263894    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-10d367-29889780-qhjkk"]
Oct 30 19:00:00.267799 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:00.267784    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-3d1aa8-29889780-kkcgg"]
Oct 30 19:00:00.268583 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:00.268544    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3d1aa8-2x2zv" podStartSLOduration=2.268532267 podStartE2EDuration="2.268532267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:59:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:00.223903882 +0000 UTC m=+1422.849573883" watchObservedRunningTime="2026-10-30 19:00:00.268532267 +0000 UTC m=+1422.894202250"
Oct 30 19:00:00.271923 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:00.271896    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dchr\" (UniqueName: \"kubernetes.io/projected/bf90b413-2e8d-4a11-8e79-639c05ab0f2f-kube-api-access-6dchr\") pod \"maas-api-key-cleanup-e2e-praxis-b-3d1aa8-29889780-kkcgg\" (UID: \"bf90b413-2e8d-4a11-8e79-639c05ab0f2f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-3d1aa8-29889780-kkcgg"
Oct 30 19:00:00.272074 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:00.272014    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plnlp\" (UniqueName: \"kubernetes.io/projected/653dd99e-09c9-4e7f-af3c-b18bc7612fae-kube-api-access-plnlp\") pod \"maas-api-key-cleanup-e2e-worker-w1-85cb41-29889780-4gb6k\" (UID: \"653dd99e-09c9-4e7f-af3c-b18bc7612fae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-85cb41-29889780-4gb6k"
Oct 30 19:00:00.272159 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:00.272098    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3d8e6fc-8e1b-472f-9c8c-aab9b9c06e73-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-39bde3-29889780-jcgz6\" (UID: \"e3d8e6fc-8e1b-472f-9c8c-aab9b9c06e73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39bde3-29889780-jcgz6"
Oct 30 19:00:00.272219 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:00.272156    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f67c70b1-a75f-47f3-b6e5-fb3ec635707d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-10d367-29889780-qhjkk\" (UID: \"f67c70b1-a75f-47f3-b6e5-fb3ec635707d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-10d367-29889780-qhjkk"
Oct 30 19:00:00.272219 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:00.272202    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqxqg\" (UniqueName: \"kubernetes.io/projected/e3d8e6fc-8e1b-472f-9c8c-aab9b9c06e73-kube-api-access-jqxqg\") pod \"maas-api-key-cleanup-e2e-worker-w2-39bde3-29889780-jcgz6\" (UID: \"e3d8e6fc-8e1b-472f-9c8c-aab9b9c06e73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39bde3-29889780-jcgz6"
Oct 30 19:00:00.272329 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:00.272306    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf90b413-2e8d-4a11-8e79-639c05ab0f2f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-3d1aa8-29889780-kkcgg\" (UID: \"bf90b413-2e8d-4a11-8e79-639c05ab0f2f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-3d1aa8-29889780-kkcgg"
Oct 30 19:00:00.272385 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:00.272362    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22f70fc7-4efd-423a-b7a1-a0c74be53b60-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-24fceb-29889780-p99bs\" (UID: \"22f70fc7-4efd-423a-b7a1-a0c74be53b60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-24fceb-29889780-p99bs"
Oct 30 19:00:00.272445 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:00.272421    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-947gk\" (UniqueName: \"kubernetes.io/projected/49b2daff-5ea2-4a05-9152-9cc75d4eeba5-kube-api-access-947gk\") pod \"maas-api-key-cleanup-e2e-worker-w4-0458df-29889780-dn85w\" (UID: \"49b2daff-5ea2-4a05-9152-9cc75d4eeba5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0458df-29889780-dn85w"
Oct 30 19:00:00.272504 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:00.272465    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fgjm\" (UniqueName: \"kubernetes.io/projected/f67c70b1-a75f-47f3-b6e5-fb3ec635707d-kube-api-access-6fgjm\") pod \"maas-api-key-cleanup-e2e-praxis-b-10d367-29889780-qhjkk\" (UID: \"f67c70b1-a75f-47f3-b6e5-fb3ec635707d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-10d367-29889780-qhjkk"
Oct 30 19:00:00.273223 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:00.272555    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/653dd99e-09c9-4e7f-af3c-b18bc7612fae-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-85cb41-29889780-4gb6k\" (UID: \"653dd99e-09c9-4e7f-af3c-b18bc7612fae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-85cb41-29889780-4gb6k"
Oct 30 19:00:00.273223 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:00.272641    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49b2daff-5ea2-4a05-9152-9cc75d4eeba5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-0458df-29889780-dn85w\" (UID: \"49b2daff-5ea2-4a05-9152-9cc75d4eeba5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0458df-29889780-dn85w"
Oct 30 19:00:00.273223 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:00.272865    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qg4mq\" (UniqueName: \"kubernetes.io/projected/22f70fc7-4efd-423a-b7a1-a0c74be53b60-kube-api-access-qg4mq\") pod \"maas-api-key-cleanup-e2e-ipp-b-24fceb-29889780-p99bs\" (UID: \"22f70fc7-4efd-423a-b7a1-a0c74be53b60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-24fceb-29889780-p99bs"
Oct 30 19:00:00.273223 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:00.273186    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22f70fc7-4efd-423a-b7a1-a0c74be53b60-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-24fceb-29889780-p99bs\" (UID: \"22f70fc7-4efd-423a-b7a1-a0c74be53b60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-24fceb-29889780-p99bs"
Oct 30 19:00:00.309493 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:00.309457    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qg4mq\" (UniqueName: \"kubernetes.io/projected/22f70fc7-4efd-423a-b7a1-a0c74be53b60-kube-api-access-qg4mq\") pod \"maas-api-key-cleanup-e2e-ipp-b-24fceb-29889780-p99bs\" (UID: \"22f70fc7-4efd-423a-b7a1-a0c74be53b60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-24fceb-29889780-p99bs"
Oct 30 19:00:00.373569 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:00.373537    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf90b413-2e8d-4a11-8e79-639c05ab0f2f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-3d1aa8-29889780-kkcgg\" (UID: \"bf90b413-2e8d-4a11-8e79-639c05ab0f2f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-3d1aa8-29889780-kkcgg"
Oct 30 19:00:00.373813 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:00.373598    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-947gk\" (UniqueName: \"kubernetes.io/projected/49b2daff-5ea2-4a05-9152-9cc75d4eeba5-kube-api-access-947gk\") pod \"maas-api-key-cleanup-e2e-worker-w4-0458df-29889780-dn85w\" (UID: \"49b2daff-5ea2-4a05-9152-9cc75d4eeba5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0458df-29889780-dn85w"
Oct 30 19:00:00.373813 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:00.373632    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fgjm\" (UniqueName: \"kubernetes.io/projected/f67c70b1-a75f-47f3-b6e5-fb3ec635707d-kube-api-access-6fgjm\") pod \"maas-api-key-cleanup-e2e-praxis-b-10d367-29889780-qhjkk\" (UID: \"f67c70b1-a75f-47f3-b6e5-fb3ec635707d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-10d367-29889780-qhjkk"
Oct 30 19:00:00.373957 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:00.373847    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/653dd99e-09c9-4e7f-af3c-b18bc7612fae-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-85cb41-29889780-4gb6k\" (UID: \"653dd99e-09c9-4e7f-af3c-b18bc7612fae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-85cb41-29889780-4gb6k"
Oct 30 19:00:00.373957 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:00.373877    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49b2daff-5ea2-4a05-9152-9cc75d4eeba5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-0458df-29889780-dn85w\" (UID: \"49b2daff-5ea2-4a05-9152-9cc75d4eeba5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0458df-29889780-dn85w"
Oct 30 19:00:00.373957 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:00.373940    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dchr\" (UniqueName: \"kubernetes.io/projected/bf90b413-2e8d-4a11-8e79-639c05ab0f2f-kube-api-access-6dchr\") pod \"maas-api-key-cleanup-e2e-praxis-b-3d1aa8-29889780-kkcgg\" (UID: \"bf90b413-2e8d-4a11-8e79-639c05ab0f2f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-3d1aa8-29889780-kkcgg"
Oct 30 19:00:00.374118 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:00.374028    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plnlp\" (UniqueName: \"kubernetes.io/projected/653dd99e-09c9-4e7f-af3c-b18bc7612fae-kube-api-access-plnlp\") pod \"maas-api-key-cleanup-e2e-worker-w1-85cb41-29889780-4gb6k\" (UID: \"653dd99e-09c9-4e7f-af3c-b18bc7612fae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-85cb41-29889780-4gb6k"
Oct 30 19:00:00.374118 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:00.374052    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3d8e6fc-8e1b-472f-9c8c-aab9b9c06e73-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-39bde3-29889780-jcgz6\" (UID: \"e3d8e6fc-8e1b-472f-9c8c-aab9b9c06e73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39bde3-29889780-jcgz6"
Oct 30 19:00:00.374118 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:00.374103    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f67c70b1-a75f-47f3-b6e5-fb3ec635707d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-10d367-29889780-qhjkk\" (UID: \"f67c70b1-a75f-47f3-b6e5-fb3ec635707d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-10d367-29889780-qhjkk"
Oct 30 19:00:00.374264 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:00.374137    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqxqg\" (UniqueName: \"kubernetes.io/projected/e3d8e6fc-8e1b-472f-9c8c-aab9b9c06e73-kube-api-access-jqxqg\") pod \"maas-api-key-cleanup-e2e-worker-w2-39bde3-29889780-jcgz6\" (UID: \"e3d8e6fc-8e1b-472f-9c8c-aab9b9c06e73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39bde3-29889780-jcgz6"
Oct 30 19:00:00.374392 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:00.374369    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf90b413-2e8d-4a11-8e79-639c05ab0f2f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-3d1aa8-29889780-kkcgg\" (UID: \"bf90b413-2e8d-4a11-8e79-639c05ab0f2f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-3d1aa8-29889780-kkcgg"
Oct 30 19:00:00.374601 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:00.374579    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/653dd99e-09c9-4e7f-af3c-b18bc7612fae-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-85cb41-29889780-4gb6k\" (UID: \"653dd99e-09c9-4e7f-af3c-b18bc7612fae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-85cb41-29889780-4gb6k"
Oct 30 19:00:00.374945 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:00.374926    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49b2daff-5ea2-4a05-9152-9cc75d4eeba5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-0458df-29889780-dn85w\" (UID: \"49b2daff-5ea2-4a05-9152-9cc75d4eeba5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0458df-29889780-dn85w"
Oct 30 19:00:00.375338 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:00.375312    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3d8e6fc-8e1b-472f-9c8c-aab9b9c06e73-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-39bde3-29889780-jcgz6\" (UID: \"e3d8e6fc-8e1b-472f-9c8c-aab9b9c06e73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39bde3-29889780-jcgz6"
Oct 30 19:00:00.375478 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:00.375453    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f67c70b1-a75f-47f3-b6e5-fb3ec635707d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-10d367-29889780-qhjkk\" (UID: \"f67c70b1-a75f-47f3-b6e5-fb3ec635707d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-10d367-29889780-qhjkk"
Oct 30 19:00:00.417512 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:00.417479    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fgjm\" (UniqueName: \"kubernetes.io/projected/f67c70b1-a75f-47f3-b6e5-fb3ec635707d-kube-api-access-6fgjm\") pod \"maas-api-key-cleanup-e2e-praxis-b-10d367-29889780-qhjkk\" (UID: \"f67c70b1-a75f-47f3-b6e5-fb3ec635707d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-10d367-29889780-qhjkk"
Oct 30 19:00:00.421364 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:00.421295    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-947gk\" (UniqueName: \"kubernetes.io/projected/49b2daff-5ea2-4a05-9152-9cc75d4eeba5-kube-api-access-947gk\") pod \"maas-api-key-cleanup-e2e-worker-w4-0458df-29889780-dn85w\" (UID: \"49b2daff-5ea2-4a05-9152-9cc75d4eeba5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0458df-29889780-dn85w"
Oct 30 19:00:00.421903 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:00.421511    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plnlp\" (UniqueName: \"kubernetes.io/projected/653dd99e-09c9-4e7f-af3c-b18bc7612fae-kube-api-access-plnlp\") pod \"maas-api-key-cleanup-e2e-worker-w1-85cb41-29889780-4gb6k\" (UID: \"653dd99e-09c9-4e7f-af3c-b18bc7612fae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-85cb41-29889780-4gb6k"
Oct 30 19:00:00.421903 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:00.421864    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqxqg\" (UniqueName: \"kubernetes.io/projected/e3d8e6fc-8e1b-472f-9c8c-aab9b9c06e73-kube-api-access-jqxqg\") pod \"maas-api-key-cleanup-e2e-worker-w2-39bde3-29889780-jcgz6\" (UID: \"e3d8e6fc-8e1b-472f-9c8c-aab9b9c06e73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39bde3-29889780-jcgz6"
Oct 30 19:00:00.424604 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:00.423928    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dchr\" (UniqueName: \"kubernetes.io/projected/bf90b413-2e8d-4a11-8e79-639c05ab0f2f-kube-api-access-6dchr\") pod \"maas-api-key-cleanup-e2e-praxis-b-3d1aa8-29889780-kkcgg\" (UID: \"bf90b413-2e8d-4a11-8e79-639c05ab0f2f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-3d1aa8-29889780-kkcgg"
Oct 30 19:00:00.434785 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:00.434168    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-10d367-29889780-qhjkk"]
Oct 30 19:00:00.434785 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:00.434568    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-10d367-29889780-qhjkk"
Oct 30 19:00:00.452780 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:00.452744    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfa695b1-kmg2l"
Oct 30 19:00:00.506708 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:00.475241    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/340f330d-01ae-46bd-bc7f-98d8984ef925-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/340f330d-01ae-46bd-bc7f-98d8984ef925-maas-api-xks-ca\") pod \"340f330d-01ae-46bd-bc7f-98d8984ef925\" (UID: \"340f330d-01ae-46bd-bc7f-98d8984ef925\") "
Oct 30 19:00:00.506708 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:00.475312    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/340f330d-01ae-46bd-bc7f-98d8984ef925-kube-api-access-6rp9g\" (UniqueName: \"kubernetes.io/projected/340f330d-01ae-46bd-bc7f-98d8984ef925-kube-api-access-6rp9g\") pod \"340f330d-01ae-46bd-bc7f-98d8984ef925\" (UID: \"340f330d-01ae-46bd-bc7f-98d8984ef925\") "
Oct 30 19:00:00.506708 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:00.475479    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/340f330d-01ae-46bd-bc7f-98d8984ef925-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/340f330d-01ae-46bd-bc7f-98d8984ef925-maas-api-openshift-ca\") pod \"340f330d-01ae-46bd-bc7f-98d8984ef925\" (UID: \"340f330d-01ae-46bd-bc7f-98d8984ef925\") "
Oct 30 19:00:00.508594 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:00.508543    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/340f330d-01ae-46bd-bc7f-98d8984ef925-maas-api-openshift-ca" pod "340f330d-01ae-46bd-bc7f-98d8984ef925" (UID: "340f330d-01ae-46bd-bc7f-98d8984ef925"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:00:00.510272 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:00.510240    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/340f330d-01ae-46bd-bc7f-98d8984ef925-maas-api-xks-ca" pod "340f330d-01ae-46bd-bc7f-98d8984ef925" (UID: "340f330d-01ae-46bd-bc7f-98d8984ef925"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:00:00.511472 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:00.511275    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/340f330d-01ae-46bd-bc7f-98d8984ef925-kube-api-access-6rp9g" pod "340f330d-01ae-46bd-bc7f-98d8984ef925" (UID: "340f330d-01ae-46bd-bc7f-98d8984ef925"). InnerVolumeSpecName "kube-api-access-6rp9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:00:00.513370 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:00.513344    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0458df-29889780-dn85w"
Oct 30 19:00:00.522033 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:00.521869    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-85cb41-29889780-4gb6k"
Oct 30 19:00:00.534397 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:00.534189    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-24fceb-29889780-p99bs"
Oct 30 19:00:00.557941 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:00.557912    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39bde3-29889780-jcgz6"
Oct 30 19:00:00.612736 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:00.592062    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/340f330d-01ae-46bd-bc7f-98d8984ef925-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:00.612736 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:00.592105    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/340f330d-01ae-46bd-bc7f-98d8984ef925-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:00.612736 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:00.592124    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rp9g\" (UniqueName: \"kubernetes.io/projected/340f330d-01ae-46bd-bc7f-98d8984ef925-kube-api-access-6rp9g\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:00.612736 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:00.611965    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-3d1aa8-29889780-kkcgg"
Oct 30 19:00:00.659317 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:00.656284    4525 patch_prober.go:29] interesting pod/e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Oct 30 19:00:00.659317 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:00.656339    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589" podUID="af747a06-3fa4-4c7f-a5a3-7e33a6d68746" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Oct 30 19:00:00.889633 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:00.887352    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-10d367-75pxd"]
Oct 30 19:00:00.937865 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:00.937726    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-10d367-75pxd"]
Oct 30 19:00:00.949233 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:00.949181    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-10d367-29889780-qhjkk"]
Oct 30 19:00:01.026975 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:00:01.026001    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf67c70b1_a75f_47f3_b6e5_fb3ec635707d.slice/crio-7a9bc33089b08dcc5a1bcdf6b0ea21ae14118f907ffb23ef1c160ba77311664d WatchSource:0}: Error finding container 7a9bc33089b08dcc5a1bcdf6b0ea21ae14118f907ffb23ef1c160ba77311664d: Status 404 returned error can't find the container with id 7a9bc33089b08dcc5a1bcdf6b0ea21ae14118f907ffb23ef1c160ba77311664d
Oct 30 19:00:01.220938 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:01.217906    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfa695b1-kmg2l"
Oct 30 19:00:01.220938 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:01.217907    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfa695b1-kmg2l" event={"ID":"340f330d-01ae-46bd-bc7f-98d8984ef925","Type":"ContainerDied","Data":"2da78851c752f10a0cc030fe50a154a60b73bafb894af3e62a6e0f70c12cf55a"}
Oct 30 19:00:01.220938 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:01.218004    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2da78851c752f10a0cc030fe50a154a60b73bafb894af3e62a6e0f70c12cf55a"
Oct 30 19:00:01.226327 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:01.225292    4525 generic.go:356] "Generic (PLEG): container finished" podID="e2a4a34a-0549-436e-9a18-f73d265b949e" containerID="787ec368354820bad674908a98df2bed483589dba61c803f3e6858a66afc4635" exitCode=0
Oct 30 19:00:01.226327 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:01.225867    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3d1aa8-2x2zv" event={"ID":"e2a4a34a-0549-436e-9a18-f73d265b949e","Type":"ContainerDied","Data":"787ec368354820bad674908a98df2bed483589dba61c803f3e6858a66afc4635"}
Oct 30 19:00:01.227705 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:01.227661    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-10d367-29889780-qhjkk" event={"ID":"f67c70b1-a75f-47f3-b6e5-fb3ec635707d","Type":"ContainerStarted","Data":"7a9bc33089b08dcc5a1bcdf6b0ea21ae14118f907ffb23ef1c160ba77311664d"}
Oct 30 19:00:01.319414 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:01.319386    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0458df-29889780-dn85w"]
Oct 30 19:00:01.323827 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:01.323799    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-85cb41-29889780-4gb6k"]
Oct 30 19:00:01.328434 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:01.328408    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-24fceb-29889780-p99bs"]
Oct 30 19:00:01.336184 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:01.336144    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39bde3-29889780-jcgz6"]
Oct 30 19:00:01.342661 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:00:01.342630    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3d8e6fc_8e1b_472f_9c8c_aab9b9c06e73.slice/crio-7e404bc2ca8e10db427008ebd0fcde068d69230d0ecdbcf1e34d5a65375abbf1 WatchSource:0}: Error finding container 7e404bc2ca8e10db427008ebd0fcde068d69230d0ecdbcf1e34d5a65375abbf1: Status 404 returned error can't find the container with id 7e404bc2ca8e10db427008ebd0fcde068d69230d0ecdbcf1e34d5a65375abbf1
Oct 30 19:00:01.412178 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:01.412147    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-3d1aa8-29889780-kkcgg"]
Oct 30 19:00:01.546188 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:01.546161    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c43ad6b-98ac-40a2-96e9-d39497988a67" path="/var/lib/kubelet/pods/1c43ad6b-98ac-40a2-96e9-d39497988a67/volumes"
Oct 30 19:00:01.653852 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:01.653817    4525 patch_prober.go:29] interesting pod/e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Oct 30 19:00:01.654010 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:01.653876    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589" podUID="af747a06-3fa4-4c7f-a5a3-7e33a6d68746" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Oct 30 19:00:02.240005 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:02.239978    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39bde3-29889780-jcgz6" event={"ID":"e3d8e6fc-8e1b-472f-9c8c-aab9b9c06e73","Type":"ContainerStarted","Data":"7382121d00ad66ad0075d27bf0ba7d816fe5e29d32e607308339901bb2e839c1"}
Oct 30 19:00:02.240422 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:02.240053    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39bde3-29889780-jcgz6" event={"ID":"e3d8e6fc-8e1b-472f-9c8c-aab9b9c06e73","Type":"ContainerStarted","Data":"7e404bc2ca8e10db427008ebd0fcde068d69230d0ecdbcf1e34d5a65375abbf1"}
Oct 30 19:00:02.241734 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:02.241710    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0458df-29889780-dn85w" event={"ID":"49b2daff-5ea2-4a05-9152-9cc75d4eeba5","Type":"ContainerStarted","Data":"2b5206bdabd41aae0622710367514bc2ff725aa38419176e748db547b2223701"}
Oct 30 19:00:02.241841 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:02.241752    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0458df-29889780-dn85w" event={"ID":"49b2daff-5ea2-4a05-9152-9cc75d4eeba5","Type":"ContainerStarted","Data":"3ec8f6b62ffa34579741d05635ab0a591c5fee4afdc1374141e304fbfb56854b"}
Oct 30 19:00:02.243401 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:02.243377    4525 generic.go:356] "Generic (PLEG): container finished" podID="f67c70b1-a75f-47f3-b6e5-fb3ec635707d" containerID="22e8b81c99ff53ad968567ed1410c08bc3dde4cabcb307ef5f471eb21539ac97" exitCode=6
Oct 30 19:00:02.243667 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:02.243464    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-10d367-29889780-qhjkk" event={"ID":"f67c70b1-a75f-47f3-b6e5-fb3ec635707d","Type":"ContainerDied","Data":"22e8b81c99ff53ad968567ed1410c08bc3dde4cabcb307ef5f471eb21539ac97"}
Oct 30 19:00:02.245246 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:02.245198    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-3d1aa8-29889780-kkcgg" event={"ID":"bf90b413-2e8d-4a11-8e79-639c05ab0f2f","Type":"ContainerStarted","Data":"a2cf2f65a1950948b9b5174135a09157ab3f96ceddabeacbd4b6abbb4608c886"}
Oct 30 19:00:02.245246 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:02.245236    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-3d1aa8-29889780-kkcgg" event={"ID":"bf90b413-2e8d-4a11-8e79-639c05ab0f2f","Type":"ContainerStarted","Data":"fa3163911c823757c971fb4b2eb358dd9295f791ecfd6df66cd17d23f6dd5624"}
Oct 30 19:00:02.246915 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:02.246893    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-24fceb-29889780-p99bs" event={"ID":"22f70fc7-4efd-423a-b7a1-a0c74be53b60","Type":"ContainerStarted","Data":"b7284386525d4d5f5fa5ca3458da4e92155f138c695a1915b34e806fb1844401"}
Oct 30 19:00:02.247016 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:02.246943    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-24fceb-29889780-p99bs" event={"ID":"22f70fc7-4efd-423a-b7a1-a0c74be53b60","Type":"ContainerStarted","Data":"1e9d1f301a74818d4f1f8a1d7a7aa56d27b50d1948dde4b943f19ba3ecde39d5"}
Oct 30 19:00:02.248584 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:02.248561    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-85cb41-29889780-4gb6k" event={"ID":"653dd99e-09c9-4e7f-af3c-b18bc7612fae","Type":"ContainerStarted","Data":"8e35bfa7013e1d3e8c9bc8777eb52956b384ce3f9f9a4d8bd1854bb09c91c3ae"}
Oct 30 19:00:02.248664 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:02.248609    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-85cb41-29889780-4gb6k" event={"ID":"653dd99e-09c9-4e7f-af3c-b18bc7612fae","Type":"ContainerStarted","Data":"a77101a6aaf76a9ad9096f331e5aad5441dfa23560141758efbe467e36ce3d85"}
Oct 30 19:00:02.259290 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:02.258163    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39bde3-29889780-jcgz6" podStartSLOduration=2.258151113 podStartE2EDuration="2.258151113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:02.256575619 +0000 UTC m=+1424.882245584" watchObservedRunningTime="2026-10-30 19:00:02.258151113 +0000 UTC m=+1424.883821110"
Oct 30 19:00:02.275339 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:02.275283    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0458df-29889780-dn85w" podStartSLOduration=2.275268868 podStartE2EDuration="2.275268868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:02.271509733 +0000 UTC m=+1424.897179762" watchObservedRunningTime="2026-10-30 19:00:02.275268868 +0000 UTC m=+1424.900938832"
Oct 30 19:00:02.320834 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:02.320632    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-85cb41-29889780-4gb6k" podStartSLOduration=2.320613543 podStartE2EDuration="2.320613543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:02.291080097 +0000 UTC m=+1424.916750062" watchObservedRunningTime="2026-10-30 19:00:02.320613543 +0000 UTC m=+1424.946283507"
Oct 30 19:00:02.324453 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:02.324403    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-3d1aa8-29889780-kkcgg" podStartSLOduration=2.324385456 podStartE2EDuration="2.324385456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:02.313161547 +0000 UTC m=+1424.938831512" watchObservedRunningTime="2026-10-30 19:00:02.324385456 +0000 UTC m=+1424.950055424"
Oct 30 19:00:02.337890 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:02.337842    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-24fceb-29889780-p99bs" podStartSLOduration=2.337824669 podStartE2EDuration="2.337824669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:02.331372044 +0000 UTC m=+1424.957042008" watchObservedRunningTime="2026-10-30 19:00:02.337824669 +0000 UTC m=+1424.963494636"
Oct 30 19:00:02.470710 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:02.470669    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-10d367-29889780-qhjkk"
Oct 30 19:00:02.563896 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:02.563858    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f67c70b1-a75f-47f3-b6e5-fb3ec635707d-kube-api-access-6fgjm\" (UniqueName: \"kubernetes.io/projected/f67c70b1-a75f-47f3-b6e5-fb3ec635707d-kube-api-access-6fgjm\") pod \"f67c70b1-a75f-47f3-b6e5-fb3ec635707d\" (UID: \"f67c70b1-a75f-47f3-b6e5-fb3ec635707d\") "
Oct 30 19:00:02.564059 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:02.564022    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f67c70b1-a75f-47f3-b6e5-fb3ec635707d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f67c70b1-a75f-47f3-b6e5-fb3ec635707d-maas-api-service-ca\") pod \"f67c70b1-a75f-47f3-b6e5-fb3ec635707d\" (UID: \"f67c70b1-a75f-47f3-b6e5-fb3ec635707d\") "
Oct 30 19:00:02.564599 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:02.564405    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f67c70b1-a75f-47f3-b6e5-fb3ec635707d-maas-api-service-ca" pod "f67c70b1-a75f-47f3-b6e5-fb3ec635707d" (UID: "f67c70b1-a75f-47f3-b6e5-fb3ec635707d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:00:02.564816 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:02.564797    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f67c70b1-a75f-47f3-b6e5-fb3ec635707d-maas-api-service-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:02.566142 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:02.566120    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f67c70b1-a75f-47f3-b6e5-fb3ec635707d-kube-api-access-6fgjm" pod "f67c70b1-a75f-47f3-b6e5-fb3ec635707d" (UID: "f67c70b1-a75f-47f3-b6e5-fb3ec635707d"). InnerVolumeSpecName "kube-api-access-6fgjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:00:02.611689 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:02.611660    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3d1aa8-2x2zv"
Oct 30 19:00:02.699464 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:02.695344    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fgjm\" (UniqueName: \"kubernetes.io/projected/f67c70b1-a75f-47f3-b6e5-fb3ec635707d-kube-api-access-6fgjm\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:02.744284 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:02.743768    4525 patch_prober.go:29] interesting pod/e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Oct 30 19:00:02.744284 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:02.743830    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589" podUID="af747a06-3fa4-4c7f-a5a3-7e33a6d68746" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Oct 30 19:00:02.797706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:02.797158    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2a4a34a-0549-436e-9a18-f73d265b949e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2a4a34a-0549-436e-9a18-f73d265b949e-maas-api-openshift-ca\") pod \"e2a4a34a-0549-436e-9a18-f73d265b949e\" (UID: \"e2a4a34a-0549-436e-9a18-f73d265b949e\") "
Oct 30 19:00:02.797706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:02.797306    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2a4a34a-0549-436e-9a18-f73d265b949e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2a4a34a-0549-436e-9a18-f73d265b949e-maas-api-xks-ca\") pod \"e2a4a34a-0549-436e-9a18-f73d265b949e\" (UID: \"e2a4a34a-0549-436e-9a18-f73d265b949e\") "
Oct 30 19:00:02.797706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:02.797461    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2a4a34a-0549-436e-9a18-f73d265b949e-kube-api-access-zlsks\" (UniqueName: \"kubernetes.io/projected/e2a4a34a-0549-436e-9a18-f73d265b949e-kube-api-access-zlsks\") pod \"e2a4a34a-0549-436e-9a18-f73d265b949e\" (UID: \"e2a4a34a-0549-436e-9a18-f73d265b949e\") "
Oct 30 19:00:02.801818 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:02.801730    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2a4a34a-0549-436e-9a18-f73d265b949e-maas-api-openshift-ca" pod "e2a4a34a-0549-436e-9a18-f73d265b949e" (UID: "e2a4a34a-0549-436e-9a18-f73d265b949e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:00:02.803614 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:02.803541    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2a4a34a-0549-436e-9a18-f73d265b949e-maas-api-xks-ca" pod "e2a4a34a-0549-436e-9a18-f73d265b949e" (UID: "e2a4a34a-0549-436e-9a18-f73d265b949e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:00:02.804123 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:02.804098    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2a4a34a-0549-436e-9a18-f73d265b949e-kube-api-access-zlsks" pod "e2a4a34a-0549-436e-9a18-f73d265b949e" (UID: "e2a4a34a-0549-436e-9a18-f73d265b949e"). InnerVolumeSpecName "kube-api-access-zlsks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:00:02.903432 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:02.903390    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2a4a34a-0549-436e-9a18-f73d265b949e-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:02.903432 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:02.903436    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2a4a34a-0549-436e-9a18-f73d265b949e-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:02.903724 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:02.903452    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlsks\" (UniqueName: \"kubernetes.io/projected/e2a4a34a-0549-436e-9a18-f73d265b949e-kube-api-access-zlsks\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:03.260930 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:03.260899    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3d1aa8-2x2zv" event={"ID":"e2a4a34a-0549-436e-9a18-f73d265b949e","Type":"ContainerDied","Data":"866b70d54380e4bf6c3f5cd40246103af674270fe607f0e2d1a48ce72ef3792b"}
Oct 30 19:00:03.261316 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:03.260949    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="866b70d54380e4bf6c3f5cd40246103af674270fe607f0e2d1a48ce72ef3792b"
Oct 30 19:00:03.261316 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:03.260997    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3d1aa8-2x2zv"
Oct 30 19:00:03.267297 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:03.267265    4525 generic.go:356] "Generic (PLEG): container finished" podID="653dd99e-09c9-4e7f-af3c-b18bc7612fae" containerID="8e35bfa7013e1d3e8c9bc8777eb52956b384ce3f9f9a4d8bd1854bb09c91c3ae" exitCode=60
Oct 30 19:00:03.267544 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:03.267350    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-85cb41-29889780-4gb6k" event={"ID":"653dd99e-09c9-4e7f-af3c-b18bc7612fae","Type":"ContainerDied","Data":"8e35bfa7013e1d3e8c9bc8777eb52956b384ce3f9f9a4d8bd1854bb09c91c3ae"}
Oct 30 19:00:03.267758 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:03.267738    4525 scope.go:122] "RemoveContainer" containerID="8e35bfa7013e1d3e8c9bc8777eb52956b384ce3f9f9a4d8bd1854bb09c91c3ae"
Oct 30 19:00:03.270026 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:03.269990    4525 generic.go:356] "Generic (PLEG): container finished" podID="e3d8e6fc-8e1b-472f-9c8c-aab9b9c06e73" containerID="7382121d00ad66ad0075d27bf0ba7d816fe5e29d32e607308339901bb2e839c1" exitCode=60
Oct 30 19:00:03.270144 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:03.270034    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39bde3-29889780-jcgz6" event={"ID":"e3d8e6fc-8e1b-472f-9c8c-aab9b9c06e73","Type":"ContainerDied","Data":"7382121d00ad66ad0075d27bf0ba7d816fe5e29d32e607308339901bb2e839c1"}
Oct 30 19:00:03.270353 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:03.270336    4525 scope.go:122] "RemoveContainer" containerID="7382121d00ad66ad0075d27bf0ba7d816fe5e29d32e607308339901bb2e839c1"
Oct 30 19:00:03.271826 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:03.271808    4525 generic.go:356] "Generic (PLEG): container finished" podID="49b2daff-5ea2-4a05-9152-9cc75d4eeba5" containerID="2b5206bdabd41aae0622710367514bc2ff725aa38419176e748db547b2223701" exitCode=60
Oct 30 19:00:03.271895 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:03.271873    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0458df-29889780-dn85w" event={"ID":"49b2daff-5ea2-4a05-9152-9cc75d4eeba5","Type":"ContainerDied","Data":"2b5206bdabd41aae0622710367514bc2ff725aa38419176e748db547b2223701"}
Oct 30 19:00:03.272130 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:03.272116    4525 scope.go:122] "RemoveContainer" containerID="2b5206bdabd41aae0622710367514bc2ff725aa38419176e748db547b2223701"
Oct 30 19:00:03.273319 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:03.273306    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-10d367-29889780-qhjkk"
Oct 30 19:00:03.273390 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:03.273346    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-10d367-29889780-qhjkk" event={"ID":"f67c70b1-a75f-47f3-b6e5-fb3ec635707d","Type":"ContainerDied","Data":"7a9bc33089b08dcc5a1bcdf6b0ea21ae14118f907ffb23ef1c160ba77311664d"}
Oct 30 19:00:03.273436 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:03.273416    4525 scope.go:122] "RemoveContainer" containerID="22e8b81c99ff53ad968567ed1410c08bc3dde4cabcb307ef5f471eb21539ac97"
Oct 30 19:00:03.275316 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:03.275294    4525 generic.go:356] "Generic (PLEG): container finished" podID="bf90b413-2e8d-4a11-8e79-639c05ab0f2f" containerID="a2cf2f65a1950948b9b5174135a09157ab3f96ceddabeacbd4b6abbb4608c886" exitCode=60
Oct 30 19:00:03.275410 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:03.275371    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-3d1aa8-29889780-kkcgg" event={"ID":"bf90b413-2e8d-4a11-8e79-639c05ab0f2f","Type":"ContainerDied","Data":"a2cf2f65a1950948b9b5174135a09157ab3f96ceddabeacbd4b6abbb4608c886"}
Oct 30 19:00:03.275607 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:03.275591    4525 scope.go:122] "RemoveContainer" containerID="a2cf2f65a1950948b9b5174135a09157ab3f96ceddabeacbd4b6abbb4608c886"
Oct 30 19:00:03.333341 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:03.325655    4525 generic.go:356] "Generic (PLEG): container finished" podID="22f70fc7-4efd-423a-b7a1-a0c74be53b60" containerID="b7284386525d4d5f5fa5ca3458da4e92155f138c695a1915b34e806fb1844401" exitCode=60
Oct 30 19:00:03.333341 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:03.325808    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-24fceb-29889780-p99bs" event={"ID":"22f70fc7-4efd-423a-b7a1-a0c74be53b60","Type":"ContainerDied","Data":"b7284386525d4d5f5fa5ca3458da4e92155f138c695a1915b34e806fb1844401"}
Oct 30 19:00:03.333341 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:03.326464    4525 scope.go:122] "RemoveContainer" containerID="b7284386525d4d5f5fa5ca3458da4e92155f138c695a1915b34e806fb1844401"
Oct 30 19:00:03.522417 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:03.522338    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-10d367-29889780-qhjkk"]
Oct 30 19:00:03.527997 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:03.527970    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-10d367-29889780-qhjkk"]
Oct 30 19:00:03.546327 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:03.546303    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f67c70b1-a75f-47f3-b6e5-fb3ec635707d" path="/var/lib/kubelet/pods/f67c70b1-a75f-47f3-b6e5-fb3ec635707d/volumes"
Oct 30 19:00:03.652654 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:03.652621    4525 patch_prober.go:29] interesting pod/e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Oct 30 19:00:03.652840 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:03.652673    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589" podUID="af747a06-3fa4-4c7f-a5a3-7e33a6d68746" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Oct 30 19:00:04.355990 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:04.355508    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-85cb41-29889780-4gb6k" event={"ID":"653dd99e-09c9-4e7f-af3c-b18bc7612fae","Type":"ContainerStarted","Data":"2a73b95fdc634571dcddfbf4ecc715fa28c8a6be824243fcbec1449e5ca8896f"}
Oct 30 19:00:04.360644 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:04.360590    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39bde3-29889780-jcgz6" event={"ID":"e3d8e6fc-8e1b-472f-9c8c-aab9b9c06e73","Type":"ContainerStarted","Data":"65e8e744841281e474dcfac0c440e2f3992275088afb110564d345325af18272"}
Oct 30 19:00:04.366153 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:04.366097    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0458df-29889780-dn85w" event={"ID":"49b2daff-5ea2-4a05-9152-9cc75d4eeba5","Type":"ContainerStarted","Data":"54df706b5019abff3efb1cd492350636daf0b1e6bf9eff1fc4b735d40ca64085"}
Oct 30 19:00:04.372381 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:04.371657    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-3d1aa8-29889780-kkcgg" event={"ID":"bf90b413-2e8d-4a11-8e79-639c05ab0f2f","Type":"ContainerStarted","Data":"b8279571794f2c4abd5a39ceeb2cab1ef699eb3a971ad3b9af249bec8117fade"}
Oct 30 19:00:04.394192 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:04.394123    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-24fceb-29889780-p99bs" event={"ID":"22f70fc7-4efd-423a-b7a1-a0c74be53b60","Type":"ContainerStarted","Data":"4f381f743ad681fbcf3ed76f9e829efba7fc1db87ff7c020c35d4d635ccf7d6c"}
Oct 30 19:00:04.653175 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:04.653134    4525 patch_prober.go:29] interesting pod/e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Oct 30 19:00:04.653357 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:04.653191    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589" podUID="af747a06-3fa4-4c7f-a5a3-7e33a6d68746" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Oct 30 19:00:05.409819 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:05.409776    4525 generic.go:356] "Generic (PLEG): container finished" podID="bf90b413-2e8d-4a11-8e79-639c05ab0f2f" containerID="b8279571794f2c4abd5a39ceeb2cab1ef699eb3a971ad3b9af249bec8117fade" exitCode=60
Oct 30 19:00:05.410333 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:05.409907    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-3d1aa8-29889780-kkcgg" event={"ID":"bf90b413-2e8d-4a11-8e79-639c05ab0f2f","Type":"ContainerDied","Data":"b8279571794f2c4abd5a39ceeb2cab1ef699eb3a971ad3b9af249bec8117fade"}
Oct 30 19:00:05.410333 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:05.410005    4525 scope.go:122] "RemoveContainer" containerID="a2cf2f65a1950948b9b5174135a09157ab3f96ceddabeacbd4b6abbb4608c886"
Oct 30 19:00:05.410450 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:05.410393    4525 scope.go:122] "RemoveContainer" containerID="b8279571794f2c4abd5a39ceeb2cab1ef699eb3a971ad3b9af249bec8117fade"
Oct 30 19:00:05.410951 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:00:05.410919    4525 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-3d1aa8-29889780-kkcgg_odh-ai-gateway-infra(bf90b413-2e8d-4a11-8e79-639c05ab0f2f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-3d1aa8-29889780-kkcgg" podUID="bf90b413-2e8d-4a11-8e79-639c05ab0f2f"
Oct 30 19:00:05.416332 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:05.416311    4525 generic.go:356] "Generic (PLEG): container finished" podID="22f70fc7-4efd-423a-b7a1-a0c74be53b60" containerID="4f381f743ad681fbcf3ed76f9e829efba7fc1db87ff7c020c35d4d635ccf7d6c" exitCode=60
Oct 30 19:00:05.416455 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:05.416394    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-24fceb-29889780-p99bs" event={"ID":"22f70fc7-4efd-423a-b7a1-a0c74be53b60","Type":"ContainerDied","Data":"4f381f743ad681fbcf3ed76f9e829efba7fc1db87ff7c020c35d4d635ccf7d6c"}
Oct 30 19:00:05.416658 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:05.416640    4525 scope.go:122] "RemoveContainer" containerID="4f381f743ad681fbcf3ed76f9e829efba7fc1db87ff7c020c35d4d635ccf7d6c"
Oct 30 19:00:05.416946 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:00:05.416926    4525 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-24fceb-29889780-p99bs_odh-ai-gateway-infra(22f70fc7-4efd-423a-b7a1-a0c74be53b60)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-24fceb-29889780-p99bs" podUID="22f70fc7-4efd-423a-b7a1-a0c74be53b60"
Oct 30 19:00:05.418505 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:05.418486    4525 generic.go:356] "Generic (PLEG): container finished" podID="653dd99e-09c9-4e7f-af3c-b18bc7612fae" containerID="2a73b95fdc634571dcddfbf4ecc715fa28c8a6be824243fcbec1449e5ca8896f" exitCode=60
Oct 30 19:00:05.418592 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:05.418527    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-85cb41-29889780-4gb6k" event={"ID":"653dd99e-09c9-4e7f-af3c-b18bc7612fae","Type":"ContainerDied","Data":"2a73b95fdc634571dcddfbf4ecc715fa28c8a6be824243fcbec1449e5ca8896f"}
Oct 30 19:00:05.418860 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:05.418842    4525 scope.go:122] "RemoveContainer" containerID="2a73b95fdc634571dcddfbf4ecc715fa28c8a6be824243fcbec1449e5ca8896f"
Oct 30 19:00:05.419134 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:00:05.419111    4525 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-85cb41-29889780-4gb6k_odh-ai-gateway-infra(653dd99e-09c9-4e7f-af3c-b18bc7612fae)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-85cb41-29889780-4gb6k" podUID="653dd99e-09c9-4e7f-af3c-b18bc7612fae"
Oct 30 19:00:05.420541 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:05.420520    4525 generic.go:356] "Generic (PLEG): container finished" podID="e3d8e6fc-8e1b-472f-9c8c-aab9b9c06e73" containerID="65e8e744841281e474dcfac0c440e2f3992275088afb110564d345325af18272" exitCode=60
Oct 30 19:00:05.420620 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:05.420599    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39bde3-29889780-jcgz6" event={"ID":"e3d8e6fc-8e1b-472f-9c8c-aab9b9c06e73","Type":"ContainerDied","Data":"65e8e744841281e474dcfac0c440e2f3992275088afb110564d345325af18272"}
Oct 30 19:00:05.420800 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:05.420786    4525 scope.go:122] "RemoveContainer" containerID="65e8e744841281e474dcfac0c440e2f3992275088afb110564d345325af18272"
Oct 30 19:00:05.421020 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:00:05.421003    4525 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-39bde3-29889780-jcgz6_odh-ai-gateway-infra(e3d8e6fc-8e1b-472f-9c8c-aab9b9c06e73)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39bde3-29889780-jcgz6" podUID="e3d8e6fc-8e1b-472f-9c8c-aab9b9c06e73"
Oct 30 19:00:05.422814 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:05.422794    4525 generic.go:356] "Generic (PLEG): container finished" podID="49b2daff-5ea2-4a05-9152-9cc75d4eeba5" containerID="54df706b5019abff3efb1cd492350636daf0b1e6bf9eff1fc4b735d40ca64085" exitCode=60
Oct 30 19:00:05.422959 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:05.422893    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0458df-29889780-dn85w" event={"ID":"49b2daff-5ea2-4a05-9152-9cc75d4eeba5","Type":"ContainerDied","Data":"54df706b5019abff3efb1cd492350636daf0b1e6bf9eff1fc4b735d40ca64085"}
Oct 30 19:00:05.423065 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:05.423038    4525 scope.go:122] "RemoveContainer" containerID="54df706b5019abff3efb1cd492350636daf0b1e6bf9eff1fc4b735d40ca64085"
Oct 30 19:00:05.423313 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:00:05.423290    4525 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-0458df-29889780-dn85w_odh-ai-gateway-infra(49b2daff-5ea2-4a05-9152-9cc75d4eeba5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0458df-29889780-dn85w" podUID="49b2daff-5ea2-4a05-9152-9cc75d4eeba5"
Oct 30 19:00:05.429454 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:05.429439    4525 scope.go:122] "RemoveContainer" containerID="b7284386525d4d5f5fa5ca3458da4e92155f138c695a1915b34e806fb1844401"
Oct 30 19:00:05.445891 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:05.445868    4525 scope.go:122] "RemoveContainer" containerID="8e35bfa7013e1d3e8c9bc8777eb52956b384ce3f9f9a4d8bd1854bb09c91c3ae"
Oct 30 19:00:05.460513 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:05.460490    4525 scope.go:122] "RemoveContainer" containerID="7382121d00ad66ad0075d27bf0ba7d816fe5e29d32e607308339901bb2e839c1"
Oct 30 19:00:05.475693 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:05.475659    4525 scope.go:122] "RemoveContainer" containerID="2b5206bdabd41aae0622710367514bc2ff725aa38419176e748db547b2223701"
Oct 30 19:00:05.664092 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:05.664014    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589"
Oct 30 19:00:05.664515 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:05.664496    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589"
Oct 30 19:00:05.665309 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:05.665290    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589"
Oct 30 19:00:06.435153 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:06.435126    4525 scope.go:122] "RemoveContainer" containerID="b8279571794f2c4abd5a39ceeb2cab1ef699eb3a971ad3b9af249bec8117fade"
Oct 30 19:00:06.435617 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:00:06.435498    4525 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-3d1aa8-29889780-kkcgg_odh-ai-gateway-infra(bf90b413-2e8d-4a11-8e79-639c05ab0f2f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-3d1aa8-29889780-kkcgg" podUID="bf90b413-2e8d-4a11-8e79-639c05ab0f2f"
Oct 30 19:00:06.436908 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:06.436892    4525 scope.go:122] "RemoveContainer" containerID="4f381f743ad681fbcf3ed76f9e829efba7fc1db87ff7c020c35d4d635ccf7d6c"
Oct 30 19:00:06.437182 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:00:06.437163    4525 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-24fceb-29889780-p99bs_odh-ai-gateway-infra(22f70fc7-4efd-423a-b7a1-a0c74be53b60)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-24fceb-29889780-p99bs" podUID="22f70fc7-4efd-423a-b7a1-a0c74be53b60"
Oct 30 19:00:06.438620 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:06.438597    4525 scope.go:122] "RemoveContainer" containerID="2a73b95fdc634571dcddfbf4ecc715fa28c8a6be824243fcbec1449e5ca8896f"
Oct 30 19:00:06.438938 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:00:06.438918    4525 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-85cb41-29889780-4gb6k_odh-ai-gateway-infra(653dd99e-09c9-4e7f-af3c-b18bc7612fae)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-85cb41-29889780-4gb6k" podUID="653dd99e-09c9-4e7f-af3c-b18bc7612fae"
Oct 30 19:00:06.440133 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:06.440111    4525 scope.go:122] "RemoveContainer" containerID="65e8e744841281e474dcfac0c440e2f3992275088afb110564d345325af18272"
Oct 30 19:00:06.440402 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:00:06.440384    4525 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-39bde3-29889780-jcgz6_odh-ai-gateway-infra(e3d8e6fc-8e1b-472f-9c8c-aab9b9c06e73)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39bde3-29889780-jcgz6" podUID="e3d8e6fc-8e1b-472f-9c8c-aab9b9c06e73"
Oct 30 19:00:06.441725 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:06.441704    4525 scope.go:122] "RemoveContainer" containerID="54df706b5019abff3efb1cd492350636daf0b1e6bf9eff1fc4b735d40ca64085"
Oct 30 19:00:06.442006 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:00:06.441986    4525 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-0458df-29889780-dn85w_odh-ai-gateway-infra(49b2daff-5ea2-4a05-9152-9cc75d4eeba5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0458df-29889780-dn85w" podUID="49b2daff-5ea2-4a05-9152-9cc75d4eeba5"
Oct 30 19:00:08.063705 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:08.061763    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5468ba3e-2vtp9"]
Oct 30 19:00:08.141430 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:08.141225    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5468ba3e-2vtp9"]
Oct 30 19:00:09.002193 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:09.002153    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4290dae6-9spnf"]
Oct 30 19:00:09.002837 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:09.002821    4525 cpu_manager.go:414] "containerMap: removing container" podUID="340f330d-01ae-46bd-bc7f-98d8984ef925" containerName="revoke-keys"
Oct 30 19:00:09.002894 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:09.002838    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="340f330d-01ae-46bd-bc7f-98d8984ef925" containerName="revoke-keys"
Oct 30 19:00:09.002894 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:09.002849    4525 cpu_manager.go:414] "containerMap: removing container" podUID="f67c70b1-a75f-47f3-b6e5-fb3ec635707d" containerName="cleanup"
Oct 30 19:00:09.002894 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:09.002854    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f67c70b1-a75f-47f3-b6e5-fb3ec635707d" containerName="cleanup"
Oct 30 19:00:09.002894 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:09.002861    4525 cpu_manager.go:414] "containerMap: removing container" podUID="e2a4a34a-0549-436e-9a18-f73d265b949e" containerName="revoke-keys"
Oct 30 19:00:09.002894 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:09.002866    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2a4a34a-0549-436e-9a18-f73d265b949e" containerName="revoke-keys"
Oct 30 19:00:09.003054 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:09.002950    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="340f330d-01ae-46bd-bc7f-98d8984ef925" containerName="revoke-keys"
Oct 30 19:00:09.003054 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:09.002959    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="f67c70b1-a75f-47f3-b6e5-fb3ec635707d" containerName="cleanup"
Oct 30 19:00:09.003054 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:09.002964    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2a4a34a-0549-436e-9a18-f73d265b949e" containerName="revoke-keys"
Oct 30 19:00:09.006496 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:09.006475    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4290dae6-9spnf"
Oct 30 19:00:09.009914 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:09.009867    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:00:09.016531 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:09.016501    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4290dae6-9spnf"]
Oct 30 19:00:09.034492 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:09.034464    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7334df45-994f-4964-acf0-65e1405b42df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4290dae6-9spnf\" (UID: \"7334df45-994f-4964-acf0-65e1405b42df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4290dae6-9spnf"
Oct 30 19:00:09.034647 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:09.034572    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7334df45-994f-4964-acf0-65e1405b42df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4290dae6-9spnf\" (UID: \"7334df45-994f-4964-acf0-65e1405b42df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4290dae6-9spnf"
Oct 30 19:00:09.034764 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:09.034642    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5wts\" (UniqueName: \"kubernetes.io/projected/7334df45-994f-4964-acf0-65e1405b42df-kube-api-access-x5wts\") pod \"maas-api-revoke-sub-4290dae6-9spnf\" (UID: \"7334df45-994f-4964-acf0-65e1405b42df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4290dae6-9spnf"
Oct 30 19:00:09.136029 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:09.135991    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7334df45-994f-4964-acf0-65e1405b42df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4290dae6-9spnf\" (UID: \"7334df45-994f-4964-acf0-65e1405b42df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4290dae6-9spnf"
Oct 30 19:00:09.136480 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:09.136054    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5wts\" (UniqueName: \"kubernetes.io/projected/7334df45-994f-4964-acf0-65e1405b42df-kube-api-access-x5wts\") pod \"maas-api-revoke-sub-4290dae6-9spnf\" (UID: \"7334df45-994f-4964-acf0-65e1405b42df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4290dae6-9spnf"
Oct 30 19:00:09.136480 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:09.136184    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7334df45-994f-4964-acf0-65e1405b42df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4290dae6-9spnf\" (UID: \"7334df45-994f-4964-acf0-65e1405b42df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4290dae6-9spnf"
Oct 30 19:00:09.136769 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:09.136726    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7334df45-994f-4964-acf0-65e1405b42df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4290dae6-9spnf\" (UID: \"7334df45-994f-4964-acf0-65e1405b42df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4290dae6-9spnf"
Oct 30 19:00:09.138456 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:09.138431    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7334df45-994f-4964-acf0-65e1405b42df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4290dae6-9spnf\" (UID: \"7334df45-994f-4964-acf0-65e1405b42df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4290dae6-9spnf"
Oct 30 19:00:09.144341 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:09.144315    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5wts\" (UniqueName: \"kubernetes.io/projected/7334df45-994f-4964-acf0-65e1405b42df-kube-api-access-x5wts\") pod \"maas-api-revoke-sub-4290dae6-9spnf\" (UID: \"7334df45-994f-4964-acf0-65e1405b42df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4290dae6-9spnf"
Oct 30 19:00:09.333754 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:09.328946    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4290dae6-9spnf"
Oct 30 19:00:09.550025 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:09.549992    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f698200d-7b66-4e30-9908-d622f2aa620c" path="/var/lib/kubelet/pods/f698200d-7b66-4e30-9908-d622f2aa620c/volumes"
Oct 30 19:00:09.736369 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:09.736340    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4290dae6-9spnf"]
Oct 30 19:00:09.736502 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:00:09.736376    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7334df45_994f_4964_acf0_65e1405b42df.slice/crio-7c72eb49facf23ba879ad445c5e398d1abb82c67cdee685a42e4eb610b52ccd2 WatchSource:0}: Error finding container 7c72eb49facf23ba879ad445c5e398d1abb82c67cdee685a42e4eb610b52ccd2: Status 404 returned error can't find the container with id 7c72eb49facf23ba879ad445c5e398d1abb82c67cdee685a42e4eb610b52ccd2
Oct 30 19:00:10.517214 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:10.517177    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4290dae6-9spnf" event={"ID":"7334df45-994f-4964-acf0-65e1405b42df","Type":"ContainerStarted","Data":"8f96b96ded4db1e45ad05caaa6c466c3db7a8706d67eb394790ec79c30e1fab9"}
Oct 30 19:00:10.517616 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:10.517229    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4290dae6-9spnf" event={"ID":"7334df45-994f-4964-acf0-65e1405b42df","Type":"ContainerStarted","Data":"7c72eb49facf23ba879ad445c5e398d1abb82c67cdee685a42e4eb610b52ccd2"}
Oct 30 19:00:10.536195 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:10.536157    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-740adeaf-jpbwv"]
Oct 30 19:00:10.536658 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:10.536615    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4290dae6-9spnf" podStartSLOduration=2.5366023650000002 podStartE2EDuration="2.536602365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:10.534713103 +0000 UTC m=+1433.160383062" watchObservedRunningTime="2026-10-30 19:00:10.536602365 +0000 UTC m=+1433.162272329"
Oct 30 19:00:10.540292 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:10.540272    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-740adeaf-jpbwv"
Oct 30 19:00:10.549945 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:10.549917    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-740adeaf-jpbwv"]
Oct 30 19:00:10.571624 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:10.571593    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59nkq\" (UniqueName: \"kubernetes.io/projected/fbcfc0ae-880d-4c9e-a36a-db1659ce4f94-kube-api-access-59nkq\") pod \"maas-api-revoke-sub-740adeaf-jpbwv\" (UID: \"fbcfc0ae-880d-4c9e-a36a-db1659ce4f94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-740adeaf-jpbwv"
Oct 30 19:00:10.571828 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:10.571643    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbcfc0ae-880d-4c9e-a36a-db1659ce4f94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-740adeaf-jpbwv\" (UID: \"fbcfc0ae-880d-4c9e-a36a-db1659ce4f94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-740adeaf-jpbwv"
Oct 30 19:00:10.571930 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:10.571906    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbcfc0ae-880d-4c9e-a36a-db1659ce4f94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-740adeaf-jpbwv\" (UID: \"fbcfc0ae-880d-4c9e-a36a-db1659ce4f94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-740adeaf-jpbwv"
Oct 30 19:00:10.673510 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:10.673475    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59nkq\" (UniqueName: \"kubernetes.io/projected/fbcfc0ae-880d-4c9e-a36a-db1659ce4f94-kube-api-access-59nkq\") pod \"maas-api-revoke-sub-740adeaf-jpbwv\" (UID: \"fbcfc0ae-880d-4c9e-a36a-db1659ce4f94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-740adeaf-jpbwv"
Oct 30 19:00:10.673802 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:10.673521    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbcfc0ae-880d-4c9e-a36a-db1659ce4f94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-740adeaf-jpbwv\" (UID: \"fbcfc0ae-880d-4c9e-a36a-db1659ce4f94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-740adeaf-jpbwv"
Oct 30 19:00:10.673802 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:10.673597    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbcfc0ae-880d-4c9e-a36a-db1659ce4f94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-740adeaf-jpbwv\" (UID: \"fbcfc0ae-880d-4c9e-a36a-db1659ce4f94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-740adeaf-jpbwv"
Oct 30 19:00:10.674318 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:10.674293    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbcfc0ae-880d-4c9e-a36a-db1659ce4f94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-740adeaf-jpbwv\" (UID: \"fbcfc0ae-880d-4c9e-a36a-db1659ce4f94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-740adeaf-jpbwv"
Oct 30 19:00:10.676586 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:10.676146    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbcfc0ae-880d-4c9e-a36a-db1659ce4f94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-740adeaf-jpbwv\" (UID: \"fbcfc0ae-880d-4c9e-a36a-db1659ce4f94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-740adeaf-jpbwv"
Oct 30 19:00:10.681143 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:10.681118    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59nkq\" (UniqueName: \"kubernetes.io/projected/fbcfc0ae-880d-4c9e-a36a-db1659ce4f94-kube-api-access-59nkq\") pod \"maas-api-revoke-sub-740adeaf-jpbwv\" (UID: \"fbcfc0ae-880d-4c9e-a36a-db1659ce4f94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-740adeaf-jpbwv"
Oct 30 19:00:10.865076 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:10.865040    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-740adeaf-jpbwv"
Oct 30 19:00:11.150597 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:11.150553    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-740adeaf-jpbwv"]
Oct 30 19:00:11.150782 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:00:11.150695    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbcfc0ae_880d_4c9e_a36a_db1659ce4f94.slice/crio-b411fd4027da5a21b4e2d2d5f0073c47116e2e1999f8253c1e8c4c922e4ecbd4 WatchSource:0}: Error finding container b411fd4027da5a21b4e2d2d5f0073c47116e2e1999f8253c1e8c4c922e4ecbd4: Status 404 returned error can't find the container with id b411fd4027da5a21b4e2d2d5f0073c47116e2e1999f8253c1e8c4c922e4ecbd4
Oct 30 19:00:11.536179 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:11.536062    4525 generic.go:356] "Generic (PLEG): container finished" podID="7334df45-994f-4964-acf0-65e1405b42df" containerID="8f96b96ded4db1e45ad05caaa6c466c3db7a8706d67eb394790ec79c30e1fab9" exitCode=0
Oct 30 19:00:11.537674 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:11.536342    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4290dae6-9spnf" event={"ID":"7334df45-994f-4964-acf0-65e1405b42df","Type":"ContainerDied","Data":"8f96b96ded4db1e45ad05caaa6c466c3db7a8706d67eb394790ec79c30e1fab9"}
Oct 30 19:00:11.556286 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:11.556249    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-740adeaf-jpbwv" event={"ID":"fbcfc0ae-880d-4c9e-a36a-db1659ce4f94","Type":"ContainerStarted","Data":"44b3dab77ff842ffa8aefe7a4f8dc1e80159ce8353109298f80738cd5c24a4c2"}
Oct 30 19:00:11.556591 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:11.556549    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-740adeaf-jpbwv" event={"ID":"fbcfc0ae-880d-4c9e-a36a-db1659ce4f94","Type":"ContainerStarted","Data":"b411fd4027da5a21b4e2d2d5f0073c47116e2e1999f8253c1e8c4c922e4ecbd4"}
Oct 30 19:00:11.569444 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:11.569375    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-740adeaf-jpbwv" podStartSLOduration=1.569353846 podStartE2EDuration="1.569353846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:11.567294986 +0000 UTC m=+1434.192964954" watchObservedRunningTime="2026-10-30 19:00:11.569353846 +0000 UTC m=+1434.195023821"
Oct 30 19:00:12.553459 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:12.553373    4525 generic.go:356] "Generic (PLEG): container finished" podID="fbcfc0ae-880d-4c9e-a36a-db1659ce4f94" containerID="44b3dab77ff842ffa8aefe7a4f8dc1e80159ce8353109298f80738cd5c24a4c2" exitCode=0
Oct 30 19:00:12.553961 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:12.553460    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-740adeaf-jpbwv" event={"ID":"fbcfc0ae-880d-4c9e-a36a-db1659ce4f94","Type":"ContainerDied","Data":"44b3dab77ff842ffa8aefe7a4f8dc1e80159ce8353109298f80738cd5c24a4c2"}
Oct 30 19:00:12.721217 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:12.721179    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4290dae6-9spnf"
Oct 30 19:00:12.886438 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:12.886398    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7334df45-994f-4964-acf0-65e1405b42df-kube-api-access-x5wts\" (UniqueName: \"kubernetes.io/projected/7334df45-994f-4964-acf0-65e1405b42df-kube-api-access-x5wts\") pod \"7334df45-994f-4964-acf0-65e1405b42df\" (UID: \"7334df45-994f-4964-acf0-65e1405b42df\") "
Oct 30 19:00:12.886783 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:12.886750    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7334df45-994f-4964-acf0-65e1405b42df-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7334df45-994f-4964-acf0-65e1405b42df-maas-api-xks-ca\") pod \"7334df45-994f-4964-acf0-65e1405b42df\" (UID: \"7334df45-994f-4964-acf0-65e1405b42df\") "
Oct 30 19:00:12.887113 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:12.887095    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7334df45-994f-4964-acf0-65e1405b42df-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7334df45-994f-4964-acf0-65e1405b42df-maas-api-openshift-ca\") pod \"7334df45-994f-4964-acf0-65e1405b42df\" (UID: \"7334df45-994f-4964-acf0-65e1405b42df\") "
Oct 30 19:00:12.914270 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:12.914230    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7334df45-994f-4964-acf0-65e1405b42df-maas-api-openshift-ca" pod "7334df45-994f-4964-acf0-65e1405b42df" (UID: "7334df45-994f-4964-acf0-65e1405b42df"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:00:12.984509 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:12.982830    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7334df45-994f-4964-acf0-65e1405b42df-maas-api-xks-ca" pod "7334df45-994f-4964-acf0-65e1405b42df" (UID: "7334df45-994f-4964-acf0-65e1405b42df"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:00:12.984879 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:12.984834    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7334df45-994f-4964-acf0-65e1405b42df-kube-api-access-x5wts" pod "7334df45-994f-4964-acf0-65e1405b42df" (UID: "7334df45-994f-4964-acf0-65e1405b42df"). InnerVolumeSpecName "kube-api-access-x5wts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:00:13.026732 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:13.026697    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5wts\" (UniqueName: \"kubernetes.io/projected/7334df45-994f-4964-acf0-65e1405b42df-kube-api-access-x5wts\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:13.026732 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:13.026731    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7334df45-994f-4964-acf0-65e1405b42df-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:13.026992 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:13.026743    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7334df45-994f-4964-acf0-65e1405b42df-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:13.073487 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:13.073454    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-h74bt"]
Oct 30 19:00:13.074259 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:13.073666    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-h74bt" podUID="050bdbd2-11b3-41bc-92e1-e816bb463c9b" containerName="istio-proxy" containerID="cri-o://2266671aecaa481be24266f033803e4b8ad1b41e5f3d9cc884fe123c7ccad05e" gracePeriod=30
Oct 30 19:00:13.566593 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:13.566560    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4290dae6-9spnf" event={"ID":"7334df45-994f-4964-acf0-65e1405b42df","Type":"ContainerDied","Data":"7c72eb49facf23ba879ad445c5e398d1abb82c67cdee685a42e4eb610b52ccd2"}
Oct 30 19:00:13.567051 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:13.566595    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4290dae6-9spnf"
Oct 30 19:00:13.567051 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:13.566618    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c72eb49facf23ba879ad445c5e398d1abb82c67cdee685a42e4eb610b52ccd2"
Oct 30 19:00:14.183928 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:14.183897    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-740adeaf-jpbwv"
Oct 30 19:00:14.276928 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:14.275843    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbcfc0ae-880d-4c9e-a36a-db1659ce4f94-kube-api-access-59nkq\" (UniqueName: \"kubernetes.io/projected/fbcfc0ae-880d-4c9e-a36a-db1659ce4f94-kube-api-access-59nkq\") pod \"fbcfc0ae-880d-4c9e-a36a-db1659ce4f94\" (UID: \"fbcfc0ae-880d-4c9e-a36a-db1659ce4f94\") "
Oct 30 19:00:14.353702 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:14.347294    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbcfc0ae-880d-4c9e-a36a-db1659ce4f94-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbcfc0ae-880d-4c9e-a36a-db1659ce4f94-maas-api-openshift-ca\") pod \"fbcfc0ae-880d-4c9e-a36a-db1659ce4f94\" (UID: \"fbcfc0ae-880d-4c9e-a36a-db1659ce4f94\") "
Oct 30 19:00:14.353702 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:14.347456    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbcfc0ae-880d-4c9e-a36a-db1659ce4f94-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbcfc0ae-880d-4c9e-a36a-db1659ce4f94-maas-api-xks-ca\") pod \"fbcfc0ae-880d-4c9e-a36a-db1659ce4f94\" (UID: \"fbcfc0ae-880d-4c9e-a36a-db1659ce4f94\") "
Oct 30 19:00:14.353702 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:14.348875    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbcfc0ae-880d-4c9e-a36a-db1659ce4f94-maas-api-openshift-ca" pod "fbcfc0ae-880d-4c9e-a36a-db1659ce4f94" (UID: "fbcfc0ae-880d-4c9e-a36a-db1659ce4f94"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:00:14.379225 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:14.375382    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbcfc0ae-880d-4c9e-a36a-db1659ce4f94-maas-api-xks-ca" pod "fbcfc0ae-880d-4c9e-a36a-db1659ce4f94" (UID: "fbcfc0ae-880d-4c9e-a36a-db1659ce4f94"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:00:14.379225 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:14.375459    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbcfc0ae-880d-4c9e-a36a-db1659ce4f94-kube-api-access-59nkq" pod "fbcfc0ae-880d-4c9e-a36a-db1659ce4f94" (UID: "fbcfc0ae-880d-4c9e-a36a-db1659ce4f94"). InnerVolumeSpecName "kube-api-access-59nkq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:00:14.450024 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:14.449942    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59nkq\" (UniqueName: \"kubernetes.io/projected/fbcfc0ae-880d-4c9e-a36a-db1659ce4f94-kube-api-access-59nkq\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:14.450024 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:14.449981    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbcfc0ae-880d-4c9e-a36a-db1659ce4f94-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:14.450024 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:14.449997    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbcfc0ae-880d-4c9e-a36a-db1659ce4f94-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:14.577062 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:14.576974    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-740adeaf-jpbwv" event={"ID":"fbcfc0ae-880d-4c9e-a36a-db1659ce4f94","Type":"ContainerDied","Data":"b411fd4027da5a21b4e2d2d5f0073c47116e2e1999f8253c1e8c4c922e4ecbd4"}
Oct 30 19:00:14.577062 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:14.577038    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b411fd4027da5a21b4e2d2d5f0073c47116e2e1999f8253c1e8c4c922e4ecbd4"
Oct 30 19:00:14.577460 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:14.577066    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-740adeaf-jpbwv"
Oct 30 19:00:15.537706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:15.537640    4525 scope.go:122] "RemoveContainer" containerID="65e8e744841281e474dcfac0c440e2f3992275088afb110564d345325af18272"
Oct 30 19:00:16.146224 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:16.146141    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-3d1aa8-29889780-kkcgg"]
Oct 30 19:00:16.324704 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:16.320171    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-3d1aa8-29889780-kkcgg"
Oct 30 19:00:16.549473 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:16.496186    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf90b413-2e8d-4a11-8e79-639c05ab0f2f-kube-api-access-6dchr\" (UniqueName: \"kubernetes.io/projected/bf90b413-2e8d-4a11-8e79-639c05ab0f2f-kube-api-access-6dchr\") pod \"bf90b413-2e8d-4a11-8e79-639c05ab0f2f\" (UID: \"bf90b413-2e8d-4a11-8e79-639c05ab0f2f\") "
Oct 30 19:00:16.549473 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:16.496261    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf90b413-2e8d-4a11-8e79-639c05ab0f2f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf90b413-2e8d-4a11-8e79-639c05ab0f2f-maas-api-service-ca\") pod \"bf90b413-2e8d-4a11-8e79-639c05ab0f2f\" (UID: \"bf90b413-2e8d-4a11-8e79-639c05ab0f2f\") "
Oct 30 19:00:16.549473 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:16.533239    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf90b413-2e8d-4a11-8e79-639c05ab0f2f-kube-api-access-6dchr" pod "bf90b413-2e8d-4a11-8e79-639c05ab0f2f" (UID: "bf90b413-2e8d-4a11-8e79-639c05ab0f2f"). InnerVolumeSpecName "kube-api-access-6dchr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:00:16.549473 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:16.533299    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3d1aa8-2x2zv"]
Oct 30 19:00:16.549473 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:16.538787    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3d1aa8-2x2zv"]
Oct 30 19:00:16.549473 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:16.545519    4525 scope.go:122] "RemoveContainer" containerID="4f381f743ad681fbcf3ed76f9e829efba7fc1db87ff7c020c35d4d635ccf7d6c"
Oct 30 19:00:16.549473 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:16.548641    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf90b413-2e8d-4a11-8e79-639c05ab0f2f-maas-api-service-ca" pod "bf90b413-2e8d-4a11-8e79-639c05ab0f2f" (UID: "bf90b413-2e8d-4a11-8e79-639c05ab0f2f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:00:16.595636 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:16.595616    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-3d1aa8-29889780-kkcgg"
Oct 30 19:00:16.595785 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:16.595645    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-3d1aa8-29889780-kkcgg" event={"ID":"bf90b413-2e8d-4a11-8e79-639c05ab0f2f","Type":"ContainerDied","Data":"fa3163911c823757c971fb4b2eb358dd9295f791ecfd6df66cd17d23f6dd5624"}
Oct 30 19:00:16.595785 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:16.595719    4525 scope.go:122] "RemoveContainer" containerID="b8279571794f2c4abd5a39ceeb2cab1ef699eb3a971ad3b9af249bec8117fade"
Oct 30 19:00:16.597958 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:16.597927    4525 generic.go:356] "Generic (PLEG): container finished" podID="e3d8e6fc-8e1b-472f-9c8c-aab9b9c06e73" containerID="4c6c77f77bbd6bf2d0aad7005943c32282d2ca44554a0eda366f2a8c7d3468f0" exitCode=60
Oct 30 19:00:16.598091 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:16.597999    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39bde3-29889780-jcgz6" event={"ID":"e3d8e6fc-8e1b-472f-9c8c-aab9b9c06e73","Type":"ContainerDied","Data":"4c6c77f77bbd6bf2d0aad7005943c32282d2ca44554a0eda366f2a8c7d3468f0"}
Oct 30 19:00:16.598298 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:16.598278    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dchr\" (UniqueName: \"kubernetes.io/projected/bf90b413-2e8d-4a11-8e79-639c05ab0f2f-kube-api-access-6dchr\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:16.598370 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:16.598301    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf90b413-2e8d-4a11-8e79-639c05ab0f2f-maas-api-service-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:16.598370 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:16.598310    4525 scope.go:122] "RemoveContainer" containerID="4c6c77f77bbd6bf2d0aad7005943c32282d2ca44554a0eda366f2a8c7d3468f0"
Oct 30 19:00:16.598561 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:00:16.598544    4525 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-39bde3-29889780-jcgz6_odh-ai-gateway-infra(e3d8e6fc-8e1b-472f-9c8c-aab9b9c06e73)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39bde3-29889780-jcgz6" podUID="e3d8e6fc-8e1b-472f-9c8c-aab9b9c06e73"
Oct 30 19:00:16.647640 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:16.647614    4525 scope.go:122] "RemoveContainer" containerID="65e8e744841281e474dcfac0c440e2f3992275088afb110564d345325af18272"
Oct 30 19:00:16.673694 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:16.673655    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-3d1aa8-29889780-kkcgg"]
Oct 30 19:00:16.687173 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:16.687142    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-3d1aa8-29889780-kkcgg"]
Oct 30 19:00:17.398392 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:00:17.398345    4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d49l" containerName="extractor"
Oct 30 19:00:17.561523 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:17.561492    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf90b413-2e8d-4a11-8e79-639c05ab0f2f" path="/var/lib/kubelet/pods/bf90b413-2e8d-4a11-8e79-639c05ab0f2f/volumes"
Oct 30 19:00:17.562153 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:17.562127    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2a4a34a-0549-436e-9a18-f73d265b949e" path="/var/lib/kubelet/pods/e2a4a34a-0549-436e-9a18-f73d265b949e/volumes"
Oct 30 19:00:17.609137 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:17.609110    4525 generic.go:356] "Generic (PLEG): container finished" podID="22f70fc7-4efd-423a-b7a1-a0c74be53b60" containerID="db9b5ae08e1b713077d2bae4a14cdb0390dc5bddec54ff78239500aa6b9abd6e" exitCode=60
Oct 30 19:00:17.609288 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:17.609183    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-24fceb-29889780-p99bs" event={"ID":"22f70fc7-4efd-423a-b7a1-a0c74be53b60","Type":"ContainerDied","Data":"db9b5ae08e1b713077d2bae4a14cdb0390dc5bddec54ff78239500aa6b9abd6e"}
Oct 30 19:00:17.609288 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:17.609235    4525 scope.go:122] "RemoveContainer" containerID="4f381f743ad681fbcf3ed76f9e829efba7fc1db87ff7c020c35d4d635ccf7d6c"
Oct 30 19:00:17.609716 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:17.609669    4525 scope.go:122] "RemoveContainer" containerID="db9b5ae08e1b713077d2bae4a14cdb0390dc5bddec54ff78239500aa6b9abd6e"
Oct 30 19:00:17.610058 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:00:17.610032    4525 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-24fceb-29889780-p99bs_odh-ai-gateway-infra(22f70fc7-4efd-423a-b7a1-a0c74be53b60)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-24fceb-29889780-p99bs" podUID="22f70fc7-4efd-423a-b7a1-a0c74be53b60"
Oct 30 19:00:17.629070 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:17.628966    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39bde3-29889780-jcgz6"]
Oct 30 19:00:17.807625 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:17.807597    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39bde3-29889780-jcgz6"
Oct 30 19:00:17.924467 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:17.924433    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3d8e6fc-8e1b-472f-9c8c-aab9b9c06e73-kube-api-access-jqxqg\" (UniqueName: \"kubernetes.io/projected/e3d8e6fc-8e1b-472f-9c8c-aab9b9c06e73-kube-api-access-jqxqg\") pod \"e3d8e6fc-8e1b-472f-9c8c-aab9b9c06e73\" (UID: \"e3d8e6fc-8e1b-472f-9c8c-aab9b9c06e73\") "
Oct 30 19:00:17.924658 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:17.924621    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3d8e6fc-8e1b-472f-9c8c-aab9b9c06e73-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3d8e6fc-8e1b-472f-9c8c-aab9b9c06e73-maas-api-service-ca\") pod \"e3d8e6fc-8e1b-472f-9c8c-aab9b9c06e73\" (UID: \"e3d8e6fc-8e1b-472f-9c8c-aab9b9c06e73\") "
Oct 30 19:00:17.924989 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:17.924965    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3d8e6fc-8e1b-472f-9c8c-aab9b9c06e73-maas-api-service-ca" pod "e3d8e6fc-8e1b-472f-9c8c-aab9b9c06e73" (UID: "e3d8e6fc-8e1b-472f-9c8c-aab9b9c06e73"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:00:17.925382 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:17.925366    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3d8e6fc-8e1b-472f-9c8c-aab9b9c06e73-maas-api-service-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:17.926725 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:17.926672    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3d8e6fc-8e1b-472f-9c8c-aab9b9c06e73-kube-api-access-jqxqg" pod "e3d8e6fc-8e1b-472f-9c8c-aab9b9c06e73" (UID: "e3d8e6fc-8e1b-472f-9c8c-aab9b9c06e73"). InnerVolumeSpecName "kube-api-access-jqxqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:00:18.027638 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:18.027561    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqxqg\" (UniqueName: \"kubernetes.io/projected/e3d8e6fc-8e1b-472f-9c8c-aab9b9c06e73-kube-api-access-jqxqg\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:18.400229 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:18.400205    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-h74bt"
Oct 30 19:00:18.539249 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:18.539204    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/050bdbd2-11b3-41bc-92e1-e816bb463c9b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/050bdbd2-11b3-41bc-92e1-e816bb463c9b-istiod-ca-cert\") pod \"050bdbd2-11b3-41bc-92e1-e816bb463c9b\" (UID: \"050bdbd2-11b3-41bc-92e1-e816bb463c9b\") "
Oct 30 19:00:18.539444 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:18.539371    4525 scope.go:122] "RemoveContainer" containerID="2a73b95fdc634571dcddfbf4ecc715fa28c8a6be824243fcbec1449e5ca8896f"
Oct 30 19:00:18.539581 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:18.539539    4525 scope.go:122] "RemoveContainer" containerID="54df706b5019abff3efb1cd492350636daf0b1e6bf9eff1fc4b735d40ca64085"
Oct 30 19:00:18.539581 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:18.539564    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/050bdbd2-11b3-41bc-92e1-e816bb463c9b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/050bdbd2-11b3-41bc-92e1-e816bb463c9b-workload-certs\") pod \"050bdbd2-11b3-41bc-92e1-e816bb463c9b\" (UID: \"050bdbd2-11b3-41bc-92e1-e816bb463c9b\") "
Oct 30 19:00:18.539581 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:18.539594    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/050bdbd2-11b3-41bc-92e1-e816bb463c9b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/050bdbd2-11b3-41bc-92e1-e816bb463c9b-workload-socket\") pod \"050bdbd2-11b3-41bc-92e1-e816bb463c9b\" (UID: \"050bdbd2-11b3-41bc-92e1-e816bb463c9b\") "
Oct 30 19:00:18.539844 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:18.539661    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/050bdbd2-11b3-41bc-92e1-e816bb463c9b-istio-token\" (UniqueName: \"kubernetes.io/projected/050bdbd2-11b3-41bc-92e1-e816bb463c9b-istio-token\") pod \"050bdbd2-11b3-41bc-92e1-e816bb463c9b\" (UID: \"050bdbd2-11b3-41bc-92e1-e816bb463c9b\") "
Oct 30 19:00:18.539844 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:18.539664    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/050bdbd2-11b3-41bc-92e1-e816bb463c9b-istiod-ca-cert" pod "050bdbd2-11b3-41bc-92e1-e816bb463c9b" (UID: "050bdbd2-11b3-41bc-92e1-e816bb463c9b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:00:18.539844 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:18.539756    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/050bdbd2-11b3-41bc-92e1-e816bb463c9b-kube-api-access-m4d8p\" (UniqueName: \"kubernetes.io/projected/050bdbd2-11b3-41bc-92e1-e816bb463c9b-kube-api-access-m4d8p\") pod \"050bdbd2-11b3-41bc-92e1-e816bb463c9b\" (UID: \"050bdbd2-11b3-41bc-92e1-e816bb463c9b\") "
Oct 30 19:00:18.539844 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:18.539828    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/050bdbd2-11b3-41bc-92e1-e816bb463c9b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/050bdbd2-11b3-41bc-92e1-e816bb463c9b-istio-podinfo\") pod \"050bdbd2-11b3-41bc-92e1-e816bb463c9b\" (UID: \"050bdbd2-11b3-41bc-92e1-e816bb463c9b\") "
Oct 30 19:00:18.540033 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:18.539933    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/050bdbd2-11b3-41bc-92e1-e816bb463c9b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/050bdbd2-11b3-41bc-92e1-e816bb463c9b-istio-envoy\") pod \"050bdbd2-11b3-41bc-92e1-e816bb463c9b\" (UID: \"050bdbd2-11b3-41bc-92e1-e816bb463c9b\") "
Oct 30 19:00:18.540033 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:18.539967    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/050bdbd2-11b3-41bc-92e1-e816bb463c9b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/050bdbd2-11b3-41bc-92e1-e816bb463c9b-istio-data\") pod \"050bdbd2-11b3-41bc-92e1-e816bb463c9b\" (UID: \"050bdbd2-11b3-41bc-92e1-e816bb463c9b\") "
Oct 30 19:00:18.540033 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:18.540003    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/050bdbd2-11b3-41bc-92e1-e816bb463c9b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/050bdbd2-11b3-41bc-92e1-e816bb463c9b-credential-socket\") pod \"050bdbd2-11b3-41bc-92e1-e816bb463c9b\" (UID: \"050bdbd2-11b3-41bc-92e1-e816bb463c9b\") "
Oct 30 19:00:18.540178 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:18.540026    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/050bdbd2-11b3-41bc-92e1-e816bb463c9b-workload-socket" pod "050bdbd2-11b3-41bc-92e1-e816bb463c9b" (UID: "050bdbd2-11b3-41bc-92e1-e816bb463c9b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:00:18.540461 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:18.540376    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/050bdbd2-11b3-41bc-92e1-e816bb463c9b-istio-data" pod "050bdbd2-11b3-41bc-92e1-e816bb463c9b" (UID: "050bdbd2-11b3-41bc-92e1-e816bb463c9b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:00:18.540461 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:18.540401    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/050bdbd2-11b3-41bc-92e1-e816bb463c9b-workload-certs" pod "050bdbd2-11b3-41bc-92e1-e816bb463c9b" (UID: "050bdbd2-11b3-41bc-92e1-e816bb463c9b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:00:18.540602 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:18.540579    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/050bdbd2-11b3-41bc-92e1-e816bb463c9b-credential-socket" pod "050bdbd2-11b3-41bc-92e1-e816bb463c9b" (UID: "050bdbd2-11b3-41bc-92e1-e816bb463c9b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:00:18.540751 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:18.540716    4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/050bdbd2-11b3-41bc-92e1-e816bb463c9b-istio-data\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:18.540751 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:18.540738    4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/050bdbd2-11b3-41bc-92e1-e816bb463c9b-credential-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:18.540899 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:18.540755    4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/050bdbd2-11b3-41bc-92e1-e816bb463c9b-istiod-ca-cert\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:18.540899 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:18.540770    4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/050bdbd2-11b3-41bc-92e1-e816bb463c9b-workload-certs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:18.540899 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:18.540784    4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/050bdbd2-11b3-41bc-92e1-e816bb463c9b-workload-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:18.542960 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:18.542936    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/050bdbd2-11b3-41bc-92e1-e816bb463c9b-kube-api-access-m4d8p" pod "050bdbd2-11b3-41bc-92e1-e816bb463c9b" (UID: "050bdbd2-11b3-41bc-92e1-e816bb463c9b"). InnerVolumeSpecName "kube-api-access-m4d8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:00:18.543132 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:18.543112    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/050bdbd2-11b3-41bc-92e1-e816bb463c9b-istio-envoy" pod "050bdbd2-11b3-41bc-92e1-e816bb463c9b" (UID: "050bdbd2-11b3-41bc-92e1-e816bb463c9b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:00:18.543801 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:18.543775    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/050bdbd2-11b3-41bc-92e1-e816bb463c9b-istio-podinfo" pod "050bdbd2-11b3-41bc-92e1-e816bb463c9b" (UID: "050bdbd2-11b3-41bc-92e1-e816bb463c9b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:00:18.544083 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:18.544014    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/050bdbd2-11b3-41bc-92e1-e816bb463c9b-istio-token" pod "050bdbd2-11b3-41bc-92e1-e816bb463c9b" (UID: "050bdbd2-11b3-41bc-92e1-e816bb463c9b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:00:18.621915 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:18.621890    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39bde3-29889780-jcgz6"
Oct 30 19:00:18.622023 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:18.621927    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39bde3-29889780-jcgz6" event={"ID":"e3d8e6fc-8e1b-472f-9c8c-aab9b9c06e73","Type":"ContainerDied","Data":"7e404bc2ca8e10db427008ebd0fcde068d69230d0ecdbcf1e34d5a65375abbf1"}
Oct 30 19:00:18.622023 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:18.621991    4525 scope.go:122] "RemoveContainer" containerID="4c6c77f77bbd6bf2d0aad7005943c32282d2ca44554a0eda366f2a8c7d3468f0"
Oct 30 19:00:18.623632 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:18.623602    4525 generic.go:356] "Generic (PLEG): container finished" podID="050bdbd2-11b3-41bc-92e1-e816bb463c9b" containerID="2266671aecaa481be24266f033803e4b8ad1b41e5f3d9cc884fe123c7ccad05e" exitCode=0
Oct 30 19:00:18.623753 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:18.623638    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-h74bt" event={"ID":"050bdbd2-11b3-41bc-92e1-e816bb463c9b","Type":"ContainerDied","Data":"2266671aecaa481be24266f033803e4b8ad1b41e5f3d9cc884fe123c7ccad05e"}
Oct 30 19:00:18.623753 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:18.623654    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-h74bt"
Oct 30 19:00:18.623753 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:18.623703    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-h74bt" event={"ID":"050bdbd2-11b3-41bc-92e1-e816bb463c9b","Type":"ContainerDied","Data":"079d21fe64a6a2a209495724ea1322285a7b378be75da4ca44e7fb0d2da6d0a1"}
Oct 30 19:00:18.639347 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:18.639321    4525 scope.go:122] "RemoveContainer" containerID="2266671aecaa481be24266f033803e4b8ad1b41e5f3d9cc884fe123c7ccad05e"
Oct 30 19:00:18.642463 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:18.642288    4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/050bdbd2-11b3-41bc-92e1-e816bb463c9b-istio-token\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:18.642463 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:18.642316    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4d8p\" (UniqueName: \"kubernetes.io/projected/050bdbd2-11b3-41bc-92e1-e816bb463c9b-kube-api-access-m4d8p\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:18.642463 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:18.642332    4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/050bdbd2-11b3-41bc-92e1-e816bb463c9b-istio-podinfo\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:18.642463 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:18.642345    4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/050bdbd2-11b3-41bc-92e1-e816bb463c9b-istio-envoy\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:18.645907 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:18.645883    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-24fceb-29889780-p99bs"]
Oct 30 19:00:18.656440 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:18.656423    4525 scope.go:122] "RemoveContainer" containerID="2266671aecaa481be24266f033803e4b8ad1b41e5f3d9cc884fe123c7ccad05e"
Oct 30 19:00:18.656829 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:00:18.656803    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2266671aecaa481be24266f033803e4b8ad1b41e5f3d9cc884fe123c7ccad05e\": container with ID starting with 2266671aecaa481be24266f033803e4b8ad1b41e5f3d9cc884fe123c7ccad05e not found: ID does not exist" containerID="2266671aecaa481be24266f033803e4b8ad1b41e5f3d9cc884fe123c7ccad05e"
Oct 30 19:00:18.656930 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:18.656853    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2266671aecaa481be24266f033803e4b8ad1b41e5f3d9cc884fe123c7ccad05e"} err="failed to get container status \"2266671aecaa481be24266f033803e4b8ad1b41e5f3d9cc884fe123c7ccad05e\": rpc error: code = NotFound desc = could not find container \"2266671aecaa481be24266f033803e4b8ad1b41e5f3d9cc884fe123c7ccad05e\": container with ID starting with 2266671aecaa481be24266f033803e4b8ad1b41e5f3d9cc884fe123c7ccad05e not found: ID does not exist"
Oct 30 19:00:18.674107 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:18.674075    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-h74bt"]
Oct 30 19:00:18.681730 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:18.681326    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-h74bt"]
Oct 30 19:00:18.686159 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:18.686137    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39bde3-29889780-jcgz6"]
Oct 30 19:00:18.689770 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:18.689746    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39bde3-29889780-jcgz6"]
Oct 30 19:00:18.835054 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:18.835031    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-24fceb-29889780-p99bs"
Oct 30 19:00:18.946303 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:18.946275    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22f70fc7-4efd-423a-b7a1-a0c74be53b60-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22f70fc7-4efd-423a-b7a1-a0c74be53b60-maas-api-service-ca\") pod \"22f70fc7-4efd-423a-b7a1-a0c74be53b60\" (UID: \"22f70fc7-4efd-423a-b7a1-a0c74be53b60\") "
Oct 30 19:00:18.946519 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:18.946505    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22f70fc7-4efd-423a-b7a1-a0c74be53b60-kube-api-access-qg4mq\" (UniqueName: \"kubernetes.io/projected/22f70fc7-4efd-423a-b7a1-a0c74be53b60-kube-api-access-qg4mq\") pod \"22f70fc7-4efd-423a-b7a1-a0c74be53b60\" (UID: \"22f70fc7-4efd-423a-b7a1-a0c74be53b60\") "
Oct 30 19:00:18.946611 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:18.946589    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22f70fc7-4efd-423a-b7a1-a0c74be53b60-maas-api-service-ca" pod "22f70fc7-4efd-423a-b7a1-a0c74be53b60" (UID: "22f70fc7-4efd-423a-b7a1-a0c74be53b60"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:00:18.946946 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:18.946928    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22f70fc7-4efd-423a-b7a1-a0c74be53b60-maas-api-service-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:18.948641 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:18.948614    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22f70fc7-4efd-423a-b7a1-a0c74be53b60-kube-api-access-qg4mq" pod "22f70fc7-4efd-423a-b7a1-a0c74be53b60" (UID: "22f70fc7-4efd-423a-b7a1-a0c74be53b60"). InnerVolumeSpecName "kube-api-access-qg4mq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:00:19.049041 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:19.048996    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qg4mq\" (UniqueName: \"kubernetes.io/projected/22f70fc7-4efd-423a-b7a1-a0c74be53b60-kube-api-access-qg4mq\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:19.053340 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:19.053311    4525 scope.go:122] "RemoveContainer" containerID="89e3979ab1d020d4501e7cd380793ecb40bf4f31ca726a95aa4e5abebf0ae4a9"
Oct 30 19:00:19.066777 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:19.066753    4525 scope.go:122] "RemoveContainer" containerID="026934c3e4e871026d479d332ab691137acc2366bc2df0567ca437b92ba2d49e"
Oct 30 19:00:19.076892 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:19.076872    4525 scope.go:122] "RemoveContainer" containerID="cb0d6a8227a3dcc2de5fbfd178e3b0591867b38ac8b591e2e26fe99c27011b24"
Oct 30 19:00:19.553695 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:19.553651    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="050bdbd2-11b3-41bc-92e1-e816bb463c9b" path="/var/lib/kubelet/pods/050bdbd2-11b3-41bc-92e1-e816bb463c9b/volumes"
Oct 30 19:00:19.554167 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:19.554151    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3d8e6fc-8e1b-472f-9c8c-aab9b9c06e73" path="/var/lib/kubelet/pods/e3d8e6fc-8e1b-472f-9c8c-aab9b9c06e73/volumes"
Oct 30 19:00:19.635638 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:19.635608    4525 generic.go:356] "Generic (PLEG): container finished" podID="49b2daff-5ea2-4a05-9152-9cc75d4eeba5" containerID="e633af4fc65844eccfa10a89b8d8b6fdb7673d35a1563d414c8744e7bd7e899e" exitCode=60
Oct 30 19:00:19.635794 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:19.635665    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0458df-29889780-dn85w" event={"ID":"49b2daff-5ea2-4a05-9152-9cc75d4eeba5","Type":"ContainerDied","Data":"e633af4fc65844eccfa10a89b8d8b6fdb7673d35a1563d414c8744e7bd7e899e"}
Oct 30 19:00:19.635794 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:19.635750    4525 scope.go:122] "RemoveContainer" containerID="54df706b5019abff3efb1cd492350636daf0b1e6bf9eff1fc4b735d40ca64085"
Oct 30 19:00:19.636164 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:19.636145    4525 scope.go:122] "RemoveContainer" containerID="e633af4fc65844eccfa10a89b8d8b6fdb7673d35a1563d414c8744e7bd7e899e"
Oct 30 19:00:19.636532 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:00:19.636508    4525 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-0458df-29889780-dn85w_odh-ai-gateway-infra(49b2daff-5ea2-4a05-9152-9cc75d4eeba5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0458df-29889780-dn85w" podUID="49b2daff-5ea2-4a05-9152-9cc75d4eeba5"
Oct 30 19:00:19.637452 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:19.637435    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-24fceb-29889780-p99bs"
Oct 30 19:00:19.637560 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:19.637469    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-24fceb-29889780-p99bs" event={"ID":"22f70fc7-4efd-423a-b7a1-a0c74be53b60","Type":"ContainerDied","Data":"1e9d1f301a74818d4f1f8a1d7a7aa56d27b50d1948dde4b943f19ba3ecde39d5"}
Oct 30 19:00:19.639733 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:19.639667    4525 generic.go:356] "Generic (PLEG): container finished" podID="653dd99e-09c9-4e7f-af3c-b18bc7612fae" containerID="64d8fb004ac566be45808fafd1d70d26ff6d273a6fd606893ba685665929302e" exitCode=60
Oct 30 19:00:19.639733 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:19.639711    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-85cb41-29889780-4gb6k" event={"ID":"653dd99e-09c9-4e7f-af3c-b18bc7612fae","Type":"ContainerDied","Data":"64d8fb004ac566be45808fafd1d70d26ff6d273a6fd606893ba685665929302e"}
Oct 30 19:00:19.640134 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:19.640119    4525 scope.go:122] "RemoveContainer" containerID="64d8fb004ac566be45808fafd1d70d26ff6d273a6fd606893ba685665929302e"
Oct 30 19:00:19.640468 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:00:19.640445    4525 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-85cb41-29889780-4gb6k_odh-ai-gateway-infra(653dd99e-09c9-4e7f-af3c-b18bc7612fae)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-85cb41-29889780-4gb6k" podUID="653dd99e-09c9-4e7f-af3c-b18bc7612fae"
Oct 30 19:00:19.656249 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:19.656218    4525 scope.go:122] "RemoveContainer" containerID="db9b5ae08e1b713077d2bae4a14cdb0390dc5bddec54ff78239500aa6b9abd6e"
Oct 30 19:00:19.667071 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:19.667047    4525 scope.go:122] "RemoveContainer" containerID="2a73b95fdc634571dcddfbf4ecc715fa28c8a6be824243fcbec1449e5ca8896f"
Oct 30 19:00:19.694399 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:19.694368    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-24fceb-29889780-p99bs"]
Oct 30 19:00:19.698868 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:19.698812    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-24fceb-29889780-p99bs"]
Oct 30 19:00:20.671124 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:20.671090    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0458df-29889780-dn85w"]
Oct 30 19:00:20.692937 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:20.692900    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-85cb41-29889780-4gb6k"]
Oct 30 19:00:20.853973 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:20.853941    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0458df-29889780-dn85w"
Oct 30 19:00:20.888720 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:20.888670    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-85cb41-29889780-4gb6k"
Oct 30 19:00:20.977113 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:20.977025    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49b2daff-5ea2-4a05-9152-9cc75d4eeba5-kube-api-access-947gk\" (UniqueName: \"kubernetes.io/projected/49b2daff-5ea2-4a05-9152-9cc75d4eeba5-kube-api-access-947gk\") pod \"49b2daff-5ea2-4a05-9152-9cc75d4eeba5\" (UID: \"49b2daff-5ea2-4a05-9152-9cc75d4eeba5\") "
Oct 30 19:00:20.977423 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:20.977169    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49b2daff-5ea2-4a05-9152-9cc75d4eeba5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49b2daff-5ea2-4a05-9152-9cc75d4eeba5-maas-api-service-ca\") pod \"49b2daff-5ea2-4a05-9152-9cc75d4eeba5\" (UID: \"49b2daff-5ea2-4a05-9152-9cc75d4eeba5\") "
Oct 30 19:00:20.977923 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:20.977755    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49b2daff-5ea2-4a05-9152-9cc75d4eeba5-maas-api-service-ca" pod "49b2daff-5ea2-4a05-9152-9cc75d4eeba5" (UID: "49b2daff-5ea2-4a05-9152-9cc75d4eeba5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:00:20.978046 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:20.978027    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49b2daff-5ea2-4a05-9152-9cc75d4eeba5-maas-api-service-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:20.979842 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:20.979819    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49b2daff-5ea2-4a05-9152-9cc75d4eeba5-kube-api-access-947gk" pod "49b2daff-5ea2-4a05-9152-9cc75d4eeba5" (UID: "49b2daff-5ea2-4a05-9152-9cc75d4eeba5"). InnerVolumeSpecName "kube-api-access-947gk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:00:21.079215 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:21.079191    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/653dd99e-09c9-4e7f-af3c-b18bc7612fae-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/653dd99e-09c9-4e7f-af3c-b18bc7612fae-maas-api-service-ca\") pod \"653dd99e-09c9-4e7f-af3c-b18bc7612fae\" (UID: \"653dd99e-09c9-4e7f-af3c-b18bc7612fae\") "
Oct 30 19:00:21.079360 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:21.079341    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/653dd99e-09c9-4e7f-af3c-b18bc7612fae-kube-api-access-plnlp\" (UniqueName: \"kubernetes.io/projected/653dd99e-09c9-4e7f-af3c-b18bc7612fae-kube-api-access-plnlp\") pod \"653dd99e-09c9-4e7f-af3c-b18bc7612fae\" (UID: \"653dd99e-09c9-4e7f-af3c-b18bc7612fae\") "
Oct 30 19:00:21.079575 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:21.079549    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/653dd99e-09c9-4e7f-af3c-b18bc7612fae-maas-api-service-ca" pod "653dd99e-09c9-4e7f-af3c-b18bc7612fae" (UID: "653dd99e-09c9-4e7f-af3c-b18bc7612fae"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:00:21.079867 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:21.079849    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/653dd99e-09c9-4e7f-af3c-b18bc7612fae-maas-api-service-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:21.079937 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:21.079871    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-947gk\" (UniqueName: \"kubernetes.io/projected/49b2daff-5ea2-4a05-9152-9cc75d4eeba5-kube-api-access-947gk\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:21.081483 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:21.081465    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/653dd99e-09c9-4e7f-af3c-b18bc7612fae-kube-api-access-plnlp" pod "653dd99e-09c9-4e7f-af3c-b18bc7612fae" (UID: "653dd99e-09c9-4e7f-af3c-b18bc7612fae"). InnerVolumeSpecName "kube-api-access-plnlp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:00:21.181599 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:21.181570    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plnlp\" (UniqueName: \"kubernetes.io/projected/653dd99e-09c9-4e7f-af3c-b18bc7612fae-kube-api-access-plnlp\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:21.545148 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:21.545108    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22f70fc7-4efd-423a-b7a1-a0c74be53b60" path="/var/lib/kubelet/pods/22f70fc7-4efd-423a-b7a1-a0c74be53b60/volumes"
Oct 30 19:00:21.664479 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:21.664444    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-85cb41-29889780-4gb6k"
Oct 30 19:00:21.664671 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:21.664472    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-85cb41-29889780-4gb6k" event={"ID":"653dd99e-09c9-4e7f-af3c-b18bc7612fae","Type":"ContainerDied","Data":"a77101a6aaf76a9ad9096f331e5aad5441dfa23560141758efbe467e36ce3d85"}
Oct 30 19:00:21.664671 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:21.664543    4525 scope.go:122] "RemoveContainer" containerID="64d8fb004ac566be45808fafd1d70d26ff6d273a6fd606893ba685665929302e"
Oct 30 19:00:21.666173 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:21.666145    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0458df-29889780-dn85w"
Oct 30 19:00:21.666334 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:21.666201    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0458df-29889780-dn85w" event={"ID":"49b2daff-5ea2-4a05-9152-9cc75d4eeba5","Type":"ContainerDied","Data":"3ec8f6b62ffa34579741d05635ab0a591c5fee4afdc1374141e304fbfb56854b"}
Oct 30 19:00:21.676264 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:21.676075    4525 scope.go:122] "RemoveContainer" containerID="e633af4fc65844eccfa10a89b8d8b6fdb7673d35a1563d414c8744e7bd7e899e"
Oct 30 19:00:21.689672 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:21.689644    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0458df-29889780-dn85w"]
Oct 30 19:00:21.695067 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:21.695046    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0458df-29889780-dn85w"]
Oct 30 19:00:21.704313 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:21.704292    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-85cb41-29889780-4gb6k"]
Oct 30 19:00:21.712771 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:21.712750    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-85cb41-29889780-4gb6k"]
Oct 30 19:00:21.814620 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:21.814546    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5ec3983-sb9z7"]
Oct 30 19:00:21.815226 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:21.815206    4525 cpu_manager.go:414] "containerMap: removing container" podUID="e3d8e6fc-8e1b-472f-9c8c-aab9b9c06e73" containerName="cleanup"
Oct 30 19:00:21.815324 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:21.815227    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3d8e6fc-8e1b-472f-9c8c-aab9b9c06e73" containerName="cleanup"
Oct 30 19:00:21.815324 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:21.815242    4525 cpu_manager.go:414] "containerMap: removing container" podUID="fbcfc0ae-880d-4c9e-a36a-db1659ce4f94" containerName="revoke-keys"
Oct 30 19:00:21.815324 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:21.815249    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbcfc0ae-880d-4c9e-a36a-db1659ce4f94" containerName="revoke-keys"
Oct 30 19:00:21.815324 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:21.815262    4525 cpu_manager.go:414] "containerMap: removing container" podUID="49b2daff-5ea2-4a05-9152-9cc75d4eeba5" containerName="cleanup"
Oct 30 19:00:21.815324 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:21.815270    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49b2daff-5ea2-4a05-9152-9cc75d4eeba5" containerName="cleanup"
Oct 30 19:00:21.815324 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:21.815284    4525 cpu_manager.go:414] "containerMap: removing container" podUID="653dd99e-09c9-4e7f-af3c-b18bc7612fae" containerName="cleanup"
Oct 30 19:00:21.815324 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:21.815290    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="653dd99e-09c9-4e7f-af3c-b18bc7612fae" containerName="cleanup"
Oct 30 19:00:21.815324 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:21.815303    4525 cpu_manager.go:414] "containerMap: removing container" podUID="49b2daff-5ea2-4a05-9152-9cc75d4eeba5" containerName="cleanup"
Oct 30 19:00:21.815324 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:21.815309    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49b2daff-5ea2-4a05-9152-9cc75d4eeba5" containerName="cleanup"
Oct 30 19:00:21.815324 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:21.815320    4525 cpu_manager.go:414] "containerMap: removing container" podUID="050bdbd2-11b3-41bc-92e1-e816bb463c9b" containerName="istio-proxy"
Oct 30 19:00:21.815324 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:21.815326    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="050bdbd2-11b3-41bc-92e1-e816bb463c9b" containerName="istio-proxy"
Oct 30 19:00:21.815917 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:21.815338    4525 cpu_manager.go:414] "containerMap: removing container" podUID="653dd99e-09c9-4e7f-af3c-b18bc7612fae" containerName="cleanup"
Oct 30 19:00:21.815917 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:21.815345    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="653dd99e-09c9-4e7f-af3c-b18bc7612fae" containerName="cleanup"
Oct 30 19:00:21.815917 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:21.815362    4525 cpu_manager.go:414] "containerMap: removing container" podUID="653dd99e-09c9-4e7f-af3c-b18bc7612fae" containerName="cleanup"
Oct 30 19:00:21.815917 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:21.815369    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="653dd99e-09c9-4e7f-af3c-b18bc7612fae" containerName="cleanup"
Oct 30 19:00:21.815917 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:21.815379    4525 cpu_manager.go:414] "containerMap: removing container" podUID="22f70fc7-4efd-423a-b7a1-a0c74be53b60" containerName="cleanup"
Oct 30 19:00:21.815917 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:21.815387    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22f70fc7-4efd-423a-b7a1-a0c74be53b60" containerName="cleanup"
Oct 30 19:00:21.815917 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:21.815404    4525 cpu_manager.go:414] "containerMap: removing container" podUID="7334df45-994f-4964-acf0-65e1405b42df" containerName="revoke-keys"
Oct 30 19:00:21.815917 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:21.815410    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7334df45-994f-4964-acf0-65e1405b42df" containerName="revoke-keys"
Oct 30 19:00:21.815917 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:21.815421    4525 cpu_manager.go:414] "containerMap: removing container" podUID="22f70fc7-4efd-423a-b7a1-a0c74be53b60" containerName="cleanup"
Oct 30 19:00:21.815917 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:21.815427    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22f70fc7-4efd-423a-b7a1-a0c74be53b60" containerName="cleanup"
Oct 30 19:00:21.815917 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:21.815448    4525 cpu_manager.go:414] "containerMap: removing container" podUID="bf90b413-2e8d-4a11-8e79-639c05ab0f2f" containerName="cleanup"
Oct 30 19:00:21.815917 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:21.815454    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf90b413-2e8d-4a11-8e79-639c05ab0f2f" containerName="cleanup"
Oct 30 19:00:21.815917 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:21.815466    4525 cpu_manager.go:414] "containerMap: removing container" podUID="bf90b413-2e8d-4a11-8e79-639c05ab0f2f" containerName="cleanup"
Oct 30 19:00:21.815917 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:21.815473    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf90b413-2e8d-4a11-8e79-639c05ab0f2f" containerName="cleanup"
Oct 30 19:00:21.815917 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:21.815485    4525 cpu_manager.go:414] "containerMap: removing container" podUID="e3d8e6fc-8e1b-472f-9c8c-aab9b9c06e73" containerName="cleanup"
Oct 30 19:00:21.815917 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:21.815491    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3d8e6fc-8e1b-472f-9c8c-aab9b9c06e73" containerName="cleanup"
Oct 30 19:00:21.815917 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:21.815643    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="7334df45-994f-4964-acf0-65e1405b42df" containerName="revoke-keys"
Oct 30 19:00:21.815917 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:21.815659    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3d8e6fc-8e1b-472f-9c8c-aab9b9c06e73" containerName="cleanup"
Oct 30 19:00:21.815917 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:21.815668    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="22f70fc7-4efd-423a-b7a1-a0c74be53b60" containerName="cleanup"
Oct 30 19:00:21.815917 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:21.815697    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="49b2daff-5ea2-4a05-9152-9cc75d4eeba5" containerName="cleanup"
Oct 30 19:00:21.815917 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:21.815709    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf90b413-2e8d-4a11-8e79-639c05ab0f2f" containerName="cleanup"
Oct 30 19:00:21.815917 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:21.815718    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf90b413-2e8d-4a11-8e79-639c05ab0f2f" containerName="cleanup"
Oct 30 19:00:21.815917 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:21.815726    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3d8e6fc-8e1b-472f-9c8c-aab9b9c06e73" containerName="cleanup"
Oct 30 19:00:21.815917 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:21.815735    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="22f70fc7-4efd-423a-b7a1-a0c74be53b60" containerName="cleanup"
Oct 30 19:00:21.815917 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:21.815749    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3d8e6fc-8e1b-472f-9c8c-aab9b9c06e73" containerName="cleanup"
Oct 30 19:00:21.815917 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:21.815757    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbcfc0ae-880d-4c9e-a36a-db1659ce4f94" containerName="revoke-keys"
Oct 30 19:00:21.815917 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:21.815766    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="653dd99e-09c9-4e7f-af3c-b18bc7612fae" containerName="cleanup"
Oct 30 19:00:21.815917 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:21.815775    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="49b2daff-5ea2-4a05-9152-9cc75d4eeba5" containerName="cleanup"
Oct 30 19:00:21.815917 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:21.815782    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="050bdbd2-11b3-41bc-92e1-e816bb463c9b" containerName="istio-proxy"
Oct 30 19:00:21.815917 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:21.815789    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="653dd99e-09c9-4e7f-af3c-b18bc7612fae" containerName="cleanup"
Oct 30 19:00:21.815917 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:21.815798    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="653dd99e-09c9-4e7f-af3c-b18bc7612fae" containerName="cleanup"
Oct 30 19:00:21.821239 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:21.821095    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5ec3983-sb9z7"
Oct 30 19:00:21.825168 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:21.825141    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:00:21.825276 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:21.825149    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94hcn\""
Oct 30 19:00:21.827105 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:21.827082    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5ec3983-sb9z7"]
Oct 30 19:00:21.994132 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:21.994101    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f98b875e-ea67-402e-a050-67e67f925367-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5ec3983-sb9z7\" (UID: \"f98b875e-ea67-402e-a050-67e67f925367\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5ec3983-sb9z7"
Oct 30 19:00:21.994335 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:21.994255    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vdk7\" (UniqueName: \"kubernetes.io/projected/f98b875e-ea67-402e-a050-67e67f925367-kube-api-access-6vdk7\") pod \"maas-api-revoke-sub-a5ec3983-sb9z7\" (UID: \"f98b875e-ea67-402e-a050-67e67f925367\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5ec3983-sb9z7"
Oct 30 19:00:21.994335 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:21.994319    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f98b875e-ea67-402e-a050-67e67f925367-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5ec3983-sb9z7\" (UID: \"f98b875e-ea67-402e-a050-67e67f925367\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5ec3983-sb9z7"
Oct 30 19:00:22.095429 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:22.095343    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f98b875e-ea67-402e-a050-67e67f925367-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5ec3983-sb9z7\" (UID: \"f98b875e-ea67-402e-a050-67e67f925367\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5ec3983-sb9z7"
Oct 30 19:00:22.095429 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:22.095428    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vdk7\" (UniqueName: \"kubernetes.io/projected/f98b875e-ea67-402e-a050-67e67f925367-kube-api-access-6vdk7\") pod \"maas-api-revoke-sub-a5ec3983-sb9z7\" (UID: \"f98b875e-ea67-402e-a050-67e67f925367\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5ec3983-sb9z7"
Oct 30 19:00:22.095641 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:22.095448    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f98b875e-ea67-402e-a050-67e67f925367-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5ec3983-sb9z7\" (UID: \"f98b875e-ea67-402e-a050-67e67f925367\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5ec3983-sb9z7"
Oct 30 19:00:22.095986 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:22.095963    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f98b875e-ea67-402e-a050-67e67f925367-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5ec3983-sb9z7\" (UID: \"f98b875e-ea67-402e-a050-67e67f925367\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5ec3983-sb9z7"
Oct 30 19:00:22.098099 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:22.098063    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f98b875e-ea67-402e-a050-67e67f925367-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5ec3983-sb9z7\" (UID: \"f98b875e-ea67-402e-a050-67e67f925367\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5ec3983-sb9z7"
Oct 30 19:00:22.103792 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:22.103771    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vdk7\" (UniqueName: \"kubernetes.io/projected/f98b875e-ea67-402e-a050-67e67f925367-kube-api-access-6vdk7\") pod \"maas-api-revoke-sub-a5ec3983-sb9z7\" (UID: \"f98b875e-ea67-402e-a050-67e67f925367\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5ec3983-sb9z7"
Oct 30 19:00:22.152276 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:22.152251    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5ec3983-sb9z7"
Oct 30 19:00:22.496086 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:22.494816    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5ec3983-sb9z7"]
Oct 30 19:00:22.789228 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:22.789141    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5ec3983-sb9z7" event={"ID":"f98b875e-ea67-402e-a050-67e67f925367","Type":"ContainerStarted","Data":"770fe3285278380cfe90421cc15b4eef21f371966c5ecfc0e6a9cf472229ae65"}
Oct 30 19:00:23.544876 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:23.544845    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49b2daff-5ea2-4a05-9152-9cc75d4eeba5" path="/var/lib/kubelet/pods/49b2daff-5ea2-4a05-9152-9cc75d4eeba5/volumes"
Oct 30 19:00:23.545290 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:23.545276    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="653dd99e-09c9-4e7f-af3c-b18bc7612fae" path="/var/lib/kubelet/pods/653dd99e-09c9-4e7f-af3c-b18bc7612fae/volumes"
Oct 30 19:00:23.582327 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:23.582295    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fcad69e-99x4r"]
Oct 30 19:00:23.582902 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:23.582885    4525 cpu_manager.go:414] "containerMap: removing container" podUID="e3d8e6fc-8e1b-472f-9c8c-aab9b9c06e73" containerName="cleanup"
Oct 30 19:00:23.582902 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:23.582902    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3d8e6fc-8e1b-472f-9c8c-aab9b9c06e73" containerName="cleanup"
Oct 30 19:00:23.583031 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:23.582917    4525 cpu_manager.go:414] "containerMap: removing container" podUID="49b2daff-5ea2-4a05-9152-9cc75d4eeba5" containerName="cleanup"
Oct 30 19:00:23.583031 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:23.582922    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49b2daff-5ea2-4a05-9152-9cc75d4eeba5" containerName="cleanup"
Oct 30 19:00:23.583031 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:23.582941    4525 cpu_manager.go:414] "containerMap: removing container" podUID="22f70fc7-4efd-423a-b7a1-a0c74be53b60" containerName="cleanup"
Oct 30 19:00:23.583031 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:23.582946    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22f70fc7-4efd-423a-b7a1-a0c74be53b60" containerName="cleanup"
Oct 30 19:00:23.583031 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:23.583026    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="49b2daff-5ea2-4a05-9152-9cc75d4eeba5" containerName="cleanup"
Oct 30 19:00:23.583183 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:23.583035    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="22f70fc7-4efd-423a-b7a1-a0c74be53b60" containerName="cleanup"
Oct 30 19:00:23.587880 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:23.587848    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fcad69e-99x4r"
Oct 30 19:00:23.600454 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:23.595036    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fcad69e-99x4r"]
Oct 30 19:00:23.638188 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:23.638147    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/128e7ce4-6afa-400b-9585-d5f112f2e102-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fcad69e-99x4r\" (UID: \"128e7ce4-6afa-400b-9585-d5f112f2e102\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fcad69e-99x4r"
Oct 30 19:00:23.638349 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:23.638229    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbqrx\" (UniqueName: \"kubernetes.io/projected/128e7ce4-6afa-400b-9585-d5f112f2e102-kube-api-access-sbqrx\") pod \"maas-api-revoke-sub-5fcad69e-99x4r\" (UID: \"128e7ce4-6afa-400b-9585-d5f112f2e102\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fcad69e-99x4r"
Oct 30 19:00:23.638349 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:23.638342    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/128e7ce4-6afa-400b-9585-d5f112f2e102-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fcad69e-99x4r\" (UID: \"128e7ce4-6afa-400b-9585-d5f112f2e102\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fcad69e-99x4r"
Oct 30 19:00:23.739525 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:23.739493    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/128e7ce4-6afa-400b-9585-d5f112f2e102-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fcad69e-99x4r\" (UID: \"128e7ce4-6afa-400b-9585-d5f112f2e102\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fcad69e-99x4r"
Oct 30 19:00:23.739701 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:23.739638    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/128e7ce4-6afa-400b-9585-d5f112f2e102-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fcad69e-99x4r\" (UID: \"128e7ce4-6afa-400b-9585-d5f112f2e102\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fcad69e-99x4r"
Oct 30 19:00:23.739701 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:23.739695    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbqrx\" (UniqueName: \"kubernetes.io/projected/128e7ce4-6afa-400b-9585-d5f112f2e102-kube-api-access-sbqrx\") pod \"maas-api-revoke-sub-5fcad69e-99x4r\" (UID: \"128e7ce4-6afa-400b-9585-d5f112f2e102\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fcad69e-99x4r"
Oct 30 19:00:23.740214 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:23.740186    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/128e7ce4-6afa-400b-9585-d5f112f2e102-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fcad69e-99x4r\" (UID: \"128e7ce4-6afa-400b-9585-d5f112f2e102\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fcad69e-99x4r"
Oct 30 19:00:23.742092 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:23.742067    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/128e7ce4-6afa-400b-9585-d5f112f2e102-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fcad69e-99x4r\" (UID: \"128e7ce4-6afa-400b-9585-d5f112f2e102\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fcad69e-99x4r"
Oct 30 19:00:23.747382 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:23.747362    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbqrx\" (UniqueName: \"kubernetes.io/projected/128e7ce4-6afa-400b-9585-d5f112f2e102-kube-api-access-sbqrx\") pod \"maas-api-revoke-sub-5fcad69e-99x4r\" (UID: \"128e7ce4-6afa-400b-9585-d5f112f2e102\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fcad69e-99x4r"
Oct 30 19:00:23.800739 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:23.800655    4525 generic.go:356] "Generic (PLEG): container finished" podID="f98b875e-ea67-402e-a050-67e67f925367" containerID="76e15e6ca54c0aab216dd95559367fe1e5e4bbf8cbcfc604ab2a64b4b9dac0eb" exitCode=0
Oct 30 19:00:23.801120 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:23.800744    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5ec3983-sb9z7" event={"ID":"f98b875e-ea67-402e-a050-67e67f925367","Type":"ContainerDied","Data":"76e15e6ca54c0aab216dd95559367fe1e5e4bbf8cbcfc604ab2a64b4b9dac0eb"}
Oct 30 19:00:23.965461 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:23.965420    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fcad69e-99x4r"
Oct 30 19:00:24.288928 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:24.288890    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fcad69e-99x4r"]
Oct 30 19:00:24.289767 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:00:24.289740    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod128e7ce4_6afa_400b_9585_d5f112f2e102.slice/crio-29bdcd89a3e2b14f3451d87b94ed28aebbb5683be57db2eb5af16f04d973bea2 WatchSource:0}: Error finding container 29bdcd89a3e2b14f3451d87b94ed28aebbb5683be57db2eb5af16f04d973bea2: Status 404 returned error can't find the container with id 29bdcd89a3e2b14f3451d87b94ed28aebbb5683be57db2eb5af16f04d973bea2
Oct 30 19:00:24.396937 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:24.396886    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589"]
Oct 30 19:00:24.397180 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:24.397147    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589" podUID="af747a06-3fa4-4c7f-a5a3-7e33a6d68746" containerName="istio-proxy" containerID="cri-o://7d8b22f61a5ce3938d4ce4476451de0f79008caa8503e488122f8256130a4bfd" gracePeriod=30
Oct 30 19:00:24.813026 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:24.812988    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fcad69e-99x4r" event={"ID":"128e7ce4-6afa-400b-9585-d5f112f2e102","Type":"ContainerStarted","Data":"5dd62a8df86f68ce7a012b10637fc9189959475bcb961817382d06a4312cceff"}
Oct 30 19:00:24.813491 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:24.813052    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fcad69e-99x4r" event={"ID":"128e7ce4-6afa-400b-9585-d5f112f2e102","Type":"ContainerStarted","Data":"29bdcd89a3e2b14f3451d87b94ed28aebbb5683be57db2eb5af16f04d973bea2"}
Oct 30 19:00:25.012521 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:25.012493    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5ec3983-sb9z7"
Oct 30 19:00:25.029583 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:25.029531    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fcad69e-99x4r" podStartSLOduration=2.029517402 podStartE2EDuration="2.029517402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:24.838575354 +0000 UTC m=+1447.464245376" watchObservedRunningTime="2026-10-30 19:00:25.029517402 +0000 UTC m=+1447.655187365"
Oct 30 19:00:25.083130 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:25.083090    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f98b875e-ea67-402e-a050-67e67f925367-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f98b875e-ea67-402e-a050-67e67f925367-maas-api-xks-ca\") pod \"f98b875e-ea67-402e-a050-67e67f925367\" (UID: \"f98b875e-ea67-402e-a050-67e67f925367\") "
Oct 30 19:00:25.083375 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:25.083359    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f98b875e-ea67-402e-a050-67e67f925367-kube-api-access-6vdk7\" (UniqueName: \"kubernetes.io/projected/f98b875e-ea67-402e-a050-67e67f925367-kube-api-access-6vdk7\") pod \"f98b875e-ea67-402e-a050-67e67f925367\" (UID: \"f98b875e-ea67-402e-a050-67e67f925367\") "
Oct 30 19:00:25.083478 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:25.083466    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f98b875e-ea67-402e-a050-67e67f925367-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f98b875e-ea67-402e-a050-67e67f925367-maas-api-openshift-ca\") pod \"f98b875e-ea67-402e-a050-67e67f925367\" (UID: \"f98b875e-ea67-402e-a050-67e67f925367\") "
Oct 30 19:00:25.083904 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:25.083840    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f98b875e-ea67-402e-a050-67e67f925367-maas-api-openshift-ca" pod "f98b875e-ea67-402e-a050-67e67f925367" (UID: "f98b875e-ea67-402e-a050-67e67f925367"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:00:25.084046 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:25.084021    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f98b875e-ea67-402e-a050-67e67f925367-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:25.085307 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:25.085281    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f98b875e-ea67-402e-a050-67e67f925367-maas-api-xks-ca" pod "f98b875e-ea67-402e-a050-67e67f925367" (UID: "f98b875e-ea67-402e-a050-67e67f925367"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:00:25.085424 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:25.085409    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f98b875e-ea67-402e-a050-67e67f925367-kube-api-access-6vdk7" pod "f98b875e-ea67-402e-a050-67e67f925367" (UID: "f98b875e-ea67-402e-a050-67e67f925367"). InnerVolumeSpecName "kube-api-access-6vdk7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:00:25.185656 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:25.185619    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vdk7\" (UniqueName: \"kubernetes.io/projected/f98b875e-ea67-402e-a050-67e67f925367-kube-api-access-6vdk7\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:25.185656 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:25.185658    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f98b875e-ea67-402e-a050-67e67f925367-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:25.822043 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:25.821966    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5ec3983-sb9z7"
Oct 30 19:00:25.822043 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:25.821984    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5ec3983-sb9z7" event={"ID":"f98b875e-ea67-402e-a050-67e67f925367","Type":"ContainerDied","Data":"770fe3285278380cfe90421cc15b4eef21f371966c5ecfc0e6a9cf472229ae65"}
Oct 30 19:00:25.822043 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:25.822045    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="770fe3285278380cfe90421cc15b4eef21f371966c5ecfc0e6a9cf472229ae65"
Oct 30 19:00:25.823856 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:25.823829    4525 generic.go:356] "Generic (PLEG): container finished" podID="128e7ce4-6afa-400b-9585-d5f112f2e102" containerID="5dd62a8df86f68ce7a012b10637fc9189959475bcb961817382d06a4312cceff" exitCode=0
Oct 30 19:00:25.823959 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:25.823855    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fcad69e-99x4r" event={"ID":"128e7ce4-6afa-400b-9585-d5f112f2e102","Type":"ContainerDied","Data":"5dd62a8df86f68ce7a012b10637fc9189959475bcb961817382d06a4312cceff"}
Oct 30 19:00:27.058780 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:27.058749    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fcad69e-99x4r"
Oct 30 19:00:27.123581 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:27.123507    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/128e7ce4-6afa-400b-9585-d5f112f2e102-kube-api-access-sbqrx\" (UniqueName: \"kubernetes.io/projected/128e7ce4-6afa-400b-9585-d5f112f2e102-kube-api-access-sbqrx\") pod \"128e7ce4-6afa-400b-9585-d5f112f2e102\" (UID: \"128e7ce4-6afa-400b-9585-d5f112f2e102\") "
Oct 30 19:00:27.123802 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:27.123767    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/128e7ce4-6afa-400b-9585-d5f112f2e102-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/128e7ce4-6afa-400b-9585-d5f112f2e102-maas-api-xks-ca\") pod \"128e7ce4-6afa-400b-9585-d5f112f2e102\" (UID: \"128e7ce4-6afa-400b-9585-d5f112f2e102\") "
Oct 30 19:00:27.123879 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:27.123804    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/128e7ce4-6afa-400b-9585-d5f112f2e102-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/128e7ce4-6afa-400b-9585-d5f112f2e102-maas-api-openshift-ca\") pod \"128e7ce4-6afa-400b-9585-d5f112f2e102\" (UID: \"128e7ce4-6afa-400b-9585-d5f112f2e102\") "
Oct 30 19:00:27.124652 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:27.124623    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/128e7ce4-6afa-400b-9585-d5f112f2e102-maas-api-openshift-ca" pod "128e7ce4-6afa-400b-9585-d5f112f2e102" (UID: "128e7ce4-6afa-400b-9585-d5f112f2e102"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:00:27.127319 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:27.127295    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/128e7ce4-6afa-400b-9585-d5f112f2e102-maas-api-xks-ca" pod "128e7ce4-6afa-400b-9585-d5f112f2e102" (UID: "128e7ce4-6afa-400b-9585-d5f112f2e102"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:00:27.129040 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:27.129015    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/128e7ce4-6afa-400b-9585-d5f112f2e102-kube-api-access-sbqrx" pod "128e7ce4-6afa-400b-9585-d5f112f2e102" (UID: "128e7ce4-6afa-400b-9585-d5f112f2e102"). InnerVolumeSpecName "kube-api-access-sbqrx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:00:27.225816 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:27.225774    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbqrx\" (UniqueName: \"kubernetes.io/projected/128e7ce4-6afa-400b-9585-d5f112f2e102-kube-api-access-sbqrx\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:27.225816 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:27.225812    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/128e7ce4-6afa-400b-9585-d5f112f2e102-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:27.225816 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:27.225824    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/128e7ce4-6afa-400b-9585-d5f112f2e102-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:27.300554 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:27.300522    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f9b7116-4ntcq"]
Oct 30 19:00:27.301117 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:27.301102    4525 cpu_manager.go:414] "containerMap: removing container" podUID="f98b875e-ea67-402e-a050-67e67f925367" containerName="revoke-keys"
Oct 30 19:00:27.301183 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:27.301118    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f98b875e-ea67-402e-a050-67e67f925367" containerName="revoke-keys"
Oct 30 19:00:27.301183 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:27.301138    4525 cpu_manager.go:414] "containerMap: removing container" podUID="128e7ce4-6afa-400b-9585-d5f112f2e102" containerName="revoke-keys"
Oct 30 19:00:27.301183 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:27.301144    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="128e7ce4-6afa-400b-9585-d5f112f2e102" containerName="revoke-keys"
Oct 30 19:00:27.301277 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:27.301233    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="f98b875e-ea67-402e-a050-67e67f925367" containerName="revoke-keys"
Oct 30 19:00:27.301277 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:27.301245    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="128e7ce4-6afa-400b-9585-d5f112f2e102" containerName="revoke-keys"
Oct 30 19:00:27.305509 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:27.305487    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f9b7116-4ntcq"
Oct 30 19:00:27.313429 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:27.313401    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f9b7116-4ntcq"]
Oct 30 19:00:27.332737 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:27.327246    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03ce656d-ce81-4db9-b834-3d3ea7be50b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f9b7116-4ntcq\" (UID: \"03ce656d-ce81-4db9-b834-3d3ea7be50b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f9b7116-4ntcq"
Oct 30 19:00:27.332737 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:27.327557    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03ce656d-ce81-4db9-b834-3d3ea7be50b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f9b7116-4ntcq\" (UID: \"03ce656d-ce81-4db9-b834-3d3ea7be50b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f9b7116-4ntcq"
Oct 30 19:00:27.332737 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:27.327833    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2qlc\" (UniqueName: \"kubernetes.io/projected/03ce656d-ce81-4db9-b834-3d3ea7be50b2-kube-api-access-f2qlc\") pod \"maas-api-revoke-sub-9f9b7116-4ntcq\" (UID: \"03ce656d-ce81-4db9-b834-3d3ea7be50b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f9b7116-4ntcq"
Oct 30 19:00:27.429545 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:27.429473    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2qlc\" (UniqueName: \"kubernetes.io/projected/03ce656d-ce81-4db9-b834-3d3ea7be50b2-kube-api-access-f2qlc\") pod \"maas-api-revoke-sub-9f9b7116-4ntcq\" (UID: \"03ce656d-ce81-4db9-b834-3d3ea7be50b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f9b7116-4ntcq"
Oct 30 19:00:27.429545 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:27.429534    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03ce656d-ce81-4db9-b834-3d3ea7be50b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f9b7116-4ntcq\" (UID: \"03ce656d-ce81-4db9-b834-3d3ea7be50b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f9b7116-4ntcq"
Oct 30 19:00:27.429815 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:27.429625    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03ce656d-ce81-4db9-b834-3d3ea7be50b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f9b7116-4ntcq\" (UID: \"03ce656d-ce81-4db9-b834-3d3ea7be50b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f9b7116-4ntcq"
Oct 30 19:00:27.430393 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:27.430365    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03ce656d-ce81-4db9-b834-3d3ea7be50b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f9b7116-4ntcq\" (UID: \"03ce656d-ce81-4db9-b834-3d3ea7be50b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f9b7116-4ntcq"
Oct 30 19:00:27.432446 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:27.432424    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03ce656d-ce81-4db9-b834-3d3ea7be50b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f9b7116-4ntcq\" (UID: \"03ce656d-ce81-4db9-b834-3d3ea7be50b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f9b7116-4ntcq"
Oct 30 19:00:27.437226 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:27.437203    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2qlc\" (UniqueName: \"kubernetes.io/projected/03ce656d-ce81-4db9-b834-3d3ea7be50b2-kube-api-access-f2qlc\") pod \"maas-api-revoke-sub-9f9b7116-4ntcq\" (UID: \"03ce656d-ce81-4db9-b834-3d3ea7be50b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f9b7116-4ntcq"
Oct 30 19:00:27.631696 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:27.631652    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f9b7116-4ntcq"
Oct 30 19:00:27.877393 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:27.877356    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fcad69e-99x4r" event={"ID":"128e7ce4-6afa-400b-9585-d5f112f2e102","Type":"ContainerDied","Data":"29bdcd89a3e2b14f3451d87b94ed28aebbb5683be57db2eb5af16f04d973bea2"}
Oct 30 19:00:27.877393 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:27.877390    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fcad69e-99x4r"
Oct 30 19:00:27.877665 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:27.877418    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29bdcd89a3e2b14f3451d87b94ed28aebbb5683be57db2eb5af16f04d973bea2"
Oct 30 19:00:27.913433 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:00:27.913397    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03ce656d_ce81_4db9_b834_3d3ea7be50b2.slice/crio-10ceb890655715e1999e6b14ad3081c92db7fdfacf9fb37e2a5a55223de46446 WatchSource:0}: Error finding container 10ceb890655715e1999e6b14ad3081c92db7fdfacf9fb37e2a5a55223de46446: Status 404 returned error can't find the container with id 10ceb890655715e1999e6b14ad3081c92db7fdfacf9fb37e2a5a55223de46446
Oct 30 19:00:27.914988 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:27.914960    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f9b7116-4ntcq"]
Oct 30 19:00:28.887237 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:28.887203    4525 generic.go:356] "Generic (PLEG): container finished" podID="03ce656d-ce81-4db9-b834-3d3ea7be50b2" containerID="cabb95626b839ca4d6c8cfd21fed0970f54fa7b02d355f56745abc3070fd5c63" exitCode=0
Oct 30 19:00:28.887651 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:28.887283    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f9b7116-4ntcq" event={"ID":"03ce656d-ce81-4db9-b834-3d3ea7be50b2","Type":"ContainerDied","Data":"cabb95626b839ca4d6c8cfd21fed0970f54fa7b02d355f56745abc3070fd5c63"}
Oct 30 19:00:28.887651 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:28.887335    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f9b7116-4ntcq" event={"ID":"03ce656d-ce81-4db9-b834-3d3ea7be50b2","Type":"ContainerStarted","Data":"10ceb890655715e1999e6b14ad3081c92db7fdfacf9fb37e2a5a55223de46446"}
Oct 30 19:00:29.681409 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:29.681349    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589"
Oct 30 19:00:29.787299 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:29.787262    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af747a06-3fa4-4c7f-a5a3-7e33a6d68746-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af747a06-3fa4-4c7f-a5a3-7e33a6d68746-workload-socket\") pod \"af747a06-3fa4-4c7f-a5a3-7e33a6d68746\" (UID: \"af747a06-3fa4-4c7f-a5a3-7e33a6d68746\") "
Oct 30 19:00:29.787492 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:29.787311    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af747a06-3fa4-4c7f-a5a3-7e33a6d68746-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af747a06-3fa4-4c7f-a5a3-7e33a6d68746-istio-data\") pod \"af747a06-3fa4-4c7f-a5a3-7e33a6d68746\" (UID: \"af747a06-3fa4-4c7f-a5a3-7e33a6d68746\") "
Oct 30 19:00:29.787492 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:29.787349    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af747a06-3fa4-4c7f-a5a3-7e33a6d68746-istio-token\" (UniqueName: \"kubernetes.io/projected/af747a06-3fa4-4c7f-a5a3-7e33a6d68746-istio-token\") pod \"af747a06-3fa4-4c7f-a5a3-7e33a6d68746\" (UID: \"af747a06-3fa4-4c7f-a5a3-7e33a6d68746\") "
Oct 30 19:00:29.787492 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:29.787395    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af747a06-3fa4-4c7f-a5a3-7e33a6d68746-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af747a06-3fa4-4c7f-a5a3-7e33a6d68746-workload-certs\") pod \"af747a06-3fa4-4c7f-a5a3-7e33a6d68746\" (UID: \"af747a06-3fa4-4c7f-a5a3-7e33a6d68746\") "
Oct 30 19:00:29.787492 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:29.787430    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af747a06-3fa4-4c7f-a5a3-7e33a6d68746-kube-api-access-7b9ln\" (UniqueName: \"kubernetes.io/projected/af747a06-3fa4-4c7f-a5a3-7e33a6d68746-kube-api-access-7b9ln\") pod \"af747a06-3fa4-4c7f-a5a3-7e33a6d68746\" (UID: \"af747a06-3fa4-4c7f-a5a3-7e33a6d68746\") "
Oct 30 19:00:29.787492 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:29.787483    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/af747a06-3fa4-4c7f-a5a3-7e33a6d68746-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af747a06-3fa4-4c7f-a5a3-7e33a6d68746-istio-podinfo\") pod \"af747a06-3fa4-4c7f-a5a3-7e33a6d68746\" (UID: \"af747a06-3fa4-4c7f-a5a3-7e33a6d68746\") "
Oct 30 19:00:29.787803 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:29.787522    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af747a06-3fa4-4c7f-a5a3-7e33a6d68746-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af747a06-3fa4-4c7f-a5a3-7e33a6d68746-istio-envoy\") pod \"af747a06-3fa4-4c7f-a5a3-7e33a6d68746\" (UID: \"af747a06-3fa4-4c7f-a5a3-7e33a6d68746\") "
Oct 30 19:00:29.787803 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:29.787534    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af747a06-3fa4-4c7f-a5a3-7e33a6d68746-workload-socket" pod "af747a06-3fa4-4c7f-a5a3-7e33a6d68746" (UID: "af747a06-3fa4-4c7f-a5a3-7e33a6d68746"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:00:29.787803 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:29.787587    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af747a06-3fa4-4c7f-a5a3-7e33a6d68746-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af747a06-3fa4-4c7f-a5a3-7e33a6d68746-credential-socket\") pod \"af747a06-3fa4-4c7f-a5a3-7e33a6d68746\" (UID: \"af747a06-3fa4-4c7f-a5a3-7e33a6d68746\") "
Oct 30 19:00:29.787803 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:29.787661    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af747a06-3fa4-4c7f-a5a3-7e33a6d68746-istio-data" pod "af747a06-3fa4-4c7f-a5a3-7e33a6d68746" (UID: "af747a06-3fa4-4c7f-a5a3-7e33a6d68746"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:00:29.787803 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:29.787738    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af747a06-3fa4-4c7f-a5a3-7e33a6d68746-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af747a06-3fa4-4c7f-a5a3-7e33a6d68746-istiod-ca-cert\") pod \"af747a06-3fa4-4c7f-a5a3-7e33a6d68746\" (UID: \"af747a06-3fa4-4c7f-a5a3-7e33a6d68746\") "
Oct 30 19:00:29.787803 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:29.787785    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af747a06-3fa4-4c7f-a5a3-7e33a6d68746-workload-certs" pod "af747a06-3fa4-4c7f-a5a3-7e33a6d68746" (UID: "af747a06-3fa4-4c7f-a5a3-7e33a6d68746"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:00:29.788055 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:29.787896    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af747a06-3fa4-4c7f-a5a3-7e33a6d68746-credential-socket" pod "af747a06-3fa4-4c7f-a5a3-7e33a6d68746" (UID: "af747a06-3fa4-4c7f-a5a3-7e33a6d68746"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:00:29.788163 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:29.788136    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af747a06-3fa4-4c7f-a5a3-7e33a6d68746-istiod-ca-cert" pod "af747a06-3fa4-4c7f-a5a3-7e33a6d68746" (UID: "af747a06-3fa4-4c7f-a5a3-7e33a6d68746"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:00:29.788629 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:29.788604    4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af747a06-3fa4-4c7f-a5a3-7e33a6d68746-istiod-ca-cert\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:29.788750 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:29.788635    4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af747a06-3fa4-4c7f-a5a3-7e33a6d68746-workload-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:29.788750 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:29.788652    4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af747a06-3fa4-4c7f-a5a3-7e33a6d68746-istio-data\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:29.788750 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:29.788666    4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af747a06-3fa4-4c7f-a5a3-7e33a6d68746-workload-certs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:29.788750 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:29.788698    4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af747a06-3fa4-4c7f-a5a3-7e33a6d68746-credential-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:29.790016 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:29.789989    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af747a06-3fa4-4c7f-a5a3-7e33a6d68746-istio-token" pod "af747a06-3fa4-4c7f-a5a3-7e33a6d68746" (UID: "af747a06-3fa4-4c7f-a5a3-7e33a6d68746"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:00:29.790126 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:29.790015    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af747a06-3fa4-4c7f-a5a3-7e33a6d68746-istio-envoy" pod "af747a06-3fa4-4c7f-a5a3-7e33a6d68746" (UID: "af747a06-3fa4-4c7f-a5a3-7e33a6d68746"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:00:29.790126 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:29.790102    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/af747a06-3fa4-4c7f-a5a3-7e33a6d68746-istio-podinfo" pod "af747a06-3fa4-4c7f-a5a3-7e33a6d68746" (UID: "af747a06-3fa4-4c7f-a5a3-7e33a6d68746"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:00:29.791286 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:29.791259    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af747a06-3fa4-4c7f-a5a3-7e33a6d68746-kube-api-access-7b9ln" pod "af747a06-3fa4-4c7f-a5a3-7e33a6d68746" (UID: "af747a06-3fa4-4c7f-a5a3-7e33a6d68746"). InnerVolumeSpecName "kube-api-access-7b9ln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:00:29.890351 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:29.890319    4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af747a06-3fa4-4c7f-a5a3-7e33a6d68746-istio-token\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:29.890351 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:29.890350    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7b9ln\" (UniqueName: \"kubernetes.io/projected/af747a06-3fa4-4c7f-a5a3-7e33a6d68746-kube-api-access-7b9ln\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:29.890351 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:29.890361    4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af747a06-3fa4-4c7f-a5a3-7e33a6d68746-istio-podinfo\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:29.890809 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:29.890370    4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af747a06-3fa4-4c7f-a5a3-7e33a6d68746-istio-envoy\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:29.899163 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:29.899129    4525 generic.go:356] "Generic (PLEG): container finished" podID="af747a06-3fa4-4c7f-a5a3-7e33a6d68746" containerID="7d8b22f61a5ce3938d4ce4476451de0f79008caa8503e488122f8256130a4bfd" exitCode=0
Oct 30 19:00:29.899409 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:29.899205    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589"
Oct 30 19:00:29.899409 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:29.899212    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589" event={"ID":"af747a06-3fa4-4c7f-a5a3-7e33a6d68746","Type":"ContainerDied","Data":"7d8b22f61a5ce3938d4ce4476451de0f79008caa8503e488122f8256130a4bfd"}
Oct 30 19:00:29.899409 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:29.899283    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589" event={"ID":"af747a06-3fa4-4c7f-a5a3-7e33a6d68746","Type":"ContainerDied","Data":"d1036e7fdeab2598026bbc44cb98f4554467cb1f6040b17e4eb979991a7d330b"}
Oct 30 19:00:29.899409 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:29.899315    4525 scope.go:122] "RemoveContainer" containerID="7d8b22f61a5ce3938d4ce4476451de0f79008caa8503e488122f8256130a4bfd"
Oct 30 19:00:29.916992 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:29.916964    4525 scope.go:122] "RemoveContainer" containerID="4cf262d2a7d076ae846899bee2c71fb71c18aab346e48f1914fb85cc3249adf0"
Oct 30 19:00:29.928723 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:29.928696    4525 scope.go:122] "RemoveContainer" containerID="7d8b22f61a5ce3938d4ce4476451de0f79008caa8503e488122f8256130a4bfd"
Oct 30 19:00:29.929024 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:00:29.929006    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d8b22f61a5ce3938d4ce4476451de0f79008caa8503e488122f8256130a4bfd\": container with ID starting with 7d8b22f61a5ce3938d4ce4476451de0f79008caa8503e488122f8256130a4bfd not found: ID does not exist" containerID="7d8b22f61a5ce3938d4ce4476451de0f79008caa8503e488122f8256130a4bfd"
Oct 30 19:00:29.929091 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:29.929034    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d8b22f61a5ce3938d4ce4476451de0f79008caa8503e488122f8256130a4bfd"} err="failed to get container status \"7d8b22f61a5ce3938d4ce4476451de0f79008caa8503e488122f8256130a4bfd\": rpc error: code = NotFound desc = could not find container \"7d8b22f61a5ce3938d4ce4476451de0f79008caa8503e488122f8256130a4bfd\": container with ID starting with 7d8b22f61a5ce3938d4ce4476451de0f79008caa8503e488122f8256130a4bfd not found: ID does not exist"
Oct 30 19:00:29.929091 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:29.929051    4525 scope.go:122] "RemoveContainer" containerID="4cf262d2a7d076ae846899bee2c71fb71c18aab346e48f1914fb85cc3249adf0"
Oct 30 19:00:29.929298 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:00:29.929280    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cf262d2a7d076ae846899bee2c71fb71c18aab346e48f1914fb85cc3249adf0\": container with ID starting with 4cf262d2a7d076ae846899bee2c71fb71c18aab346e48f1914fb85cc3249adf0 not found: ID does not exist" containerID="4cf262d2a7d076ae846899bee2c71fb71c18aab346e48f1914fb85cc3249adf0"
Oct 30 19:00:29.929338 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:29.929308    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cf262d2a7d076ae846899bee2c71fb71c18aab346e48f1914fb85cc3249adf0"} err="failed to get container status \"4cf262d2a7d076ae846899bee2c71fb71c18aab346e48f1914fb85cc3249adf0\": rpc error: code = NotFound desc = could not find container \"4cf262d2a7d076ae846899bee2c71fb71c18aab346e48f1914fb85cc3249adf0\": container with ID starting with 4cf262d2a7d076ae846899bee2c71fb71c18aab346e48f1914fb85cc3249adf0 not found: ID does not exist"
Oct 30 19:00:29.933242 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:29.933184    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589"]
Oct 30 19:00:29.939808 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:29.939784    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-3d1aa8-openshift-default-7d4db8c885-69589"]
Oct 30 19:00:30.092184 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:30.092162    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f9b7116-4ntcq"
Oct 30 19:00:30.194536 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:30.194448    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03ce656d-ce81-4db9-b834-3d3ea7be50b2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03ce656d-ce81-4db9-b834-3d3ea7be50b2-maas-api-xks-ca\") pod \"03ce656d-ce81-4db9-b834-3d3ea7be50b2\" (UID: \"03ce656d-ce81-4db9-b834-3d3ea7be50b2\") "
Oct 30 19:00:30.194536 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:30.194484    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03ce656d-ce81-4db9-b834-3d3ea7be50b2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03ce656d-ce81-4db9-b834-3d3ea7be50b2-maas-api-openshift-ca\") pod \"03ce656d-ce81-4db9-b834-3d3ea7be50b2\" (UID: \"03ce656d-ce81-4db9-b834-3d3ea7be50b2\") "
Oct 30 19:00:30.194768 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:30.194637    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03ce656d-ce81-4db9-b834-3d3ea7be50b2-kube-api-access-f2qlc\" (UniqueName: \"kubernetes.io/projected/03ce656d-ce81-4db9-b834-3d3ea7be50b2-kube-api-access-f2qlc\") pod \"03ce656d-ce81-4db9-b834-3d3ea7be50b2\" (UID: \"03ce656d-ce81-4db9-b834-3d3ea7be50b2\") "
Oct 30 19:00:30.194902 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:30.194879    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03ce656d-ce81-4db9-b834-3d3ea7be50b2-maas-api-openshift-ca" pod "03ce656d-ce81-4db9-b834-3d3ea7be50b2" (UID: "03ce656d-ce81-4db9-b834-3d3ea7be50b2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:00:30.195204 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:30.195185    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03ce656d-ce81-4db9-b834-3d3ea7be50b2-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:30.196821 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:30.196801    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03ce656d-ce81-4db9-b834-3d3ea7be50b2-maas-api-xks-ca" pod "03ce656d-ce81-4db9-b834-3d3ea7be50b2" (UID: "03ce656d-ce81-4db9-b834-3d3ea7be50b2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:00:30.196895 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:30.196829    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03ce656d-ce81-4db9-b834-3d3ea7be50b2-kube-api-access-f2qlc" pod "03ce656d-ce81-4db9-b834-3d3ea7be50b2" (UID: "03ce656d-ce81-4db9-b834-3d3ea7be50b2"). InnerVolumeSpecName "kube-api-access-f2qlc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:00:30.296420 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:30.296387    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03ce656d-ce81-4db9-b834-3d3ea7be50b2-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:30.296420 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:30.296419    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2qlc\" (UniqueName: \"kubernetes.io/projected/03ce656d-ce81-4db9-b834-3d3ea7be50b2-kube-api-access-f2qlc\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:30.910929 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:30.910885    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f9b7116-4ntcq"
Oct 30 19:00:30.911453 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:30.910915    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f9b7116-4ntcq" event={"ID":"03ce656d-ce81-4db9-b834-3d3ea7be50b2","Type":"ContainerDied","Data":"10ceb890655715e1999e6b14ad3081c92db7fdfacf9fb37e2a5a55223de46446"}
Oct 30 19:00:30.911453 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:30.910978    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10ceb890655715e1999e6b14ad3081c92db7fdfacf9fb37e2a5a55223de46446"
Oct 30 19:00:31.544625 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:31.544591    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af747a06-3fa4-4c7f-a5a3-7e33a6d68746" path="/var/lib/kubelet/pods/af747a06-3fa4-4c7f-a5a3-7e33a6d68746/volumes"
Oct 30 19:00:33.890203 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:33.889938    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ae07349-gg5jk"]
Oct 30 19:00:33.894643 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:33.894611    4525 cpu_manager.go:414] "containerMap: removing container" podUID="af747a06-3fa4-4c7f-a5a3-7e33a6d68746" containerName="istio-proxy"
Oct 30 19:00:33.894643 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:33.894641    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af747a06-3fa4-4c7f-a5a3-7e33a6d68746" containerName="istio-proxy"
Oct 30 19:00:33.895583 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:33.894669    4525 cpu_manager.go:414] "containerMap: removing container" podUID="af747a06-3fa4-4c7f-a5a3-7e33a6d68746" containerName="istio-proxy"
Oct 30 19:00:33.895583 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:33.894695    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af747a06-3fa4-4c7f-a5a3-7e33a6d68746" containerName="istio-proxy"
Oct 30 19:00:33.895583 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:33.894731    4525 cpu_manager.go:414] "containerMap: removing container" podUID="03ce656d-ce81-4db9-b834-3d3ea7be50b2" containerName="revoke-keys"
Oct 30 19:00:33.895583 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:33.894741    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03ce656d-ce81-4db9-b834-3d3ea7be50b2" containerName="revoke-keys"
Oct 30 19:00:33.895583 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:33.894907    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="af747a06-3fa4-4c7f-a5a3-7e33a6d68746" containerName="istio-proxy"
Oct 30 19:00:33.895583 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:33.894923    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="af747a06-3fa4-4c7f-a5a3-7e33a6d68746" containerName="istio-proxy"
Oct 30 19:00:33.895583 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:33.894938    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="03ce656d-ce81-4db9-b834-3d3ea7be50b2" containerName="revoke-keys"
Oct 30 19:00:33.927878 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:33.909922    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ae07349-gg5jk"]
Oct 30 19:00:33.927878 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:33.910040    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ae07349-gg5jk"
Oct 30 19:00:33.927878 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:33.916317    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:00:33.927878 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:33.919773    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94hcn\""
Oct 30 19:00:34.052378 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:34.052345    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l8pd\" (UniqueName: \"kubernetes.io/projected/7e658646-ee3c-4e32-8ec1-f4572b96b1b2-kube-api-access-7l8pd\") pod \"maas-api-revoke-sub-7ae07349-gg5jk\" (UID: \"7e658646-ee3c-4e32-8ec1-f4572b96b1b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ae07349-gg5jk"
Oct 30 19:00:34.052520 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:34.052387    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e658646-ee3c-4e32-8ec1-f4572b96b1b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ae07349-gg5jk\" (UID: \"7e658646-ee3c-4e32-8ec1-f4572b96b1b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ae07349-gg5jk"
Oct 30 19:00:34.052520 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:34.052427    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e658646-ee3c-4e32-8ec1-f4572b96b1b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ae07349-gg5jk\" (UID: \"7e658646-ee3c-4e32-8ec1-f4572b96b1b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ae07349-gg5jk"
Oct 30 19:00:34.153561 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:34.153479    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l8pd\" (UniqueName: \"kubernetes.io/projected/7e658646-ee3c-4e32-8ec1-f4572b96b1b2-kube-api-access-7l8pd\") pod \"maas-api-revoke-sub-7ae07349-gg5jk\" (UID: \"7e658646-ee3c-4e32-8ec1-f4572b96b1b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ae07349-gg5jk"
Oct 30 19:00:34.153561 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:34.153521    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e658646-ee3c-4e32-8ec1-f4572b96b1b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ae07349-gg5jk\" (UID: \"7e658646-ee3c-4e32-8ec1-f4572b96b1b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ae07349-gg5jk"
Oct 30 19:00:34.153561 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:34.153554    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e658646-ee3c-4e32-8ec1-f4572b96b1b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ae07349-gg5jk\" (UID: \"7e658646-ee3c-4e32-8ec1-f4572b96b1b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ae07349-gg5jk"
Oct 30 19:00:34.154338 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:34.154318    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e658646-ee3c-4e32-8ec1-f4572b96b1b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ae07349-gg5jk\" (UID: \"7e658646-ee3c-4e32-8ec1-f4572b96b1b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ae07349-gg5jk"
Oct 30 19:00:34.156141 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:34.156120    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e658646-ee3c-4e32-8ec1-f4572b96b1b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ae07349-gg5jk\" (UID: \"7e658646-ee3c-4e32-8ec1-f4572b96b1b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ae07349-gg5jk"
Oct 30 19:00:34.161817 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:34.161798    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l8pd\" (UniqueName: \"kubernetes.io/projected/7e658646-ee3c-4e32-8ec1-f4572b96b1b2-kube-api-access-7l8pd\") pod \"maas-api-revoke-sub-7ae07349-gg5jk\" (UID: \"7e658646-ee3c-4e32-8ec1-f4572b96b1b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ae07349-gg5jk"
Oct 30 19:00:34.244363 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:34.244329    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ae07349-gg5jk"
Oct 30 19:00:34.246485 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:34.246458    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd5b9f09-7j65v"]
Oct 30 19:00:34.250776 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:34.250758    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd5b9f09-7j65v"
Oct 30 19:00:34.261722 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:34.260986    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd5b9f09-7j65v"]
Oct 30 19:00:34.355856 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:34.355821    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9a64a60-8374-4a31-a87f-a35e80c0e6af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd5b9f09-7j65v\" (UID: \"a9a64a60-8374-4a31-a87f-a35e80c0e6af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd5b9f09-7j65v"
Oct 30 19:00:34.356029 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:34.355995    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flzjv\" (UniqueName: \"kubernetes.io/projected/a9a64a60-8374-4a31-a87f-a35e80c0e6af-kube-api-access-flzjv\") pod \"maas-api-revoke-sub-dd5b9f09-7j65v\" (UID: \"a9a64a60-8374-4a31-a87f-a35e80c0e6af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd5b9f09-7j65v"
Oct 30 19:00:34.356088 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:34.356031    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9a64a60-8374-4a31-a87f-a35e80c0e6af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd5b9f09-7j65v\" (UID: \"a9a64a60-8374-4a31-a87f-a35e80c0e6af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd5b9f09-7j65v"
Oct 30 19:00:34.457284 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:34.457202    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9a64a60-8374-4a31-a87f-a35e80c0e6af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd5b9f09-7j65v\" (UID: \"a9a64a60-8374-4a31-a87f-a35e80c0e6af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd5b9f09-7j65v"
Oct 30 19:00:34.457437 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:34.457377    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flzjv\" (UniqueName: \"kubernetes.io/projected/a9a64a60-8374-4a31-a87f-a35e80c0e6af-kube-api-access-flzjv\") pod \"maas-api-revoke-sub-dd5b9f09-7j65v\" (UID: \"a9a64a60-8374-4a31-a87f-a35e80c0e6af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd5b9f09-7j65v"
Oct 30 19:00:34.457437 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:34.457410    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9a64a60-8374-4a31-a87f-a35e80c0e6af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd5b9f09-7j65v\" (UID: \"a9a64a60-8374-4a31-a87f-a35e80c0e6af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd5b9f09-7j65v"
Oct 30 19:00:34.457853 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:34.457833    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9a64a60-8374-4a31-a87f-a35e80c0e6af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd5b9f09-7j65v\" (UID: \"a9a64a60-8374-4a31-a87f-a35e80c0e6af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd5b9f09-7j65v"
Oct 30 19:00:34.459902 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:34.459878    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9a64a60-8374-4a31-a87f-a35e80c0e6af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd5b9f09-7j65v\" (UID: \"a9a64a60-8374-4a31-a87f-a35e80c0e6af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd5b9f09-7j65v"
Oct 30 19:00:34.469523 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:34.469493    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flzjv\" (UniqueName: \"kubernetes.io/projected/a9a64a60-8374-4a31-a87f-a35e80c0e6af-kube-api-access-flzjv\") pod \"maas-api-revoke-sub-dd5b9f09-7j65v\" (UID: \"a9a64a60-8374-4a31-a87f-a35e80c0e6af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd5b9f09-7j65v"
Oct 30 19:00:34.540164 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:34.540135    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ae07349-gg5jk"]
Oct 30 19:00:34.540374 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:00:34.540351    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e658646_ee3c_4e32_8ec1_f4572b96b1b2.slice/crio-11495ae7e2f089d5b5a8740e7b5493ff8b6e5d0b7630bd3c3b4e7d2b99fc9ad0 WatchSource:0}: Error finding container 11495ae7e2f089d5b5a8740e7b5493ff8b6e5d0b7630bd3c3b4e7d2b99fc9ad0: Status 404 returned error can't find the container with id 11495ae7e2f089d5b5a8740e7b5493ff8b6e5d0b7630bd3c3b4e7d2b99fc9ad0
Oct 30 19:00:34.595021 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:34.594981    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd5b9f09-7j65v"
Oct 30 19:00:34.891978 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:34.891952    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd5b9f09-7j65v"]
Oct 30 19:00:34.892848 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:00:34.892825    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9a64a60_8374_4a31_a87f_a35e80c0e6af.slice/crio-807a1b2437dca1625bc6a574912c9791d57e37a0befdbbe4c90e9ad4d295d716 WatchSource:0}: Error finding container 807a1b2437dca1625bc6a574912c9791d57e37a0befdbbe4c90e9ad4d295d716: Status 404 returned error can't find the container with id 807a1b2437dca1625bc6a574912c9791d57e37a0befdbbe4c90e9ad4d295d716
Oct 30 19:00:34.978286 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:34.978256    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd5b9f09-7j65v" event={"ID":"a9a64a60-8374-4a31-a87f-a35e80c0e6af","Type":"ContainerStarted","Data":"807a1b2437dca1625bc6a574912c9791d57e37a0befdbbe4c90e9ad4d295d716"}
Oct 30 19:00:34.980090 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:34.980060    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ae07349-gg5jk" event={"ID":"7e658646-ee3c-4e32-8ec1-f4572b96b1b2","Type":"ContainerStarted","Data":"11495ae7e2f089d5b5a8740e7b5493ff8b6e5d0b7630bd3c3b4e7d2b99fc9ad0"}
Oct 30 19:00:35.844364 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:35.844321    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9"]
Oct 30 19:00:35.844644 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:35.844591    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9" podUID="5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1" containerName="istio-proxy" containerID="cri-o://86a63fef72d63a6becfdf391e043a8dead722c8899591dd7ff0429ec8d41cdfe" gracePeriod=30
Oct 30 19:00:36.110100 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:36.105187    4525 generic.go:356] "Generic (PLEG): container finished" podID="7e658646-ee3c-4e32-8ec1-f4572b96b1b2" containerID="7b81547120503d9584661d9fb11bd06c14b52c2bc240f1e2eb2c1ff9402e778e" exitCode=0
Oct 30 19:00:36.110100 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:36.105670    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ae07349-gg5jk" event={"ID":"7e658646-ee3c-4e32-8ec1-f4572b96b1b2","Type":"ContainerDied","Data":"7b81547120503d9584661d9fb11bd06c14b52c2bc240f1e2eb2c1ff9402e778e"}
Oct 30 19:00:36.110100 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:36.108471    4525 generic.go:356] "Generic (PLEG): container finished" podID="a9a64a60-8374-4a31-a87f-a35e80c0e6af" containerID="35e92a1dce5ec9c65b77fc41286d318e6d9512a8d7369abcd0b1d38f11ee4a81" exitCode=0
Oct 30 19:00:36.110100 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:36.108772    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd5b9f09-7j65v" event={"ID":"a9a64a60-8374-4a31-a87f-a35e80c0e6af","Type":"ContainerDied","Data":"35e92a1dce5ec9c65b77fc41286d318e6d9512a8d7369abcd0b1d38f11ee4a81"}
Oct 30 19:00:36.253724 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:36.253670    4525 patch_prober.go:29] interesting pod/e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:00:36.254042 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:36.253747    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9" podUID="5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:00:38.250154 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:38.204762    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t" podUID="424fca14-6d8a-4663-86ec-0c91165f34ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:38.250154 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:38.205183    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv" podUID="595230e3-55f3-43eb-9a7b-6d89250af9b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 19:00:38.250154 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:38.214763    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj" podUID="84c99c49-1d60-4176-bda4-cad29c7d801c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:38.250154 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:38.215443    4525 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj" podUID="84c99c49-1d60-4176-bda4-cad29c7d801c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/health\": context deadline exceeded"
Oct 30 19:00:38.414913 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:38.414872    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts" podUID="be6d4969-285b-4da0-883e-4847f2cbf3af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:38.462241 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:38.442154    4525 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw" podUID="ac6852fd-01be-4d69-8c9b-0c591e50b889" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": context deadline exceeded"
Oct 30 19:00:38.474731 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:38.464782    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr" podUID="82bbbf85-ed93-48d2-9f17-49ac1f31c0a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:38.807766 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:38.801985    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw" podUID="ac6852fd-01be-4d69-8c9b-0c591e50b889" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 19:00:39.063294 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:38.851844    4525 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t" podUID="424fca14-6d8a-4663-86ec-0c91165f34ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": context deadline exceeded"
Oct 30 19:00:39.217708 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:39.216664    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ae07349-gg5jk"
Oct 30 19:00:39.217708 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:39.216967    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb" podUID="12ebc8b9-c794-47b5-b87b-a63c4bd993e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:39.217708 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:39.217662    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv" podUID="595230e3-55f3-43eb-9a7b-6d89250af9b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:39.231709 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:39.221782    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-6a327a/praxis-route-6a327a-kserve-5cb7997949-4vzjq" podUID="0dcb92f5-5de5-4c91-87cc-27764bea9683" containerName="main" probeResult="failure" output="Get \"https://10.133.0.149:8000/ready\": context deadline exceeded"
Oct 30 19:00:39.231709 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:39.222489    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj" podUID="84c99c49-1d60-4176-bda4-cad29c7d801c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 19:00:39.231709 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:39.227666    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t" podUID="424fca14-6d8a-4663-86ec-0c91165f34ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 19:00:39.256707 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:39.232752    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv"
Oct 30 19:00:39.256707 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:39.232814    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj"
Oct 30 19:00:39.256707 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:39.232852    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t"
Oct 30 19:00:39.416549 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:39.391868    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd5b9f09-7j65v"
Oct 30 19:00:39.454706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:39.440642    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e658646-ee3c-4e32-8ec1-f4572b96b1b2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e658646-ee3c-4e32-8ec1-f4572b96b1b2-maas-api-openshift-ca\") pod \"7e658646-ee3c-4e32-8ec1-f4572b96b1b2\" (UID: \"7e658646-ee3c-4e32-8ec1-f4572b96b1b2\") "
Oct 30 19:00:39.455838 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:39.455440    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e658646-ee3c-4e32-8ec1-f4572b96b1b2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e658646-ee3c-4e32-8ec1-f4572b96b1b2-maas-api-xks-ca\") pod \"7e658646-ee3c-4e32-8ec1-f4572b96b1b2\" (UID: \"7e658646-ee3c-4e32-8ec1-f4572b96b1b2\") "
Oct 30 19:00:39.455838 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:39.455611    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e658646-ee3c-4e32-8ec1-f4572b96b1b2-kube-api-access-7l8pd\" (UniqueName: \"kubernetes.io/projected/7e658646-ee3c-4e32-8ec1-f4572b96b1b2-kube-api-access-7l8pd\") pod \"7e658646-ee3c-4e32-8ec1-f4572b96b1b2\" (UID: \"7e658646-ee3c-4e32-8ec1-f4572b96b1b2\") "
Oct 30 19:00:39.485577 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:39.485538    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr" podUID="82bbbf85-ed93-48d2-9f17-49ac1f31c0a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 30 19:00:39.493080 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:39.441671    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts" podUID="be6d4969-285b-4da0-883e-4847f2cbf3af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:39.493618 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:39.493592    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr"
Oct 30 19:00:39.493836 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:39.493819    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts"
Oct 30 19:00:39.494827 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:39.486572    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e658646-ee3c-4e32-8ec1-f4572b96b1b2-maas-api-openshift-ca" pod "7e658646-ee3c-4e32-8ec1-f4572b96b1b2" (UID: "7e658646-ee3c-4e32-8ec1-f4572b96b1b2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:00:39.518445 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:39.517877    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv"
Oct 30 19:00:39.518445 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:39.517971    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t"
Oct 30 19:00:39.554796 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:39.554759    4525 patch_prober.go:29] interesting pod/e2e-worker-w0-9de872-openshift-default-77cf8df555-txf2f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:00:39.555064 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:39.555036    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-txf2f" podUID="f94f5889-fce0-46bc-a055-1c5e5e51d6a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:39.588650 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:39.588615    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9a64a60-8374-4a31-a87f-a35e80c0e6af-kube-api-access-flzjv\" (UniqueName: \"kubernetes.io/projected/a9a64a60-8374-4a31-a87f-a35e80c0e6af-kube-api-access-flzjv\") pod \"a9a64a60-8374-4a31-a87f-a35e80c0e6af\" (UID: \"a9a64a60-8374-4a31-a87f-a35e80c0e6af\") "
Oct 30 19:00:39.590622 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:39.590595    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9a64a60-8374-4a31-a87f-a35e80c0e6af-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9a64a60-8374-4a31-a87f-a35e80c0e6af-maas-api-xks-ca\") pod \"a9a64a60-8374-4a31-a87f-a35e80c0e6af\" (UID: \"a9a64a60-8374-4a31-a87f-a35e80c0e6af\") "
Oct 30 19:00:39.591285 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:39.591265    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9a64a60-8374-4a31-a87f-a35e80c0e6af-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9a64a60-8374-4a31-a87f-a35e80c0e6af-maas-api-openshift-ca\") pod \"a9a64a60-8374-4a31-a87f-a35e80c0e6af\" (UID: \"a9a64a60-8374-4a31-a87f-a35e80c0e6af\") "
Oct 30 19:00:39.592625 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:39.592601    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e658646-ee3c-4e32-8ec1-f4572b96b1b2-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:39.593821 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:39.591740    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9a64a60-8374-4a31-a87f-a35e80c0e6af-maas-api-openshift-ca" pod "a9a64a60-8374-4a31-a87f-a35e80c0e6af" (UID: "a9a64a60-8374-4a31-a87f-a35e80c0e6af"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:00:39.602176 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:39.602143    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e658646-ee3c-4e32-8ec1-f4572b96b1b2-maas-api-xks-ca" pod "7e658646-ee3c-4e32-8ec1-f4572b96b1b2" (UID: "7e658646-ee3c-4e32-8ec1-f4572b96b1b2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:00:39.741492 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:39.741222    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e658646-ee3c-4e32-8ec1-f4572b96b1b2-kube-api-access-7l8pd" pod "7e658646-ee3c-4e32-8ec1-f4572b96b1b2" (UID: "7e658646-ee3c-4e32-8ec1-f4572b96b1b2"). InnerVolumeSpecName "kube-api-access-7l8pd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:00:39.746935 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:39.746895    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9a64a60-8374-4a31-a87f-a35e80c0e6af-kube-api-access-flzjv" pod "a9a64a60-8374-4a31-a87f-a35e80c0e6af" (UID: "a9a64a60-8374-4a31-a87f-a35e80c0e6af"). InnerVolumeSpecName "kube-api-access-flzjv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:00:39.802750 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:39.802711    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw" podUID="ac6852fd-01be-4d69-8c9b-0c591e50b889" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:39.804039 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:39.804001    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw"
Oct 30 19:00:39.819819 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:39.819770    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9a64a60-8374-4a31-a87f-a35e80c0e6af-maas-api-xks-ca" pod "a9a64a60-8374-4a31-a87f-a35e80c0e6af" (UID: "a9a64a60-8374-4a31-a87f-a35e80c0e6af"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:00:39.826142 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:39.826090    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e658646-ee3c-4e32-8ec1-f4572b96b1b2-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:39.826384 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:39.826365    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flzjv\" (UniqueName: \"kubernetes.io/projected/a9a64a60-8374-4a31-a87f-a35e80c0e6af-kube-api-access-flzjv\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:39.826506 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:39.826490    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7l8pd\" (UniqueName: \"kubernetes.io/projected/7e658646-ee3c-4e32-8ec1-f4572b96b1b2-kube-api-access-7l8pd\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:39.826611 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:39.826597    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9a64a60-8374-4a31-a87f-a35e80c0e6af-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:39.826744 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:39.826729    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9a64a60-8374-4a31-a87f-a35e80c0e6af-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:40.085618 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:40.085524    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ae07349-gg5jk" event={"ID":"7e658646-ee3c-4e32-8ec1-f4572b96b1b2","Type":"ContainerDied","Data":"11495ae7e2f089d5b5a8740e7b5493ff8b6e5d0b7630bd3c3b4e7d2b99fc9ad0"}
Oct 30 19:00:40.085886 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:40.085866    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11495ae7e2f089d5b5a8740e7b5493ff8b6e5d0b7630bd3c3b4e7d2b99fc9ad0"
Oct 30 19:00:40.086092 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:40.086066    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ae07349-gg5jk"
Oct 30 19:00:40.158836 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:40.158792    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd5b9f09-7j65v" event={"ID":"a9a64a60-8374-4a31-a87f-a35e80c0e6af","Type":"ContainerDied","Data":"807a1b2437dca1625bc6a574912c9791d57e37a0befdbbe4c90e9ad4d295d716"}
Oct 30 19:00:40.162797 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:40.162763    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="807a1b2437dca1625bc6a574912c9791d57e37a0befdbbe4c90e9ad4d295d716"
Oct 30 19:00:40.172766 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:40.164606    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd5b9f09-7j65v"
Oct 30 19:00:40.220844 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:40.220794    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb" podUID="12ebc8b9-c794-47b5-b87b-a63c4bd993e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:40.222146 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:40.222090    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb"
Oct 30 19:00:40.320707 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:40.288874    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-6a327a/praxis-route-6a327a-kserve-5cb7997949-4vzjq" podUID="0dcb92f5-5de5-4c91-87cc-27764bea9683" containerName="main" probeResult="failure" output="Get \"https://10.133.0.149:8000/ready\": context deadline exceeded"
Oct 30 19:00:40.320707 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:40.289262    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj" podUID="84c99c49-1d60-4176-bda4-cad29c7d801c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 19:00:40.320707 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:40.314600    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-6a327a/praxis-route-6a327a-kserve-5cb7997949-4vzjq"
Oct 30 19:00:40.492874 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:40.492831    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr" podUID="82bbbf85-ed93-48d2-9f17-49ac1f31c0a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:40.495346 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:40.494937    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts" podUID="be6d4969-285b-4da0-883e-4847f2cbf3af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:40.733950 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:40.733922    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj"
Oct 30 19:00:40.820348 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:40.818789    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw" podUID="ac6852fd-01be-4d69-8c9b-0c591e50b889" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:41.158506 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:41.140798    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb"
Oct 30 19:00:41.158506 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:41.140878    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-6a327a/praxis-route-6a327a-kserve-5cb7997949-4vzjq"
Oct 30 19:00:41.158506 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:41.140926    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr"
Oct 30 19:00:41.214781 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:41.214743    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-635398a3-h52fw"]
Oct 30 19:00:41.219829 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:41.219788    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-635398a3-h52fw"]
Oct 30 19:00:41.220816 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:41.220512    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw"
Oct 30 19:00:41.235448 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:41.235425    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-24fceb-gpqzf"]
Oct 30 19:00:41.236432 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:41.236409    4525 cpu_manager.go:414] "containerMap: removing container" podUID="7e658646-ee3c-4e32-8ec1-f4572b96b1b2" containerName="revoke-keys"
Oct 30 19:00:41.236576 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:41.236563    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e658646-ee3c-4e32-8ec1-f4572b96b1b2" containerName="revoke-keys"
Oct 30 19:00:41.236719 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:41.236707    4525 cpu_manager.go:414] "containerMap: removing container" podUID="a9a64a60-8374-4a31-a87f-a35e80c0e6af" containerName="revoke-keys"
Oct 30 19:00:41.236809 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:41.236799    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9a64a60-8374-4a31-a87f-a35e80c0e6af" containerName="revoke-keys"
Oct 30 19:00:41.237037 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:41.237025    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e658646-ee3c-4e32-8ec1-f4572b96b1b2" containerName="revoke-keys"
Oct 30 19:00:41.237129 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:41.237121    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9a64a60-8374-4a31-a87f-a35e80c0e6af" containerName="revoke-keys"
Oct 30 19:00:41.251388 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:41.246745    4525 generic.go:356] "Generic (PLEG): container finished" podID="5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1" containerID="86a63fef72d63a6becfdf391e043a8dead722c8899591dd7ff0429ec8d41cdfe" exitCode=0
Oct 30 19:00:41.258316 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:41.258020    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts"
Oct 30 19:00:41.258600 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:41.258515    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9" event={"ID":"5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1","Type":"ContainerDied","Data":"86a63fef72d63a6becfdf391e043a8dead722c8899591dd7ff0429ec8d41cdfe"}
Oct 30 19:00:41.258871 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:41.258851    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-24fceb-gpqzf"]
Oct 30 19:00:41.259030 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:41.259016    4525 scope.go:122] "RemoveContainer" containerID="c18a57044ec8e94244aacce3bb29a64cfdd8474ebd9781aa20b51a70a84298c6"
Oct 30 19:00:41.259324 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:41.259307    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-24fceb-gpqzf"
Oct 30 19:00:41.275633 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:41.275602    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94hcn\""
Oct 30 19:00:41.275925 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:41.275908    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:00:41.324909 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:41.324876    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00911e2d-63e4-47f1-8935-cd8dcb9bed54-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-24fceb-gpqzf\" (UID: \"00911e2d-63e4-47f1-8935-cd8dcb9bed54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-24fceb-gpqzf"
Oct 30 19:00:41.325334 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:41.324929    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckn6s\" (UniqueName: \"kubernetes.io/projected/00911e2d-63e4-47f1-8935-cd8dcb9bed54-kube-api-access-ckn6s\") pod \"maas-api-revoke-keys-e2e-ipp-b-24fceb-gpqzf\" (UID: \"00911e2d-63e4-47f1-8935-cd8dcb9bed54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-24fceb-gpqzf"
Oct 30 19:00:41.325334 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:41.325046    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00911e2d-63e4-47f1-8935-cd8dcb9bed54-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-24fceb-gpqzf\" (UID: \"00911e2d-63e4-47f1-8935-cd8dcb9bed54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-24fceb-gpqzf"
Oct 30 19:00:41.417068 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:41.417035    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9"
Oct 30 19:00:41.427515 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:41.427490    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00911e2d-63e4-47f1-8935-cd8dcb9bed54-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-24fceb-gpqzf\" (UID: \"00911e2d-63e4-47f1-8935-cd8dcb9bed54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-24fceb-gpqzf"
Oct 30 19:00:41.427652 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:41.427523    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckn6s\" (UniqueName: \"kubernetes.io/projected/00911e2d-63e4-47f1-8935-cd8dcb9bed54-kube-api-access-ckn6s\") pod \"maas-api-revoke-keys-e2e-ipp-b-24fceb-gpqzf\" (UID: \"00911e2d-63e4-47f1-8935-cd8dcb9bed54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-24fceb-gpqzf"
Oct 30 19:00:41.427652 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:41.427587    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00911e2d-63e4-47f1-8935-cd8dcb9bed54-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-24fceb-gpqzf\" (UID: \"00911e2d-63e4-47f1-8935-cd8dcb9bed54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-24fceb-gpqzf"
Oct 30 19:00:41.428155 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:41.428133    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00911e2d-63e4-47f1-8935-cd8dcb9bed54-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-24fceb-gpqzf\" (UID: \"00911e2d-63e4-47f1-8935-cd8dcb9bed54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-24fceb-gpqzf"
Oct 30 19:00:41.430086 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:41.430063    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00911e2d-63e4-47f1-8935-cd8dcb9bed54-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-24fceb-gpqzf\" (UID: \"00911e2d-63e4-47f1-8935-cd8dcb9bed54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-24fceb-gpqzf"
Oct 30 19:00:41.438892 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:41.438861    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckn6s\" (UniqueName: \"kubernetes.io/projected/00911e2d-63e4-47f1-8935-cd8dcb9bed54-kube-api-access-ckn6s\") pod \"maas-api-revoke-keys-e2e-ipp-b-24fceb-gpqzf\" (UID: \"00911e2d-63e4-47f1-8935-cd8dcb9bed54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-24fceb-gpqzf"
Oct 30 19:00:41.529203 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:41.529172    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1-workload-certs\") pod \"5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1\" (UID: \"5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1\") "
Oct 30 19:00:41.529348 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:41.529222    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1-istio-podinfo\") pod \"5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1\" (UID: \"5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1\") "
Oct 30 19:00:41.529348 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:41.529274    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1-credential-socket\") pod \"5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1\" (UID: \"5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1\") "
Oct 30 19:00:41.529348 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:41.529295    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1-istio-token\" (UniqueName: \"kubernetes.io/projected/5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1-istio-token\") pod \"5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1\" (UID: \"5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1\") "
Oct 30 19:00:41.529348 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:41.529336    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1-kube-api-access-h5kxs\" (UniqueName: \"kubernetes.io/projected/5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1-kube-api-access-h5kxs\") pod \"5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1\" (UID: \"5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1\") "
Oct 30 19:00:41.529557 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:41.529363    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1-istio-envoy\") pod \"5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1\" (UID: \"5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1\") "
Oct 30 19:00:41.529557 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:41.529415    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1-istiod-ca-cert\") pod \"5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1\" (UID: \"5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1\") "
Oct 30 19:00:41.529557 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:41.529423    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1-workload-certs" pod "5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1" (UID: "5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:00:41.529749 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:41.529561    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1-credential-socket" pod "5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1" (UID: "5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:00:41.529864 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:41.529843    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1-istiod-ca-cert" pod "5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1" (UID: "5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:00:41.529928 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:41.529909    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1-istio-data\") pod \"5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1\" (UID: \"5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1\") "
Oct 30 19:00:41.530196 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:41.530094    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1-workload-socket\") pod \"5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1\" (UID: \"5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1\") "
Oct 30 19:00:41.530528 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:41.530360    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1-istio-data" pod "5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1" (UID: "5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:00:41.530626 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:41.530425    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1-workload-socket" pod "5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1" (UID: "5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:00:41.531653 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:41.531628    4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1-istio-data\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:41.531766 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:41.531741    4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1-workload-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:41.531766 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:41.531763    4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1-workload-certs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:41.531884 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:41.531787    4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1-credential-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:41.531884 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:41.531804    4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1-istiod-ca-cert\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:41.541009 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:41.540975    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1-kube-api-access-h5kxs" pod "5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1" (UID: "5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1"). InnerVolumeSpecName "kube-api-access-h5kxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:00:41.541383 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:41.541356    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1-istio-envoy" pod "5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1" (UID: "5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:00:41.541383 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:41.541367    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1-istio-token" pod "5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1" (UID: "5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:00:41.541383 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:41.541372    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1-istio-podinfo" pod "5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1" (UID: "5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:00:41.549420 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:41.549396    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac4b3887-d485-4fdc-848d-b9dff0adf0b0" path="/var/lib/kubelet/pods/ac4b3887-d485-4fdc-848d-b9dff0adf0b0/volumes"
Oct 30 19:00:41.600966 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:41.600936    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-24fceb-gpqzf"
Oct 30 19:00:41.632833 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:41.632801    4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1-istio-podinfo\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:41.632833 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:41.632837    4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1-istio-token\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:41.633013 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:41.632853    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5kxs\" (UniqueName: \"kubernetes.io/projected/5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1-kube-api-access-h5kxs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:41.633013 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:41.632868    4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1-istio-envoy\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:41.911259 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:00:41.906378    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00911e2d_63e4_47f1_8935_cd8dcb9bed54.slice/crio-e54c1ee3ad94ea115e39c72e8768e4a5ee395243bc5508df1d0c13d7535c6abf WatchSource:0}: Error finding container e54c1ee3ad94ea115e39c72e8768e4a5ee395243bc5508df1d0c13d7535c6abf: Status 404 returned error can't find the container with id e54c1ee3ad94ea115e39c72e8768e4a5ee395243bc5508df1d0c13d7535c6abf
Oct 30 19:00:41.913048 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:41.912918    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-24fceb-gpqzf"]
Oct 30 19:00:42.257148 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:42.257117    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9"
Oct 30 19:00:42.257148 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:42.257128    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9" event={"ID":"5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1","Type":"ContainerDied","Data":"82a4a3cfc26a03af234e7b38d7e1735e0502e3f5b365bf8be4007c5ea4d67932"}
Oct 30 19:00:42.257399 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:42.257193    4525 scope.go:122] "RemoveContainer" containerID="86a63fef72d63a6becfdf391e043a8dead722c8899591dd7ff0429ec8d41cdfe"
Oct 30 19:00:42.259384 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:42.259341    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-24fceb-gpqzf" event={"ID":"00911e2d-63e4-47f1-8935-cd8dcb9bed54","Type":"ContainerStarted","Data":"7aa78e891057a9e05eacf34765ccb80b48f34138a556bcc7b80535eff42db3c3"}
Oct 30 19:00:42.259511 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:42.259399    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-24fceb-gpqzf" event={"ID":"00911e2d-63e4-47f1-8935-cd8dcb9bed54","Type":"ContainerStarted","Data":"e54c1ee3ad94ea115e39c72e8768e4a5ee395243bc5508df1d0c13d7535c6abf"}
Oct 30 19:00:42.280667 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:42.280635    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9"]
Oct 30 19:00:42.286121 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:42.286092    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-93574a-openshift-default-54946f844f-mqjh9"]
Oct 30 19:00:42.313379 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:42.313322    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-24fceb-gpqzf" podStartSLOduration=1.313307475 podStartE2EDuration="1.313307475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:42.309798744 +0000 UTC m=+1464.935468710" watchObservedRunningTime="2026-10-30 19:00:42.313307475 +0000 UTC m=+1464.938977440"
Oct 30 19:00:43.270487 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:43.270454    4525 generic.go:356] "Generic (PLEG): container finished" podID="00911e2d-63e4-47f1-8935-cd8dcb9bed54" containerID="7aa78e891057a9e05eacf34765ccb80b48f34138a556bcc7b80535eff42db3c3" exitCode=0
Oct 30 19:00:43.270918 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:43.270543    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-24fceb-gpqzf" event={"ID":"00911e2d-63e4-47f1-8935-cd8dcb9bed54","Type":"ContainerDied","Data":"7aa78e891057a9e05eacf34765ccb80b48f34138a556bcc7b80535eff42db3c3"}
Oct 30 19:00:43.545234 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:43.545159    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1" path="/var/lib/kubelet/pods/5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1/volumes"
Oct 30 19:00:43.857519 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:43.857493    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0a29621-rmzp2"]
Oct 30 19:00:43.858124 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:43.858108    4525 cpu_manager.go:414] "containerMap: removing container" podUID="5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1" containerName="istio-proxy"
Oct 30 19:00:43.858124 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:43.858125    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1" containerName="istio-proxy"
Oct 30 19:00:43.858266 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:43.858135    4525 cpu_manager.go:414] "containerMap: removing container" podUID="5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1" containerName="istio-proxy"
Oct 30 19:00:43.858266 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:43.858141    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1" containerName="istio-proxy"
Oct 30 19:00:43.858266 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:43.858242    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1" containerName="istio-proxy"
Oct 30 19:00:43.858266 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:43.858249    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c7894e8-ae3d-4cf3-b7db-5597ec2b36a1" containerName="istio-proxy"
Oct 30 19:00:43.862052 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:43.862034    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a29621-rmzp2"
Oct 30 19:00:43.871673 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:43.871650    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0a29621-rmzp2"]
Oct 30 19:00:43.968760 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:43.968727    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98hlb\" (UniqueName: \"kubernetes.io/projected/107a0724-9d25-4b33-9866-e689ceb95234-kube-api-access-98hlb\") pod \"maas-api-revoke-sub-e0a29621-rmzp2\" (UID: \"107a0724-9d25-4b33-9866-e689ceb95234\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a29621-rmzp2"
Oct 30 19:00:43.968933 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:43.968896    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/107a0724-9d25-4b33-9866-e689ceb95234-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0a29621-rmzp2\" (UID: \"107a0724-9d25-4b33-9866-e689ceb95234\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a29621-rmzp2"
Oct 30 19:00:43.968979 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:43.968962    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/107a0724-9d25-4b33-9866-e689ceb95234-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0a29621-rmzp2\" (UID: \"107a0724-9d25-4b33-9866-e689ceb95234\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a29621-rmzp2"
Oct 30 19:00:44.070364 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:44.070330    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98hlb\" (UniqueName: \"kubernetes.io/projected/107a0724-9d25-4b33-9866-e689ceb95234-kube-api-access-98hlb\") pod \"maas-api-revoke-sub-e0a29621-rmzp2\" (UID: \"107a0724-9d25-4b33-9866-e689ceb95234\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a29621-rmzp2"
Oct 30 19:00:44.070575 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:44.070454    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/107a0724-9d25-4b33-9866-e689ceb95234-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0a29621-rmzp2\" (UID: \"107a0724-9d25-4b33-9866-e689ceb95234\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a29621-rmzp2"
Oct 30 19:00:44.070575 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:44.070516    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/107a0724-9d25-4b33-9866-e689ceb95234-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0a29621-rmzp2\" (UID: \"107a0724-9d25-4b33-9866-e689ceb95234\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a29621-rmzp2"
Oct 30 19:00:44.071258 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:44.071230    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/107a0724-9d25-4b33-9866-e689ceb95234-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0a29621-rmzp2\" (UID: \"107a0724-9d25-4b33-9866-e689ceb95234\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a29621-rmzp2"
Oct 30 19:00:44.073163 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:44.073135    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/107a0724-9d25-4b33-9866-e689ceb95234-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0a29621-rmzp2\" (UID: \"107a0724-9d25-4b33-9866-e689ceb95234\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a29621-rmzp2"
Oct 30 19:00:44.078403 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:44.078382    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98hlb\" (UniqueName: \"kubernetes.io/projected/107a0724-9d25-4b33-9866-e689ceb95234-kube-api-access-98hlb\") pod \"maas-api-revoke-sub-e0a29621-rmzp2\" (UID: \"107a0724-9d25-4b33-9866-e689ceb95234\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a29621-rmzp2"
Oct 30 19:00:44.179278 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:44.179184    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a29621-rmzp2"
Oct 30 19:00:44.492946 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:44.492921    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-24fceb-gpqzf"
Oct 30 19:00:44.507672 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:44.507636    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0a29621-rmzp2"]
Oct 30 19:00:44.509087 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:00:44.509057    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod107a0724_9d25_4b33_9866_e689ceb95234.slice/crio-705f4d3e7205ff70355f776c83445e2c8628f22fa63d900c63ab89ed66406be2 WatchSource:0}: Error finding container 705f4d3e7205ff70355f776c83445e2c8628f22fa63d900c63ab89ed66406be2: Status 404 returned error can't find the container with id 705f4d3e7205ff70355f776c83445e2c8628f22fa63d900c63ab89ed66406be2
Oct 30 19:00:44.590840 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:44.590808    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00911e2d-63e4-47f1-8935-cd8dcb9bed54-kube-api-access-ckn6s\" (UniqueName: \"kubernetes.io/projected/00911e2d-63e4-47f1-8935-cd8dcb9bed54-kube-api-access-ckn6s\") pod \"00911e2d-63e4-47f1-8935-cd8dcb9bed54\" (UID: \"00911e2d-63e4-47f1-8935-cd8dcb9bed54\") "
Oct 30 19:00:44.590992 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:44.590866    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00911e2d-63e4-47f1-8935-cd8dcb9bed54-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00911e2d-63e4-47f1-8935-cd8dcb9bed54-maas-api-xks-ca\") pod \"00911e2d-63e4-47f1-8935-cd8dcb9bed54\" (UID: \"00911e2d-63e4-47f1-8935-cd8dcb9bed54\") "
Oct 30 19:00:44.590992 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:44.590974    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00911e2d-63e4-47f1-8935-cd8dcb9bed54-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00911e2d-63e4-47f1-8935-cd8dcb9bed54-maas-api-openshift-ca\") pod \"00911e2d-63e4-47f1-8935-cd8dcb9bed54\" (UID: \"00911e2d-63e4-47f1-8935-cd8dcb9bed54\") "
Oct 30 19:00:44.591374 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:44.591347    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00911e2d-63e4-47f1-8935-cd8dcb9bed54-maas-api-openshift-ca" pod "00911e2d-63e4-47f1-8935-cd8dcb9bed54" (UID: "00911e2d-63e4-47f1-8935-cd8dcb9bed54"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:00:44.591558 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:44.591499    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00911e2d-63e4-47f1-8935-cd8dcb9bed54-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:44.593085 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:44.593053    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00911e2d-63e4-47f1-8935-cd8dcb9bed54-maas-api-xks-ca" pod "00911e2d-63e4-47f1-8935-cd8dcb9bed54" (UID: "00911e2d-63e4-47f1-8935-cd8dcb9bed54"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:00:44.593207 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:44.593101    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00911e2d-63e4-47f1-8935-cd8dcb9bed54-kube-api-access-ckn6s" pod "00911e2d-63e4-47f1-8935-cd8dcb9bed54" (UID: "00911e2d-63e4-47f1-8935-cd8dcb9bed54"). InnerVolumeSpecName "kube-api-access-ckn6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:00:44.636221 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:44.636190    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db33a84d-5x889"]
Oct 30 19:00:44.637093 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:44.637072    4525 cpu_manager.go:414] "containerMap: removing container" podUID="00911e2d-63e4-47f1-8935-cd8dcb9bed54" containerName="revoke-keys"
Oct 30 19:00:44.637137 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:44.637097    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00911e2d-63e4-47f1-8935-cd8dcb9bed54" containerName="revoke-keys"
Oct 30 19:00:44.637231 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:44.637220    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="00911e2d-63e4-47f1-8935-cd8dcb9bed54" containerName="revoke-keys"
Oct 30 19:00:44.642786 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:44.642765    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db33a84d-5x889"
Oct 30 19:00:44.650370 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:44.650347    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db33a84d-5x889"]
Oct 30 19:00:44.692840 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:44.692808    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckn6s\" (UniqueName: \"kubernetes.io/projected/00911e2d-63e4-47f1-8935-cd8dcb9bed54-kube-api-access-ckn6s\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:44.692840 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:44.692837    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00911e2d-63e4-47f1-8935-cd8dcb9bed54-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:44.794330 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:44.794304    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78qp8\" (UniqueName: \"kubernetes.io/projected/1b248c08-1780-454f-9ef9-596fb0cfb250-kube-api-access-78qp8\") pod \"maas-api-revoke-sub-db33a84d-5x889\" (UID: \"1b248c08-1780-454f-9ef9-596fb0cfb250\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db33a84d-5x889"
Oct 30 19:00:44.794453 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:44.794367    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b248c08-1780-454f-9ef9-596fb0cfb250-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db33a84d-5x889\" (UID: \"1b248c08-1780-454f-9ef9-596fb0cfb250\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db33a84d-5x889"
Oct 30 19:00:44.794453 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:44.794406    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b248c08-1780-454f-9ef9-596fb0cfb250-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db33a84d-5x889\" (UID: \"1b248c08-1780-454f-9ef9-596fb0cfb250\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db33a84d-5x889"
Oct 30 19:00:44.924733 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:44.924698    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78qp8\" (UniqueName: \"kubernetes.io/projected/1b248c08-1780-454f-9ef9-596fb0cfb250-kube-api-access-78qp8\") pod \"maas-api-revoke-sub-db33a84d-5x889\" (UID: \"1b248c08-1780-454f-9ef9-596fb0cfb250\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db33a84d-5x889"
Oct 30 19:00:44.924976 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:44.924795    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b248c08-1780-454f-9ef9-596fb0cfb250-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db33a84d-5x889\" (UID: \"1b248c08-1780-454f-9ef9-596fb0cfb250\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db33a84d-5x889"
Oct 30 19:00:44.924976 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:44.924826    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b248c08-1780-454f-9ef9-596fb0cfb250-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db33a84d-5x889\" (UID: \"1b248c08-1780-454f-9ef9-596fb0cfb250\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db33a84d-5x889"
Oct 30 19:00:44.934412 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:44.926459    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b248c08-1780-454f-9ef9-596fb0cfb250-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db33a84d-5x889\" (UID: \"1b248c08-1780-454f-9ef9-596fb0cfb250\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db33a84d-5x889"
Oct 30 19:00:44.934412 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:44.931484    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b248c08-1780-454f-9ef9-596fb0cfb250-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db33a84d-5x889\" (UID: \"1b248c08-1780-454f-9ef9-596fb0cfb250\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db33a84d-5x889"
Oct 30 19:00:44.945398 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:44.945040    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78qp8\" (UniqueName: \"kubernetes.io/projected/1b248c08-1780-454f-9ef9-596fb0cfb250-kube-api-access-78qp8\") pod \"maas-api-revoke-sub-db33a84d-5x889\" (UID: \"1b248c08-1780-454f-9ef9-596fb0cfb250\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db33a84d-5x889"
Oct 30 19:00:44.974932 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:44.974898    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db33a84d-5x889"
Oct 30 19:00:45.297898 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:45.297870    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db33a84d-5x889"]
Oct 30 19:00:45.305173 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:45.305068    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-24fceb-gpqzf"
Oct 30 19:00:45.305173 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:45.305074    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-24fceb-gpqzf" event={"ID":"00911e2d-63e4-47f1-8935-cd8dcb9bed54","Type":"ContainerDied","Data":"e54c1ee3ad94ea115e39c72e8768e4a5ee395243bc5508df1d0c13d7535c6abf"}
Oct 30 19:00:45.305173 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:45.305151    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e54c1ee3ad94ea115e39c72e8768e4a5ee395243bc5508df1d0c13d7535c6abf"
Oct 30 19:00:45.309077 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:45.308369    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a29621-rmzp2" event={"ID":"107a0724-9d25-4b33-9866-e689ceb95234","Type":"ContainerStarted","Data":"41ad94f62444fcc6d76752b311e26d4cc4e5839d15448c1202b8a47b62b95266"}
Oct 30 19:00:45.309077 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:45.308412    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a29621-rmzp2" event={"ID":"107a0724-9d25-4b33-9866-e689ceb95234","Type":"ContainerStarted","Data":"705f4d3e7205ff70355f776c83445e2c8628f22fa63d900c63ab89ed66406be2"}
Oct 30 19:00:45.327498 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:45.327457    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a29621-rmzp2" podStartSLOduration=2.3274451369999998 podStartE2EDuration="2.327445137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:45.325030862 +0000 UTC m=+1467.950700826" watchObservedRunningTime="2026-10-30 19:00:45.327445137 +0000 UTC m=+1467.953115101"
Oct 30 19:00:46.318931 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:46.318896    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db33a84d-5x889" event={"ID":"1b248c08-1780-454f-9ef9-596fb0cfb250","Type":"ContainerStarted","Data":"7310e87ef76e1f0498b0be40911bcbd08b7610a3ba2e5e7ca2ff884346550777"}
Oct 30 19:00:46.319374 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:46.318960    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db33a84d-5x889" event={"ID":"1b248c08-1780-454f-9ef9-596fb0cfb250","Type":"ContainerStarted","Data":"752b0f3be93415c14f35cf55fa5e05d10ab60290ac34d88a23c169156d8527ba"}
Oct 30 19:00:46.320634 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:46.320608    4525 generic.go:356] "Generic (PLEG): container finished" podID="107a0724-9d25-4b33-9866-e689ceb95234" containerID="41ad94f62444fcc6d76752b311e26d4cc4e5839d15448c1202b8a47b62b95266" exitCode=0
Oct 30 19:00:46.320791 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:46.320734    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a29621-rmzp2" event={"ID":"107a0724-9d25-4b33-9866-e689ceb95234","Type":"ContainerDied","Data":"41ad94f62444fcc6d76752b311e26d4cc4e5839d15448c1202b8a47b62b95266"}
Oct 30 19:00:46.338160 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:46.338102    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db33a84d-5x889" podStartSLOduration=2.3380887870000002 podStartE2EDuration="2.338088787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:46.336759777 +0000 UTC m=+1468.962429741" watchObservedRunningTime="2026-10-30 19:00:46.338088787 +0000 UTC m=+1468.963758741"
Oct 30 19:00:46.484396 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:46.484361    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9073694d-9dlpm"]
Oct 30 19:00:46.488439 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:46.488423    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9073694d-9dlpm"
Oct 30 19:00:46.508505 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:46.504407    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9073694d-9dlpm"]
Oct 30 19:00:46.649580 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:46.649543    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhjs5\" (UniqueName: \"kubernetes.io/projected/e47d550d-c724-482c-9352-077d0d9e17c9-kube-api-access-xhjs5\") pod \"maas-api-revoke-sub-9073694d-9dlpm\" (UID: \"e47d550d-c724-482c-9352-077d0d9e17c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9073694d-9dlpm"
Oct 30 19:00:46.649580 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:46.649583    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e47d550d-c724-482c-9352-077d0d9e17c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9073694d-9dlpm\" (UID: \"e47d550d-c724-482c-9352-077d0d9e17c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9073694d-9dlpm"
Oct 30 19:00:46.649941 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:46.649917    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e47d550d-c724-482c-9352-077d0d9e17c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9073694d-9dlpm\" (UID: \"e47d550d-c724-482c-9352-077d0d9e17c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9073694d-9dlpm"
Oct 30 19:00:46.678394 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:46.678367    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6a327a-z5pnw"]
Oct 30 19:00:46.682555 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:46.682537    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6a327a-z5pnw"
Oct 30 19:00:46.697932 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:46.692467    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6a327a-z5pnw"]
Oct 30 19:00:46.751055 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:46.751014    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhjs5\" (UniqueName: \"kubernetes.io/projected/e47d550d-c724-482c-9352-077d0d9e17c9-kube-api-access-xhjs5\") pod \"maas-api-revoke-sub-9073694d-9dlpm\" (UID: \"e47d550d-c724-482c-9352-077d0d9e17c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9073694d-9dlpm"
Oct 30 19:00:46.751226 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:46.751064    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e47d550d-c724-482c-9352-077d0d9e17c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9073694d-9dlpm\" (UID: \"e47d550d-c724-482c-9352-077d0d9e17c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9073694d-9dlpm"
Oct 30 19:00:46.751302 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:46.751283    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e47d550d-c724-482c-9352-077d0d9e17c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9073694d-9dlpm\" (UID: \"e47d550d-c724-482c-9352-077d0d9e17c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9073694d-9dlpm"
Oct 30 19:00:46.751796 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:46.751775    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e47d550d-c724-482c-9352-077d0d9e17c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9073694d-9dlpm\" (UID: \"e47d550d-c724-482c-9352-077d0d9e17c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9073694d-9dlpm"
Oct 30 19:00:46.753807 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:46.753788    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e47d550d-c724-482c-9352-077d0d9e17c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9073694d-9dlpm\" (UID: \"e47d550d-c724-482c-9352-077d0d9e17c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9073694d-9dlpm"
Oct 30 19:00:46.758856 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:46.758828    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhjs5\" (UniqueName: \"kubernetes.io/projected/e47d550d-c724-482c-9352-077d0d9e17c9-kube-api-access-xhjs5\") pod \"maas-api-revoke-sub-9073694d-9dlpm\" (UID: \"e47d550d-c724-482c-9352-077d0d9e17c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9073694d-9dlpm"
Oct 30 19:00:46.828807 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:46.828766    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9073694d-9dlpm"
Oct 30 19:00:46.852396 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:46.852364    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67f662f4-cbe1-4762-9678-220149cc2eb2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-6a327a-z5pnw\" (UID: \"67f662f4-cbe1-4762-9678-220149cc2eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6a327a-z5pnw"
Oct 30 19:00:46.852531 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:46.852446    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flg5n\" (UniqueName: \"kubernetes.io/projected/67f662f4-cbe1-4762-9678-220149cc2eb2-kube-api-access-flg5n\") pod \"maas-api-revoke-keys-e2e-praxis-a-6a327a-z5pnw\" (UID: \"67f662f4-cbe1-4762-9678-220149cc2eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6a327a-z5pnw"
Oct 30 19:00:46.852641 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:46.852619    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67f662f4-cbe1-4762-9678-220149cc2eb2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-6a327a-z5pnw\" (UID: \"67f662f4-cbe1-4762-9678-220149cc2eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6a327a-z5pnw"
Oct 30 19:00:46.954011 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:46.953924    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67f662f4-cbe1-4762-9678-220149cc2eb2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-6a327a-z5pnw\" (UID: \"67f662f4-cbe1-4762-9678-220149cc2eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6a327a-z5pnw"
Oct 30 19:00:46.954171 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:46.954023    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flg5n\" (UniqueName: \"kubernetes.io/projected/67f662f4-cbe1-4762-9678-220149cc2eb2-kube-api-access-flg5n\") pod \"maas-api-revoke-keys-e2e-praxis-a-6a327a-z5pnw\" (UID: \"67f662f4-cbe1-4762-9678-220149cc2eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6a327a-z5pnw"
Oct 30 19:00:46.954171 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:46.954152    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67f662f4-cbe1-4762-9678-220149cc2eb2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-6a327a-z5pnw\" (UID: \"67f662f4-cbe1-4762-9678-220149cc2eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6a327a-z5pnw"
Oct 30 19:00:46.954636 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:46.954608    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67f662f4-cbe1-4762-9678-220149cc2eb2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-6a327a-z5pnw\" (UID: \"67f662f4-cbe1-4762-9678-220149cc2eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6a327a-z5pnw"
Oct 30 19:00:46.956552 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:46.956531    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67f662f4-cbe1-4762-9678-220149cc2eb2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-6a327a-z5pnw\" (UID: \"67f662f4-cbe1-4762-9678-220149cc2eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6a327a-z5pnw"
Oct 30 19:00:46.961918 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:46.961899    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flg5n\" (UniqueName: \"kubernetes.io/projected/67f662f4-cbe1-4762-9678-220149cc2eb2-kube-api-access-flg5n\") pod \"maas-api-revoke-keys-e2e-praxis-a-6a327a-z5pnw\" (UID: \"67f662f4-cbe1-4762-9678-220149cc2eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6a327a-z5pnw"
Oct 30 19:00:46.999876 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:46.999846    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6a327a-z5pnw"
Oct 30 19:00:47.110115 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:00:47.110014    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode47d550d_c724_482c_9352_077d0d9e17c9.slice/crio-59fdb1411b7bbf03ac477c41b3731d33e245949371683165155adf5b037efe49 WatchSource:0}: Error finding container 59fdb1411b7bbf03ac477c41b3731d33e245949371683165155adf5b037efe49: Status 404 returned error can't find the container with id 59fdb1411b7bbf03ac477c41b3731d33e245949371683165155adf5b037efe49
Oct 30 19:00:47.114067 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:47.113788    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9073694d-9dlpm"]
Oct 30 19:00:47.282198 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:47.282169    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6a327a-z5pnw"]
Oct 30 19:00:47.296148 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:00:47.296115    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67f662f4_cbe1_4762_9678_220149cc2eb2.slice/crio-f5e0e44b2d21c9a4d0b338b433b05913361bd7d7ff7cade36bb43d6691890b6d WatchSource:0}: Error finding container f5e0e44b2d21c9a4d0b338b433b05913361bd7d7ff7cade36bb43d6691890b6d: Status 404 returned error can't find the container with id f5e0e44b2d21c9a4d0b338b433b05913361bd7d7ff7cade36bb43d6691890b6d
Oct 30 19:00:47.330835 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:47.330803    4525 generic.go:356] "Generic (PLEG): container finished" podID="1b248c08-1780-454f-9ef9-596fb0cfb250" containerID="7310e87ef76e1f0498b0be40911bcbd08b7610a3ba2e5e7ca2ff884346550777" exitCode=0
Oct 30 19:00:47.331245 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:47.330954    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db33a84d-5x889" event={"ID":"1b248c08-1780-454f-9ef9-596fb0cfb250","Type":"ContainerDied","Data":"7310e87ef76e1f0498b0be40911bcbd08b7610a3ba2e5e7ca2ff884346550777"}
Oct 30 19:00:47.332433 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:47.332409    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9073694d-9dlpm" event={"ID":"e47d550d-c724-482c-9352-077d0d9e17c9","Type":"ContainerStarted","Data":"59fdb1411b7bbf03ac477c41b3731d33e245949371683165155adf5b037efe49"}
Oct 30 19:00:47.333718 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:47.333700    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6a327a-z5pnw" event={"ID":"67f662f4-cbe1-4762-9678-220149cc2eb2","Type":"ContainerStarted","Data":"f5e0e44b2d21c9a4d0b338b433b05913361bd7d7ff7cade36bb43d6691890b6d"}
Oct 30 19:00:47.417526 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:00:47.417487    4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d49l" containerName="extractor"
Oct 30 19:00:47.606939 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:47.606907    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a29621-rmzp2"
Oct 30 19:00:47.774069 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:47.774034    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/107a0724-9d25-4b33-9866-e689ceb95234-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/107a0724-9d25-4b33-9866-e689ceb95234-maas-api-openshift-ca\") pod \"107a0724-9d25-4b33-9866-e689ceb95234\" (UID: \"107a0724-9d25-4b33-9866-e689ceb95234\") "
Oct 30 19:00:47.774251 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:47.774102    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/107a0724-9d25-4b33-9866-e689ceb95234-kube-api-access-98hlb\" (UniqueName: \"kubernetes.io/projected/107a0724-9d25-4b33-9866-e689ceb95234-kube-api-access-98hlb\") pod \"107a0724-9d25-4b33-9866-e689ceb95234\" (UID: \"107a0724-9d25-4b33-9866-e689ceb95234\") "
Oct 30 19:00:47.774251 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:47.774126    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/107a0724-9d25-4b33-9866-e689ceb95234-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/107a0724-9d25-4b33-9866-e689ceb95234-maas-api-xks-ca\") pod \"107a0724-9d25-4b33-9866-e689ceb95234\" (UID: \"107a0724-9d25-4b33-9866-e689ceb95234\") "
Oct 30 19:00:47.774431 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:47.774385    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/107a0724-9d25-4b33-9866-e689ceb95234-maas-api-openshift-ca" pod "107a0724-9d25-4b33-9866-e689ceb95234" (UID: "107a0724-9d25-4b33-9866-e689ceb95234"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:00:47.774651 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:47.774631    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/107a0724-9d25-4b33-9866-e689ceb95234-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:47.776384 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:47.776352    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/107a0724-9d25-4b33-9866-e689ceb95234-kube-api-access-98hlb" pod "107a0724-9d25-4b33-9866-e689ceb95234" (UID: "107a0724-9d25-4b33-9866-e689ceb95234"). InnerVolumeSpecName "kube-api-access-98hlb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:00:47.776498 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:47.776445    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/107a0724-9d25-4b33-9866-e689ceb95234-maas-api-xks-ca" pod "107a0724-9d25-4b33-9866-e689ceb95234" (UID: "107a0724-9d25-4b33-9866-e689ceb95234"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:00:47.876288 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:47.876259    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98hlb\" (UniqueName: \"kubernetes.io/projected/107a0724-9d25-4b33-9866-e689ceb95234-kube-api-access-98hlb\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:47.876288 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:47.876288    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/107a0724-9d25-4b33-9866-e689ceb95234-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:48.343860 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:48.343828    4525 generic.go:356] "Generic (PLEG): container finished" podID="e47d550d-c724-482c-9352-077d0d9e17c9" containerID="2f4509a742eef5429f304848bc20f625eb2177c435666b97307d5d6fe9a62dd2" exitCode=0
Oct 30 19:00:48.344354 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:48.343911    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9073694d-9dlpm" event={"ID":"e47d550d-c724-482c-9352-077d0d9e17c9","Type":"ContainerDied","Data":"2f4509a742eef5429f304848bc20f625eb2177c435666b97307d5d6fe9a62dd2"}
Oct 30 19:00:48.345948 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:48.345920    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6a327a-z5pnw" event={"ID":"67f662f4-cbe1-4762-9678-220149cc2eb2","Type":"ContainerStarted","Data":"acbc54293ceb9cc43d9cbd294ad19a5eba04ff2443138d843c01ec0e2a600d8e"}
Oct 30 19:00:48.347501 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:48.347473    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a29621-rmzp2"
Oct 30 19:00:48.347612 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:48.347519    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a29621-rmzp2" event={"ID":"107a0724-9d25-4b33-9866-e689ceb95234","Type":"ContainerDied","Data":"705f4d3e7205ff70355f776c83445e2c8628f22fa63d900c63ab89ed66406be2"}
Oct 30 19:00:48.347612 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:48.347572    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="705f4d3e7205ff70355f776c83445e2c8628f22fa63d900c63ab89ed66406be2"
Oct 30 19:00:48.385872 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:48.385801    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6a327a-z5pnw" podStartSLOduration=2.385775824 podStartE2EDuration="2.385775824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:48.382556355 +0000 UTC m=+1471.008226333" watchObservedRunningTime="2026-10-30 19:00:48.385775824 +0000 UTC m=+1471.011445788"
Oct 30 19:00:48.615484 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:48.615447    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db33a84d-5x889"
Oct 30 19:00:48.698495 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:48.698460    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b248c08-1780-454f-9ef9-596fb0cfb250-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b248c08-1780-454f-9ef9-596fb0cfb250-maas-api-xks-ca\") pod \"1b248c08-1780-454f-9ef9-596fb0cfb250\" (UID: \"1b248c08-1780-454f-9ef9-596fb0cfb250\") "
Oct 30 19:00:48.698892 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:48.698864    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b248c08-1780-454f-9ef9-596fb0cfb250-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b248c08-1780-454f-9ef9-596fb0cfb250-maas-api-openshift-ca\") pod \"1b248c08-1780-454f-9ef9-596fb0cfb250\" (UID: \"1b248c08-1780-454f-9ef9-596fb0cfb250\") "
Oct 30 19:00:48.699040 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:48.698954    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b248c08-1780-454f-9ef9-596fb0cfb250-kube-api-access-78qp8\" (UniqueName: \"kubernetes.io/projected/1b248c08-1780-454f-9ef9-596fb0cfb250-kube-api-access-78qp8\") pod \"1b248c08-1780-454f-9ef9-596fb0cfb250\" (UID: \"1b248c08-1780-454f-9ef9-596fb0cfb250\") "
Oct 30 19:00:48.699328 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:48.699297    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b248c08-1780-454f-9ef9-596fb0cfb250-maas-api-openshift-ca" pod "1b248c08-1780-454f-9ef9-596fb0cfb250" (UID: "1b248c08-1780-454f-9ef9-596fb0cfb250"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:00:48.699550 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:48.699527    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b248c08-1780-454f-9ef9-596fb0cfb250-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:48.701483 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:48.701416    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b248c08-1780-454f-9ef9-596fb0cfb250-kube-api-access-78qp8" pod "1b248c08-1780-454f-9ef9-596fb0cfb250" (UID: "1b248c08-1780-454f-9ef9-596fb0cfb250"). InnerVolumeSpecName "kube-api-access-78qp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:00:48.703110 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:48.703081    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b248c08-1780-454f-9ef9-596fb0cfb250-maas-api-xks-ca" pod "1b248c08-1780-454f-9ef9-596fb0cfb250" (UID: "1b248c08-1780-454f-9ef9-596fb0cfb250"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:00:48.801085 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:48.801005    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78qp8\" (UniqueName: \"kubernetes.io/projected/1b248c08-1780-454f-9ef9-596fb0cfb250-kube-api-access-78qp8\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:48.801085 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:48.801036    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b248c08-1780-454f-9ef9-596fb0cfb250-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:49.358047 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:49.358014    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db33a84d-5x889"
Oct 30 19:00:49.358455 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:49.358041    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db33a84d-5x889" event={"ID":"1b248c08-1780-454f-9ef9-596fb0cfb250","Type":"ContainerDied","Data":"752b0f3be93415c14f35cf55fa5e05d10ab60290ac34d88a23c169156d8527ba"}
Oct 30 19:00:49.358455 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:49.358138    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="752b0f3be93415c14f35cf55fa5e05d10ab60290ac34d88a23c169156d8527ba"
Oct 30 19:00:49.359949 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:49.359923    4525 generic.go:356] "Generic (PLEG): container finished" podID="67f662f4-cbe1-4762-9678-220149cc2eb2" containerID="acbc54293ceb9cc43d9cbd294ad19a5eba04ff2443138d843c01ec0e2a600d8e" exitCode=0
Oct 30 19:00:49.360061 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:49.359954    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6a327a-z5pnw" event={"ID":"67f662f4-cbe1-4762-9678-220149cc2eb2","Type":"ContainerDied","Data":"acbc54293ceb9cc43d9cbd294ad19a5eba04ff2443138d843c01ec0e2a600d8e"}
Oct 30 19:00:49.534057 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:49.533849    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9073694d-9dlpm"
Oct 30 19:00:49.612022 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:49.611916    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e47d550d-c724-482c-9352-077d0d9e17c9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e47d550d-c724-482c-9352-077d0d9e17c9-maas-api-xks-ca\") pod \"e47d550d-c724-482c-9352-077d0d9e17c9\" (UID: \"e47d550d-c724-482c-9352-077d0d9e17c9\") "
Oct 30 19:00:49.612223 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:49.612036    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e47d550d-c724-482c-9352-077d0d9e17c9-kube-api-access-xhjs5\" (UniqueName: \"kubernetes.io/projected/e47d550d-c724-482c-9352-077d0d9e17c9-kube-api-access-xhjs5\") pod \"e47d550d-c724-482c-9352-077d0d9e17c9\" (UID: \"e47d550d-c724-482c-9352-077d0d9e17c9\") "
Oct 30 19:00:49.612223 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:49.612208    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e47d550d-c724-482c-9352-077d0d9e17c9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e47d550d-c724-482c-9352-077d0d9e17c9-maas-api-openshift-ca\") pod \"e47d550d-c724-482c-9352-077d0d9e17c9\" (UID: \"e47d550d-c724-482c-9352-077d0d9e17c9\") "
Oct 30 19:00:49.612574 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:49.612548    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e47d550d-c724-482c-9352-077d0d9e17c9-maas-api-openshift-ca" pod "e47d550d-c724-482c-9352-077d0d9e17c9" (UID: "e47d550d-c724-482c-9352-077d0d9e17c9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:00:49.612907 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:49.612884    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e47d550d-c724-482c-9352-077d0d9e17c9-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:49.614296 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:49.614268    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e47d550d-c724-482c-9352-077d0d9e17c9-maas-api-xks-ca" pod "e47d550d-c724-482c-9352-077d0d9e17c9" (UID: "e47d550d-c724-482c-9352-077d0d9e17c9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:00:49.614416 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:49.614399    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e47d550d-c724-482c-9352-077d0d9e17c9-kube-api-access-xhjs5" pod "e47d550d-c724-482c-9352-077d0d9e17c9" (UID: "e47d550d-c724-482c-9352-077d0d9e17c9"). InnerVolumeSpecName "kube-api-access-xhjs5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:00:49.714847 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:49.714815    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e47d550d-c724-482c-9352-077d0d9e17c9-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:49.714847 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:49.714845    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhjs5\" (UniqueName: \"kubernetes.io/projected/e47d550d-c724-482c-9352-077d0d9e17c9-kube-api-access-xhjs5\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:50.369173 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:50.369145    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9073694d-9dlpm"
Oct 30 19:00:50.369633 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:50.369193    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9073694d-9dlpm" event={"ID":"e47d550d-c724-482c-9352-077d0d9e17c9","Type":"ContainerDied","Data":"59fdb1411b7bbf03ac477c41b3731d33e245949371683165155adf5b037efe49"}
Oct 30 19:00:50.369633 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:50.369257    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59fdb1411b7bbf03ac477c41b3731d33e245949371683165155adf5b037efe49"
Oct 30 19:00:50.575080 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:50.575054    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6a327a-z5pnw"
Oct 30 19:00:50.627773 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:50.627667    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67f662f4-cbe1-4762-9678-220149cc2eb2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67f662f4-cbe1-4762-9678-220149cc2eb2-maas-api-xks-ca\") pod \"67f662f4-cbe1-4762-9678-220149cc2eb2\" (UID: \"67f662f4-cbe1-4762-9678-220149cc2eb2\") "
Oct 30 19:00:50.627773 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:50.627746    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67f662f4-cbe1-4762-9678-220149cc2eb2-kube-api-access-flg5n\" (UniqueName: \"kubernetes.io/projected/67f662f4-cbe1-4762-9678-220149cc2eb2-kube-api-access-flg5n\") pod \"67f662f4-cbe1-4762-9678-220149cc2eb2\" (UID: \"67f662f4-cbe1-4762-9678-220149cc2eb2\") "
Oct 30 19:00:50.628006 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:50.627924    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67f662f4-cbe1-4762-9678-220149cc2eb2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67f662f4-cbe1-4762-9678-220149cc2eb2-maas-api-openshift-ca\") pod \"67f662f4-cbe1-4762-9678-220149cc2eb2\" (UID: \"67f662f4-cbe1-4762-9678-220149cc2eb2\") "
Oct 30 19:00:50.628383 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:50.628325    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67f662f4-cbe1-4762-9678-220149cc2eb2-maas-api-openshift-ca" pod "67f662f4-cbe1-4762-9678-220149cc2eb2" (UID: "67f662f4-cbe1-4762-9678-220149cc2eb2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:00:50.630637 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:50.630611    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67f662f4-cbe1-4762-9678-220149cc2eb2-maas-api-xks-ca" pod "67f662f4-cbe1-4762-9678-220149cc2eb2" (UID: "67f662f4-cbe1-4762-9678-220149cc2eb2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:00:50.630887 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:50.630750    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67f662f4-cbe1-4762-9678-220149cc2eb2-kube-api-access-flg5n" pod "67f662f4-cbe1-4762-9678-220149cc2eb2" (UID: "67f662f4-cbe1-4762-9678-220149cc2eb2"). InnerVolumeSpecName "kube-api-access-flg5n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:00:50.729703 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:50.729651    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67f662f4-cbe1-4762-9678-220149cc2eb2-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:50.729887 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:50.729727    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flg5n\" (UniqueName: \"kubernetes.io/projected/67f662f4-cbe1-4762-9678-220149cc2eb2-kube-api-access-flg5n\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:50.729887 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:50.729746    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67f662f4-cbe1-4762-9678-220149cc2eb2-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:51.379955 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:51.379919    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6a327a-z5pnw" event={"ID":"67f662f4-cbe1-4762-9678-220149cc2eb2","Type":"ContainerDied","Data":"f5e0e44b2d21c9a4d0b338b433b05913361bd7d7ff7cade36bb43d6691890b6d"}
Oct 30 19:00:51.379955 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:51.379951    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6a327a-z5pnw"
Oct 30 19:00:51.380444 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:51.379996    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f5e0e44b2d21c9a4d0b338b433b05913361bd7d7ff7cade36bb43d6691890b6d"
Oct 30 19:00:53.872071 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:53.872038    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-24fceb-gpqzf"]
Oct 30 19:00:53.876093 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:53.876066    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-24fceb-gpqzf"]
Oct 30 19:00:55.550181 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:00:55.550148    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00911e2d-63e4-47f1-8935-cd8dcb9bed54" path="/var/lib/kubelet/pods/00911e2d-63e4-47f1-8935-cd8dcb9bed54/volumes"
Oct 30 19:01:01.811763 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:01.811604    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td"]
Oct 30 19:01:01.812213 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:01.811917    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td" podUID="bce8221a-1318-4f64-908c-d927050e1ac1" containerName="istio-proxy" containerID="cri-o://8c7dee88d79b8b21cc93e60b2020ff2e7eddbbba8640bb638b4c76b7e68e12d5" gracePeriod=30
Oct 30 19:01:02.137841 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:02.137769    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr" podUID="82bbbf85-ed93-48d2-9f17-49ac1f31c0a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 30 19:01:02.148718 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:02.139187    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-6a327a/praxis-route-6a327a-kserve-5cb7997949-4vzjq" podUID="0dcb92f5-5de5-4c91-87cc-27764bea9683" containerName="main" probeResult="failure" output="Get \"https://10.133.0.149:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:02.186718 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:02.182778    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb" podUID="12ebc8b9-c794-47b5-b87b-a63c4bd993e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:02.227191 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:02.227129    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw" podUID="ac6852fd-01be-4d69-8c9b-0c591e50b889" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:02.311854 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:02.311806    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj" podUID="84c99c49-1d60-4176-bda4-cad29c7d801c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 19:01:02.398764 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:02.397815    4525 patch_prober.go:29] interesting pod/e2e-worker-w3-2889dd-openshift-default-84768d69f7-gtdr6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:01:02.398764 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:02.397904    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-gtdr6" podUID="01a5c107-dcf4-4ca5-8c9d-57c40814206d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:02.523899 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:02.523836    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts" podUID="be6d4969-285b-4da0-883e-4847f2cbf3af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:02.582914 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:02.582862    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t" podUID="424fca14-6d8a-4663-86ec-0c91165f34ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:02.584919 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:02.584886    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv" podUID="595230e3-55f3-43eb-9a7b-6d89250af9b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 19:01:03.147709 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:03.138012    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr" podUID="82bbbf85-ed93-48d2-9f17-49ac1f31c0a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 30 19:01:03.147709 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:03.138841    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr"
Oct 30 19:01:03.230723 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:03.139768    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-6a327a/praxis-route-6a327a-kserve-5cb7997949-4vzjq" podUID="0dcb92f5-5de5-4c91-87cc-27764bea9683" containerName="main" probeResult="failure" output="Get \"https://10.133.0.149:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:03.231183 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:03.231141    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw" podUID="ac6852fd-01be-4d69-8c9b-0c591e50b889" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:03.247708 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:03.205595    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb" podUID="12ebc8b9-c794-47b5-b87b-a63c4bd993e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:03.372192 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:03.251200    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-6a327a/praxis-route-6a327a-kserve-5cb7997949-4vzjq"
Oct 30 19:01:03.372533 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:03.372512    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw"
Oct 30 19:01:03.372753 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:03.372735    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb"
Oct 30 19:01:03.394844 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:03.343850    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj" podUID="84c99c49-1d60-4176-bda4-cad29c7d801c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:03.415716 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:03.395737    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj"
Oct 30 19:01:03.415716 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:03.395130    4525 patch_prober.go:29] interesting pod/e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:01:03.415716 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:03.396470    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j" podUID="6061ede7-981c-4c07-acfc-04be1cae252d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:03.528431 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:03.528376    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts" podUID="be6d4969-285b-4da0-883e-4847f2cbf3af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:03.529110 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:03.529087    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts"
Oct 30 19:01:03.593992 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:03.593927    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t" podUID="424fca14-6d8a-4663-86ec-0c91165f34ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:03.597715 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:03.595549    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv" podUID="595230e3-55f3-43eb-9a7b-6d89250af9b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:04.031454 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:04.031409    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t"
Oct 30 19:01:04.167533 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:04.167493    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv"
Oct 30 19:01:04.168308 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:04.168286    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-6a327a/praxis-route-6a327a-kserve-5cb7997949-4vzjq"
Oct 30 19:01:04.168507 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:04.168491    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb"
Oct 30 19:01:04.168665 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:04.168645    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv"
Oct 30 19:01:04.168845 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:04.168827    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts"
Oct 30 19:01:04.176940 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:04.176565    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr" podUID="82bbbf85-ed93-48d2-9f17-49ac1f31c0a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 30 19:01:04.178618 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:04.178595    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj"
Oct 30 19:01:04.228005 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:04.227975    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr"
Oct 30 19:01:04.283804 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:04.261486    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw" podUID="ac6852fd-01be-4d69-8c9b-0c591e50b889" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:04.363572 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:04.362989    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw"
Oct 30 19:01:04.366025 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:04.365990    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t"
Oct 30 19:01:04.535855 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:04.535767    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bc6e719-7xvb9"]
Oct 30 19:01:04.538281 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:04.536966    4525 cpu_manager.go:414] "containerMap: removing container" podUID="67f662f4-cbe1-4762-9678-220149cc2eb2" containerName="revoke-keys"
Oct 30 19:01:04.538491 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:04.538470    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67f662f4-cbe1-4762-9678-220149cc2eb2" containerName="revoke-keys"
Oct 30 19:01:04.538592 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:04.538581    4525 cpu_manager.go:414] "containerMap: removing container" podUID="e47d550d-c724-482c-9352-077d0d9e17c9" containerName="revoke-keys"
Oct 30 19:01:04.538666 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:04.538657    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e47d550d-c724-482c-9352-077d0d9e17c9" containerName="revoke-keys"
Oct 30 19:01:04.539587 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:04.539536    4525 cpu_manager.go:414] "containerMap: removing container" podUID="107a0724-9d25-4b33-9866-e689ceb95234" containerName="revoke-keys"
Oct 30 19:01:04.539792 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:04.539757    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="107a0724-9d25-4b33-9866-e689ceb95234" containerName="revoke-keys"
Oct 30 19:01:04.539969 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:04.539954    4525 cpu_manager.go:414] "containerMap: removing container" podUID="1b248c08-1780-454f-9ef9-596fb0cfb250" containerName="revoke-keys"
Oct 30 19:01:04.540066 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:04.540053    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b248c08-1780-454f-9ef9-596fb0cfb250" containerName="revoke-keys"
Oct 30 19:01:04.540368 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:04.540352    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="67f662f4-cbe1-4762-9678-220149cc2eb2" containerName="revoke-keys"
Oct 30 19:01:04.540486 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:04.540476    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="e47d550d-c724-482c-9352-077d0d9e17c9" containerName="revoke-keys"
Oct 30 19:01:04.540583 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:04.540574    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="107a0724-9d25-4b33-9866-e689ceb95234" containerName="revoke-keys"
Oct 30 19:01:04.540660 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:04.540652    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b248c08-1780-454f-9ef9-596fb0cfb250" containerName="revoke-keys"
Oct 30 19:01:04.554779 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:04.554741    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bc6e719-7xvb9"
Oct 30 19:01:04.568019 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:04.567996    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:01:04.568769 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:04.568294    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94hcn\""
Oct 30 19:01:04.568908 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:04.568355    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bc6e719-7xvb9"]
Oct 30 19:01:04.673564 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:04.673532    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpl9b\" (UniqueName: \"kubernetes.io/projected/d9aba5b9-263f-40b1-9543-ef813ebdc852-kube-api-access-qpl9b\") pod \"maas-api-revoke-sub-8bc6e719-7xvb9\" (UID: \"d9aba5b9-263f-40b1-9543-ef813ebdc852\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bc6e719-7xvb9"
Oct 30 19:01:04.673941 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:04.673911    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9aba5b9-263f-40b1-9543-ef813ebdc852-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8bc6e719-7xvb9\" (UID: \"d9aba5b9-263f-40b1-9543-ef813ebdc852\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bc6e719-7xvb9"
Oct 30 19:01:04.674256 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:04.674231    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9aba5b9-263f-40b1-9543-ef813ebdc852-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8bc6e719-7xvb9\" (UID: \"d9aba5b9-263f-40b1-9543-ef813ebdc852\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bc6e719-7xvb9"
Oct 30 19:01:04.775372 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:04.775337    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpl9b\" (UniqueName: \"kubernetes.io/projected/d9aba5b9-263f-40b1-9543-ef813ebdc852-kube-api-access-qpl9b\") pod \"maas-api-revoke-sub-8bc6e719-7xvb9\" (UID: \"d9aba5b9-263f-40b1-9543-ef813ebdc852\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bc6e719-7xvb9"
Oct 30 19:01:04.775553 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:04.775500    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9aba5b9-263f-40b1-9543-ef813ebdc852-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8bc6e719-7xvb9\" (UID: \"d9aba5b9-263f-40b1-9543-ef813ebdc852\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bc6e719-7xvb9"
Oct 30 19:01:04.776417 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:04.776389    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9aba5b9-263f-40b1-9543-ef813ebdc852-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8bc6e719-7xvb9\" (UID: \"d9aba5b9-263f-40b1-9543-ef813ebdc852\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bc6e719-7xvb9"
Oct 30 19:01:04.776622 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:04.776603    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9aba5b9-263f-40b1-9543-ef813ebdc852-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8bc6e719-7xvb9\" (UID: \"d9aba5b9-263f-40b1-9543-ef813ebdc852\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bc6e719-7xvb9"
Oct 30 19:01:04.779980 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:04.779513    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9aba5b9-263f-40b1-9543-ef813ebdc852-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8bc6e719-7xvb9\" (UID: \"d9aba5b9-263f-40b1-9543-ef813ebdc852\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bc6e719-7xvb9"
Oct 30 19:01:04.783857 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:04.783836    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpl9b\" (UniqueName: \"kubernetes.io/projected/d9aba5b9-263f-40b1-9543-ef813ebdc852-kube-api-access-qpl9b\") pod \"maas-api-revoke-sub-8bc6e719-7xvb9\" (UID: \"d9aba5b9-263f-40b1-9543-ef813ebdc852\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bc6e719-7xvb9"
Oct 30 19:01:04.876494 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:04.876460    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bc6e719-7xvb9"
Oct 30 19:01:05.022943 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:05.022906    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca415337-wwhn8"]
Oct 30 19:01:05.030306 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:05.030272    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca415337-wwhn8"
Oct 30 19:01:05.035191 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:05.035158    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca415337-wwhn8"]
Oct 30 19:01:05.081125 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:05.081089    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d59f7bbb-4eda-4e43-bb21-e2359fc2688c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca415337-wwhn8\" (UID: \"d59f7bbb-4eda-4e43-bb21-e2359fc2688c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca415337-wwhn8"
Oct 30 19:01:05.081125 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:05.081131    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d59f7bbb-4eda-4e43-bb21-e2359fc2688c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca415337-wwhn8\" (UID: \"d59f7bbb-4eda-4e43-bb21-e2359fc2688c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca415337-wwhn8"
Oct 30 19:01:05.081431 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:05.081402    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dn4xb\" (UniqueName: \"kubernetes.io/projected/d59f7bbb-4eda-4e43-bb21-e2359fc2688c-kube-api-access-dn4xb\") pod \"maas-api-revoke-sub-ca415337-wwhn8\" (UID: \"d59f7bbb-4eda-4e43-bb21-e2359fc2688c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca415337-wwhn8"
Oct 30 19:01:05.176651 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:05.175957    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bc6e719-7xvb9"]
Oct 30 19:01:05.184183 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:05.183595    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d59f7bbb-4eda-4e43-bb21-e2359fc2688c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca415337-wwhn8\" (UID: \"d59f7bbb-4eda-4e43-bb21-e2359fc2688c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca415337-wwhn8"
Oct 30 19:01:05.184183 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:05.183645    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d59f7bbb-4eda-4e43-bb21-e2359fc2688c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca415337-wwhn8\" (UID: \"d59f7bbb-4eda-4e43-bb21-e2359fc2688c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca415337-wwhn8"
Oct 30 19:01:05.184183 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:05.183875    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn4xb\" (UniqueName: \"kubernetes.io/projected/d59f7bbb-4eda-4e43-bb21-e2359fc2688c-kube-api-access-dn4xb\") pod \"maas-api-revoke-sub-ca415337-wwhn8\" (UID: \"d59f7bbb-4eda-4e43-bb21-e2359fc2688c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca415337-wwhn8"
Oct 30 19:01:05.184966 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:05.184939    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d59f7bbb-4eda-4e43-bb21-e2359fc2688c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca415337-wwhn8\" (UID: \"d59f7bbb-4eda-4e43-bb21-e2359fc2688c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca415337-wwhn8"
Oct 30 19:01:05.186915 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:05.186861    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d59f7bbb-4eda-4e43-bb21-e2359fc2688c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca415337-wwhn8\" (UID: \"d59f7bbb-4eda-4e43-bb21-e2359fc2688c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca415337-wwhn8"
Oct 30 19:01:05.194152 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:05.194124    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dn4xb\" (UniqueName: \"kubernetes.io/projected/d59f7bbb-4eda-4e43-bb21-e2359fc2688c-kube-api-access-dn4xb\") pod \"maas-api-revoke-sub-ca415337-wwhn8\" (UID: \"d59f7bbb-4eda-4e43-bb21-e2359fc2688c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca415337-wwhn8"
Oct 30 19:01:05.227940 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:05.227910    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bc6e719-7xvb9" event={"ID":"d9aba5b9-263f-40b1-9543-ef813ebdc852","Type":"ContainerStarted","Data":"b83501219f63e66c2d8be40f40d7e13aa933c7f28d90de1f51df32816abc72a3"}
Oct 30 19:01:05.350887 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:05.350849    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca415337-wwhn8"
Oct 30 19:01:05.637890 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:05.637858    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca415337-wwhn8"]
Oct 30 19:01:05.638351 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:01:05.638327    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd59f7bbb_4eda_4e43_bb21_e2359fc2688c.slice/crio-d019c40f425c4ed15b97ff19314bdea1814493753e30b6d1cb687a9a1b1eb776 WatchSource:0}: Error finding container d019c40f425c4ed15b97ff19314bdea1814493753e30b6d1cb687a9a1b1eb776: Status 404 returned error can't find the container with id d019c40f425c4ed15b97ff19314bdea1814493753e30b6d1cb687a9a1b1eb776
Oct 30 19:01:06.238229 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:06.238187    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca415337-wwhn8" event={"ID":"d59f7bbb-4eda-4e43-bb21-e2359fc2688c","Type":"ContainerStarted","Data":"4f7c188b7b4e5a51dc6d7099519dd4bc1436ec1781642754f3c9c3d29ca11009"}
Oct 30 19:01:06.238664 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:06.238248    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca415337-wwhn8" event={"ID":"d59f7bbb-4eda-4e43-bb21-e2359fc2688c","Type":"ContainerStarted","Data":"d019c40f425c4ed15b97ff19314bdea1814493753e30b6d1cb687a9a1b1eb776"}
Oct 30 19:01:06.239850 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:06.239824    4525 generic.go:356] "Generic (PLEG): container finished" podID="d9aba5b9-263f-40b1-9543-ef813ebdc852" containerID="aaeaeb3aa4f109147bd96c39e1544aa5951de9b7f5912e3adaeb28c56b78c883" exitCode=0
Oct 30 19:01:06.239850 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:06.239856    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bc6e719-7xvb9" event={"ID":"d9aba5b9-263f-40b1-9543-ef813ebdc852","Type":"ContainerDied","Data":"aaeaeb3aa4f109147bd96c39e1544aa5951de9b7f5912e3adaeb28c56b78c883"}
Oct 30 19:01:06.262173 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:06.262122    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca415337-wwhn8" podStartSLOduration=1.262085248 podStartE2EDuration="1.262085248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:01:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:01:06.257160139 +0000 UTC m=+1488.882830098" watchObservedRunningTime="2026-10-30 19:01:06.262085248 +0000 UTC m=+1488.887755203"
Oct 30 19:01:07.154549 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:07.154525    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td"
Oct 30 19:01:07.210487 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:07.210391    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bce8221a-1318-4f64-908c-d927050e1ac1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bce8221a-1318-4f64-908c-d927050e1ac1-credential-socket\") pod \"bce8221a-1318-4f64-908c-d927050e1ac1\" (UID: \"bce8221a-1318-4f64-908c-d927050e1ac1\") "
Oct 30 19:01:07.210653 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:07.210483    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bce8221a-1318-4f64-908c-d927050e1ac1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bce8221a-1318-4f64-908c-d927050e1ac1-istiod-ca-cert\") pod \"bce8221a-1318-4f64-908c-d927050e1ac1\" (UID: \"bce8221a-1318-4f64-908c-d927050e1ac1\") "
Oct 30 19:01:07.210653 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:07.210546    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bce8221a-1318-4f64-908c-d927050e1ac1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bce8221a-1318-4f64-908c-d927050e1ac1-workload-socket\") pod \"bce8221a-1318-4f64-908c-d927050e1ac1\" (UID: \"bce8221a-1318-4f64-908c-d927050e1ac1\") "
Oct 30 19:01:07.210653 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:07.210618    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bce8221a-1318-4f64-908c-d927050e1ac1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bce8221a-1318-4f64-908c-d927050e1ac1-istio-data\") pod \"bce8221a-1318-4f64-908c-d927050e1ac1\" (UID: \"bce8221a-1318-4f64-908c-d927050e1ac1\") "
Oct 30 19:01:07.210852 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:07.210649    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bce8221a-1318-4f64-908c-d927050e1ac1-credential-socket" pod "bce8221a-1318-4f64-908c-d927050e1ac1" (UID: "bce8221a-1318-4f64-908c-d927050e1ac1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:01:07.210852 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:07.210656    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bce8221a-1318-4f64-908c-d927050e1ac1-kube-api-access-f24h9\" (UniqueName: \"kubernetes.io/projected/bce8221a-1318-4f64-908c-d927050e1ac1-kube-api-access-f24h9\") pod \"bce8221a-1318-4f64-908c-d927050e1ac1\" (UID: \"bce8221a-1318-4f64-908c-d927050e1ac1\") "
Oct 30 19:01:07.210852 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:07.210825    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bce8221a-1318-4f64-908c-d927050e1ac1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bce8221a-1318-4f64-908c-d927050e1ac1-workload-certs\") pod \"bce8221a-1318-4f64-908c-d927050e1ac1\" (UID: \"bce8221a-1318-4f64-908c-d927050e1ac1\") "
Oct 30 19:01:07.211016 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:07.210917    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bce8221a-1318-4f64-908c-d927050e1ac1-istio-token\" (UniqueName: \"kubernetes.io/projected/bce8221a-1318-4f64-908c-d927050e1ac1-istio-token\") pod \"bce8221a-1318-4f64-908c-d927050e1ac1\" (UID: \"bce8221a-1318-4f64-908c-d927050e1ac1\") "
Oct 30 19:01:07.211016 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:07.210958    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bce8221a-1318-4f64-908c-d927050e1ac1-workload-socket" pod "bce8221a-1318-4f64-908c-d927050e1ac1" (UID: "bce8221a-1318-4f64-908c-d927050e1ac1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:01:07.211016 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:07.210963    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bce8221a-1318-4f64-908c-d927050e1ac1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bce8221a-1318-4f64-908c-d927050e1ac1-istio-podinfo\") pod \"bce8221a-1318-4f64-908c-d927050e1ac1\" (UID: \"bce8221a-1318-4f64-908c-d927050e1ac1\") "
Oct 30 19:01:07.211016 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:07.210982    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bce8221a-1318-4f64-908c-d927050e1ac1-istio-data" pod "bce8221a-1318-4f64-908c-d927050e1ac1" (UID: "bce8221a-1318-4f64-908c-d927050e1ac1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:01:07.211225 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:07.211020    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bce8221a-1318-4f64-908c-d927050e1ac1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bce8221a-1318-4f64-908c-d927050e1ac1-istio-envoy\") pod \"bce8221a-1318-4f64-908c-d927050e1ac1\" (UID: \"bce8221a-1318-4f64-908c-d927050e1ac1\") "
Oct 30 19:01:07.211348 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:07.211020    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bce8221a-1318-4f64-908c-d927050e1ac1-istiod-ca-cert" pod "bce8221a-1318-4f64-908c-d927050e1ac1" (UID: "bce8221a-1318-4f64-908c-d927050e1ac1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:01:07.211466 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:07.211244    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bce8221a-1318-4f64-908c-d927050e1ac1-workload-certs" pod "bce8221a-1318-4f64-908c-d927050e1ac1" (UID: "bce8221a-1318-4f64-908c-d927050e1ac1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:01:07.212076 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:07.212051    4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bce8221a-1318-4f64-908c-d927050e1ac1-credential-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:07.212182 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:07.212080    4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bce8221a-1318-4f64-908c-d927050e1ac1-istiod-ca-cert\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:07.212182 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:07.212096    4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bce8221a-1318-4f64-908c-d927050e1ac1-workload-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:07.212182 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:07.212111    4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bce8221a-1318-4f64-908c-d927050e1ac1-istio-data\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:07.212182 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:07.212128    4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bce8221a-1318-4f64-908c-d927050e1ac1-workload-certs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:07.213360 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:07.213337    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bce8221a-1318-4f64-908c-d927050e1ac1-istio-token" pod "bce8221a-1318-4f64-908c-d927050e1ac1" (UID: "bce8221a-1318-4f64-908c-d927050e1ac1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:01:07.213504 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:07.213384    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bce8221a-1318-4f64-908c-d927050e1ac1-istio-podinfo" pod "bce8221a-1318-4f64-908c-d927050e1ac1" (UID: "bce8221a-1318-4f64-908c-d927050e1ac1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:01:07.213650 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:07.213630    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bce8221a-1318-4f64-908c-d927050e1ac1-kube-api-access-f24h9" pod "bce8221a-1318-4f64-908c-d927050e1ac1" (UID: "bce8221a-1318-4f64-908c-d927050e1ac1"). InnerVolumeSpecName "kube-api-access-f24h9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:01:07.213843 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:07.213819    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bce8221a-1318-4f64-908c-d927050e1ac1-istio-envoy" pod "bce8221a-1318-4f64-908c-d927050e1ac1" (UID: "bce8221a-1318-4f64-908c-d927050e1ac1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:01:07.251497 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:07.251462    4525 generic.go:356] "Generic (PLEG): container finished" podID="bce8221a-1318-4f64-908c-d927050e1ac1" containerID="8c7dee88d79b8b21cc93e60b2020ff2e7eddbbba8640bb638b4c76b7e68e12d5" exitCode=0
Oct 30 19:01:07.251993 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:07.251519    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td"
Oct 30 19:01:07.251993 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:07.251542    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td" event={"ID":"bce8221a-1318-4f64-908c-d927050e1ac1","Type":"ContainerDied","Data":"8c7dee88d79b8b21cc93e60b2020ff2e7eddbbba8640bb638b4c76b7e68e12d5"}
Oct 30 19:01:07.251993 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:07.251588    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td" event={"ID":"bce8221a-1318-4f64-908c-d927050e1ac1","Type":"ContainerDied","Data":"28a17a675a4c4b80505ab34df5db5b9b2a5d0d17301ad5cec2306d03367869d0"}
Oct 30 19:01:07.251993 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:07.251610    4525 scope.go:122] "RemoveContainer" containerID="8c7dee88d79b8b21cc93e60b2020ff2e7eddbbba8640bb638b4c76b7e68e12d5"
Oct 30 19:01:07.253588 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:07.253563    4525 generic.go:356] "Generic (PLEG): container finished" podID="d59f7bbb-4eda-4e43-bb21-e2359fc2688c" containerID="4f7c188b7b4e5a51dc6d7099519dd4bc1436ec1781642754f3c9c3d29ca11009" exitCode=0
Oct 30 19:01:07.253718 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:07.253644    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca415337-wwhn8" event={"ID":"d59f7bbb-4eda-4e43-bb21-e2359fc2688c","Type":"ContainerDied","Data":"4f7c188b7b4e5a51dc6d7099519dd4bc1436ec1781642754f3c9c3d29ca11009"}
Oct 30 19:01:07.264757 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:07.264737    4525 scope.go:122] "RemoveContainer" containerID="8c925841a89466566a44959880d802bb2800584d1ab4b27d6b60ceb240d95868"
Oct 30 19:01:07.277469 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:07.277440    4525 scope.go:122] "RemoveContainer" containerID="8c7dee88d79b8b21cc93e60b2020ff2e7eddbbba8640bb638b4c76b7e68e12d5"
Oct 30 19:01:07.277897 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:01:07.277864    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c7dee88d79b8b21cc93e60b2020ff2e7eddbbba8640bb638b4c76b7e68e12d5\": container with ID starting with 8c7dee88d79b8b21cc93e60b2020ff2e7eddbbba8640bb638b4c76b7e68e12d5 not found: ID does not exist" containerID="8c7dee88d79b8b21cc93e60b2020ff2e7eddbbba8640bb638b4c76b7e68e12d5"
Oct 30 19:01:07.278009 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:07.277910    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c7dee88d79b8b21cc93e60b2020ff2e7eddbbba8640bb638b4c76b7e68e12d5"} err="failed to get container status \"8c7dee88d79b8b21cc93e60b2020ff2e7eddbbba8640bb638b4c76b7e68e12d5\": rpc error: code = NotFound desc = could not find container \"8c7dee88d79b8b21cc93e60b2020ff2e7eddbbba8640bb638b4c76b7e68e12d5\": container with ID starting with 8c7dee88d79b8b21cc93e60b2020ff2e7eddbbba8640bb638b4c76b7e68e12d5 not found: ID does not exist"
Oct 30 19:01:07.278009 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:07.277934    4525 scope.go:122] "RemoveContainer" containerID="8c925841a89466566a44959880d802bb2800584d1ab4b27d6b60ceb240d95868"
Oct 30 19:01:07.278273 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:01:07.278252    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c925841a89466566a44959880d802bb2800584d1ab4b27d6b60ceb240d95868\": container with ID starting with 8c925841a89466566a44959880d802bb2800584d1ab4b27d6b60ceb240d95868 not found: ID does not exist" containerID="8c925841a89466566a44959880d802bb2800584d1ab4b27d6b60ceb240d95868"
Oct 30 19:01:07.278324 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:07.278281    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c925841a89466566a44959880d802bb2800584d1ab4b27d6b60ceb240d95868"} err="failed to get container status \"8c925841a89466566a44959880d802bb2800584d1ab4b27d6b60ceb240d95868\": rpc error: code = NotFound desc = could not find container \"8c925841a89466566a44959880d802bb2800584d1ab4b27d6b60ceb240d95868\": container with ID starting with 8c925841a89466566a44959880d802bb2800584d1ab4b27d6b60ceb240d95868 not found: ID does not exist"
Oct 30 19:01:07.291631 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:07.291601    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td"]
Oct 30 19:01:07.295866 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:07.295838    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-24fceb-openshift-default-797c7dc6fc-hs9td"]
Oct 30 19:01:07.313778 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:07.313746    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f24h9\" (UniqueName: \"kubernetes.io/projected/bce8221a-1318-4f64-908c-d927050e1ac1-kube-api-access-f24h9\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:07.313778 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:07.313776    4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bce8221a-1318-4f64-908c-d927050e1ac1-istio-token\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:07.314011 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:07.313788    4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bce8221a-1318-4f64-908c-d927050e1ac1-istio-podinfo\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:07.314011 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:07.313797    4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bce8221a-1318-4f64-908c-d927050e1ac1-istio-envoy\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:07.433813 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:07.433790    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bc6e719-7xvb9"
Oct 30 19:01:07.516224 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:07.516136    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9aba5b9-263f-40b1-9543-ef813ebdc852-kube-api-access-qpl9b\" (UniqueName: \"kubernetes.io/projected/d9aba5b9-263f-40b1-9543-ef813ebdc852-kube-api-access-qpl9b\") pod \"d9aba5b9-263f-40b1-9543-ef813ebdc852\" (UID: \"d9aba5b9-263f-40b1-9543-ef813ebdc852\") "
Oct 30 19:01:07.516378 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:07.516329    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9aba5b9-263f-40b1-9543-ef813ebdc852-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9aba5b9-263f-40b1-9543-ef813ebdc852-maas-api-xks-ca\") pod \"d9aba5b9-263f-40b1-9543-ef813ebdc852\" (UID: \"d9aba5b9-263f-40b1-9543-ef813ebdc852\") "
Oct 30 19:01:07.516417 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:07.516391    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9aba5b9-263f-40b1-9543-ef813ebdc852-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9aba5b9-263f-40b1-9543-ef813ebdc852-maas-api-openshift-ca\") pod \"d9aba5b9-263f-40b1-9543-ef813ebdc852\" (UID: \"d9aba5b9-263f-40b1-9543-ef813ebdc852\") "
Oct 30 19:01:07.516851 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:07.516821    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9aba5b9-263f-40b1-9543-ef813ebdc852-maas-api-openshift-ca" pod "d9aba5b9-263f-40b1-9543-ef813ebdc852" (UID: "d9aba5b9-263f-40b1-9543-ef813ebdc852"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:01:07.518727 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:07.518695    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9aba5b9-263f-40b1-9543-ef813ebdc852-maas-api-xks-ca" pod "d9aba5b9-263f-40b1-9543-ef813ebdc852" (UID: "d9aba5b9-263f-40b1-9543-ef813ebdc852"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:01:07.518727 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:07.518712    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9aba5b9-263f-40b1-9543-ef813ebdc852-kube-api-access-qpl9b" pod "d9aba5b9-263f-40b1-9543-ef813ebdc852" (UID: "d9aba5b9-263f-40b1-9543-ef813ebdc852"). InnerVolumeSpecName "kube-api-access-qpl9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:01:07.545139 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:07.545109    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bce8221a-1318-4f64-908c-d927050e1ac1" path="/var/lib/kubelet/pods/bce8221a-1318-4f64-908c-d927050e1ac1/volumes"
Oct 30 19:01:07.618570 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:07.618535    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9aba5b9-263f-40b1-9543-ef813ebdc852-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:07.618570 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:07.618563    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpl9b\" (UniqueName: \"kubernetes.io/projected/d9aba5b9-263f-40b1-9543-ef813ebdc852-kube-api-access-qpl9b\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:07.618570 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:07.618574    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9aba5b9-263f-40b1-9543-ef813ebdc852-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:08.273944 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:08.273859    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bc6e719-7xvb9" event={"ID":"d9aba5b9-263f-40b1-9543-ef813ebdc852","Type":"ContainerDied","Data":"b83501219f63e66c2d8be40f40d7e13aa933c7f28d90de1f51df32816abc72a3"}
Oct 30 19:01:08.273944 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:08.273893    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bc6e719-7xvb9"
Oct 30 19:01:08.273944 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:08.273937    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b83501219f63e66c2d8be40f40d7e13aa933c7f28d90de1f51df32816abc72a3"
Oct 30 19:01:08.423289 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:08.423258    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca415337-wwhn8"
Oct 30 19:01:08.529234 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:08.529148    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d59f7bbb-4eda-4e43-bb21-e2359fc2688c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d59f7bbb-4eda-4e43-bb21-e2359fc2688c-maas-api-xks-ca\") pod \"d59f7bbb-4eda-4e43-bb21-e2359fc2688c\" (UID: \"d59f7bbb-4eda-4e43-bb21-e2359fc2688c\") "
Oct 30 19:01:08.529234 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:08.529227    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d59f7bbb-4eda-4e43-bb21-e2359fc2688c-kube-api-access-dn4xb\" (UniqueName: \"kubernetes.io/projected/d59f7bbb-4eda-4e43-bb21-e2359fc2688c-kube-api-access-dn4xb\") pod \"d59f7bbb-4eda-4e43-bb21-e2359fc2688c\" (UID: \"d59f7bbb-4eda-4e43-bb21-e2359fc2688c\") "
Oct 30 19:01:08.529441 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:08.529256    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d59f7bbb-4eda-4e43-bb21-e2359fc2688c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d59f7bbb-4eda-4e43-bb21-e2359fc2688c-maas-api-openshift-ca\") pod \"d59f7bbb-4eda-4e43-bb21-e2359fc2688c\" (UID: \"d59f7bbb-4eda-4e43-bb21-e2359fc2688c\") "
Oct 30 19:01:08.529722 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:08.529696    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d59f7bbb-4eda-4e43-bb21-e2359fc2688c-maas-api-openshift-ca" pod "d59f7bbb-4eda-4e43-bb21-e2359fc2688c" (UID: "d59f7bbb-4eda-4e43-bb21-e2359fc2688c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:01:08.531459 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:08.531434    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d59f7bbb-4eda-4e43-bb21-e2359fc2688c-maas-api-xks-ca" pod "d59f7bbb-4eda-4e43-bb21-e2359fc2688c" (UID: "d59f7bbb-4eda-4e43-bb21-e2359fc2688c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:01:08.531671 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:08.531650    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d59f7bbb-4eda-4e43-bb21-e2359fc2688c-kube-api-access-dn4xb" pod "d59f7bbb-4eda-4e43-bb21-e2359fc2688c" (UID: "d59f7bbb-4eda-4e43-bb21-e2359fc2688c"). InnerVolumeSpecName "kube-api-access-dn4xb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:01:08.630751 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:08.630715    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d59f7bbb-4eda-4e43-bb21-e2359fc2688c-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:08.630751 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:08.630746    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dn4xb\" (UniqueName: \"kubernetes.io/projected/d59f7bbb-4eda-4e43-bb21-e2359fc2688c-kube-api-access-dn4xb\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:08.631013 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:08.630762    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d59f7bbb-4eda-4e43-bb21-e2359fc2688c-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:09.153561 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:09.153526    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f4588e8-qs7k2"]
Oct 30 19:01:09.154127 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:09.154109    4525 cpu_manager.go:414] "containerMap: removing container" podUID="bce8221a-1318-4f64-908c-d927050e1ac1" containerName="istio-proxy"
Oct 30 19:01:09.154179 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:09.154129    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bce8221a-1318-4f64-908c-d927050e1ac1" containerName="istio-proxy"
Oct 30 19:01:09.154179 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:09.154151    4525 cpu_manager.go:414] "containerMap: removing container" podUID="bce8221a-1318-4f64-908c-d927050e1ac1" containerName="istio-proxy"
Oct 30 19:01:09.154179 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:09.154156    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bce8221a-1318-4f64-908c-d927050e1ac1" containerName="istio-proxy"
Oct 30 19:01:09.154179 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:09.154174    4525 cpu_manager.go:414] "containerMap: removing container" podUID="d59f7bbb-4eda-4e43-bb21-e2359fc2688c" containerName="revoke-keys"
Oct 30 19:01:09.154179 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:09.154179    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d59f7bbb-4eda-4e43-bb21-e2359fc2688c" containerName="revoke-keys"
Oct 30 19:01:09.154337 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:09.154195    4525 cpu_manager.go:414] "containerMap: removing container" podUID="d9aba5b9-263f-40b1-9543-ef813ebdc852" containerName="revoke-keys"
Oct 30 19:01:09.154337 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:09.154200    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9aba5b9-263f-40b1-9543-ef813ebdc852" containerName="revoke-keys"
Oct 30 19:01:09.154337 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:09.154278    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="d59f7bbb-4eda-4e43-bb21-e2359fc2688c" containerName="revoke-keys"
Oct 30 19:01:09.154337 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:09.154289    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9aba5b9-263f-40b1-9543-ef813ebdc852" containerName="revoke-keys"
Oct 30 19:01:09.154337 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:09.154296    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="bce8221a-1318-4f64-908c-d927050e1ac1" containerName="istio-proxy"
Oct 30 19:01:09.154337 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:09.154304    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="bce8221a-1318-4f64-908c-d927050e1ac1" containerName="istio-proxy"
Oct 30 19:01:09.157988 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:09.157963    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f4588e8-qs7k2"
Oct 30 19:01:09.166481 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:09.166453    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f4588e8-qs7k2"]
Oct 30 19:01:09.239416 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:09.239376    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c4c61cc-e88a-405f-908d-ae577af9b93a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f4588e8-qs7k2\" (UID: \"8c4c61cc-e88a-405f-908d-ae577af9b93a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f4588e8-qs7k2"
Oct 30 19:01:09.239655 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:09.239633    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c4c61cc-e88a-405f-908d-ae577af9b93a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f4588e8-qs7k2\" (UID: \"8c4c61cc-e88a-405f-908d-ae577af9b93a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f4588e8-qs7k2"
Oct 30 19:01:09.239738 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:09.239703    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wksg\" (UniqueName: \"kubernetes.io/projected/8c4c61cc-e88a-405f-908d-ae577af9b93a-kube-api-access-8wksg\") pod \"maas-api-revoke-sub-4f4588e8-qs7k2\" (UID: \"8c4c61cc-e88a-405f-908d-ae577af9b93a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f4588e8-qs7k2"
Oct 30 19:01:09.286937 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:09.286904    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca415337-wwhn8" event={"ID":"d59f7bbb-4eda-4e43-bb21-e2359fc2688c","Type":"ContainerDied","Data":"d019c40f425c4ed15b97ff19314bdea1814493753e30b6d1cb687a9a1b1eb776"}
Oct 30 19:01:09.287438 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:09.286963    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d019c40f425c4ed15b97ff19314bdea1814493753e30b6d1cb687a9a1b1eb776"
Oct 30 19:01:09.287438 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:09.287035    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca415337-wwhn8"
Oct 30 19:01:09.341130 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:09.341093    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c4c61cc-e88a-405f-908d-ae577af9b93a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f4588e8-qs7k2\" (UID: \"8c4c61cc-e88a-405f-908d-ae577af9b93a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f4588e8-qs7k2"
Oct 30 19:01:09.341318 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:09.341302    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c4c61cc-e88a-405f-908d-ae577af9b93a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f4588e8-qs7k2\" (UID: \"8c4c61cc-e88a-405f-908d-ae577af9b93a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f4588e8-qs7k2"
Oct 30 19:01:09.341367 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:09.341342    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wksg\" (UniqueName: \"kubernetes.io/projected/8c4c61cc-e88a-405f-908d-ae577af9b93a-kube-api-access-8wksg\") pod \"maas-api-revoke-sub-4f4588e8-qs7k2\" (UID: \"8c4c61cc-e88a-405f-908d-ae577af9b93a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f4588e8-qs7k2"
Oct 30 19:01:09.342049 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:09.342027    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c4c61cc-e88a-405f-908d-ae577af9b93a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f4588e8-qs7k2\" (UID: \"8c4c61cc-e88a-405f-908d-ae577af9b93a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f4588e8-qs7k2"
Oct 30 19:01:09.343648 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:09.343619    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c4c61cc-e88a-405f-908d-ae577af9b93a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f4588e8-qs7k2\" (UID: \"8c4c61cc-e88a-405f-908d-ae577af9b93a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f4588e8-qs7k2"
Oct 30 19:01:09.349431 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:09.349406    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wksg\" (UniqueName: \"kubernetes.io/projected/8c4c61cc-e88a-405f-908d-ae577af9b93a-kube-api-access-8wksg\") pod \"maas-api-revoke-sub-4f4588e8-qs7k2\" (UID: \"8c4c61cc-e88a-405f-908d-ae577af9b93a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f4588e8-qs7k2"
Oct 30 19:01:09.477313 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:09.477227    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f4588e8-qs7k2"
Oct 30 19:01:09.765513 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:09.765482    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f4588e8-qs7k2"]
Oct 30 19:01:09.765643 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:01:09.765485    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c4c61cc_e88a_405f_908d_ae577af9b93a.slice/crio-beddba287e8690686d961bf03ec489889d41ff354da143093684f5d18823a8ed WatchSource:0}: Error finding container beddba287e8690686d961bf03ec489889d41ff354da143093684f5d18823a8ed: Status 404 returned error can't find the container with id beddba287e8690686d961bf03ec489889d41ff354da143093684f5d18823a8ed
Oct 30 19:01:10.296278 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:10.296245    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f4588e8-qs7k2" event={"ID":"8c4c61cc-e88a-405f-908d-ae577af9b93a","Type":"ContainerStarted","Data":"c99b64138a16b37233f5a8674187e9132786288e6c839c483a9ef40ae4d3825b"}
Oct 30 19:01:10.296740 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:10.296307    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f4588e8-qs7k2" event={"ID":"8c4c61cc-e88a-405f-908d-ae577af9b93a","Type":"ContainerStarted","Data":"beddba287e8690686d961bf03ec489889d41ff354da143093684f5d18823a8ed"}
Oct 30 19:01:10.329842 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:10.329773    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f4588e8-qs7k2" podStartSLOduration=1.329656285 podStartE2EDuration="1.329656285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:01:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:01:10.318768394 +0000 UTC m=+1492.944438361" watchObservedRunningTime="2026-10-30 19:01:10.329656285 +0000 UTC m=+1492.955326325"
Oct 30 19:01:11.306933 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:11.306897    4525 generic.go:356] "Generic (PLEG): container finished" podID="8c4c61cc-e88a-405f-908d-ae577af9b93a" containerID="c99b64138a16b37233f5a8674187e9132786288e6c839c483a9ef40ae4d3825b" exitCode=0
Oct 30 19:01:11.307393 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:11.307026    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f4588e8-qs7k2" event={"ID":"8c4c61cc-e88a-405f-908d-ae577af9b93a","Type":"ContainerDied","Data":"c99b64138a16b37233f5a8674187e9132786288e6c839c483a9ef40ae4d3825b"}
Oct 30 19:01:12.472407 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:12.472383    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f4588e8-qs7k2"
Oct 30 19:01:12.595916 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:12.595829    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c4c61cc-e88a-405f-908d-ae577af9b93a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c4c61cc-e88a-405f-908d-ae577af9b93a-maas-api-xks-ca\") pod \"8c4c61cc-e88a-405f-908d-ae577af9b93a\" (UID: \"8c4c61cc-e88a-405f-908d-ae577af9b93a\") "
Oct 30 19:01:12.596119 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:12.595960    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c4c61cc-e88a-405f-908d-ae577af9b93a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c4c61cc-e88a-405f-908d-ae577af9b93a-maas-api-openshift-ca\") pod \"8c4c61cc-e88a-405f-908d-ae577af9b93a\" (UID: \"8c4c61cc-e88a-405f-908d-ae577af9b93a\") "
Oct 30 19:01:12.596119 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:12.596020    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c4c61cc-e88a-405f-908d-ae577af9b93a-kube-api-access-8wksg\" (UniqueName: \"kubernetes.io/projected/8c4c61cc-e88a-405f-908d-ae577af9b93a-kube-api-access-8wksg\") pod \"8c4c61cc-e88a-405f-908d-ae577af9b93a\" (UID: \"8c4c61cc-e88a-405f-908d-ae577af9b93a\") "
Oct 30 19:01:12.596392 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:12.596363    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c4c61cc-e88a-405f-908d-ae577af9b93a-maas-api-openshift-ca" pod "8c4c61cc-e88a-405f-908d-ae577af9b93a" (UID: "8c4c61cc-e88a-405f-908d-ae577af9b93a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:01:12.596921 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:12.596898    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c4c61cc-e88a-405f-908d-ae577af9b93a-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:12.598360 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:12.598335    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c4c61cc-e88a-405f-908d-ae577af9b93a-maas-api-xks-ca" pod "8c4c61cc-e88a-405f-908d-ae577af9b93a" (UID: "8c4c61cc-e88a-405f-908d-ae577af9b93a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:01:12.598871 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:12.598841    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c4c61cc-e88a-405f-908d-ae577af9b93a-kube-api-access-8wksg" pod "8c4c61cc-e88a-405f-908d-ae577af9b93a" (UID: "8c4c61cc-e88a-405f-908d-ae577af9b93a"). InnerVolumeSpecName "kube-api-access-8wksg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:01:12.698129 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:12.698097    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c4c61cc-e88a-405f-908d-ae577af9b93a-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:12.698129 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:12.698127    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wksg\" (UniqueName: \"kubernetes.io/projected/8c4c61cc-e88a-405f-908d-ae577af9b93a-kube-api-access-8wksg\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:13.330536 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:13.330505    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f4588e8-qs7k2"
Oct 30 19:01:13.330732 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:13.330537    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f4588e8-qs7k2" event={"ID":"8c4c61cc-e88a-405f-908d-ae577af9b93a","Type":"ContainerDied","Data":"beddba287e8690686d961bf03ec489889d41ff354da143093684f5d18823a8ed"}
Oct 30 19:01:13.330732 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:13.330592    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="beddba287e8690686d961bf03ec489889d41ff354da143093684f5d18823a8ed"
Oct 30 19:01:15.498887 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:15.498855    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c5f4bd6-9vh7q"]
Oct 30 19:01:15.499666 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:15.499646    4525 cpu_manager.go:414] "containerMap: removing container" podUID="8c4c61cc-e88a-405f-908d-ae577af9b93a" containerName="revoke-keys"
Oct 30 19:01:15.499821 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:15.499670    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c4c61cc-e88a-405f-908d-ae577af9b93a" containerName="revoke-keys"
Oct 30 19:01:15.499882 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:15.499874    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c4c61cc-e88a-405f-908d-ae577af9b93a" containerName="revoke-keys"
Oct 30 19:01:15.503708 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:15.503674    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c5f4bd6-9vh7q"
Oct 30 19:01:15.507939 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:15.507913    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:01:15.508107 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:15.508040    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94hcn\""
Oct 30 19:01:15.513745 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:15.513725    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c5f4bd6-9vh7q"]
Oct 30 19:01:15.638447 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:15.638412    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2865be82-0924-44fa-b215-28622b0957d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c5f4bd6-9vh7q\" (UID: \"2865be82-0924-44fa-b215-28622b0957d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c5f4bd6-9vh7q"
Oct 30 19:01:15.638651 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:15.638627    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2865be82-0924-44fa-b215-28622b0957d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c5f4bd6-9vh7q\" (UID: \"2865be82-0924-44fa-b215-28622b0957d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c5f4bd6-9vh7q"
Oct 30 19:01:15.638826 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:15.638806    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tdjj\" (UniqueName: \"kubernetes.io/projected/2865be82-0924-44fa-b215-28622b0957d3-kube-api-access-5tdjj\") pod \"maas-api-revoke-sub-0c5f4bd6-9vh7q\" (UID: \"2865be82-0924-44fa-b215-28622b0957d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c5f4bd6-9vh7q"
Oct 30 19:01:15.740135 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:15.740100    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tdjj\" (UniqueName: \"kubernetes.io/projected/2865be82-0924-44fa-b215-28622b0957d3-kube-api-access-5tdjj\") pod \"maas-api-revoke-sub-0c5f4bd6-9vh7q\" (UID: \"2865be82-0924-44fa-b215-28622b0957d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c5f4bd6-9vh7q"
Oct 30 19:01:15.740298 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:15.740217    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2865be82-0924-44fa-b215-28622b0957d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c5f4bd6-9vh7q\" (UID: \"2865be82-0924-44fa-b215-28622b0957d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c5f4bd6-9vh7q"
Oct 30 19:01:15.740342 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:15.740305    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2865be82-0924-44fa-b215-28622b0957d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c5f4bd6-9vh7q\" (UID: \"2865be82-0924-44fa-b215-28622b0957d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c5f4bd6-9vh7q"
Oct 30 19:01:15.740953 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:15.740934    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2865be82-0924-44fa-b215-28622b0957d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c5f4bd6-9vh7q\" (UID: \"2865be82-0924-44fa-b215-28622b0957d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c5f4bd6-9vh7q"
Oct 30 19:01:15.742731 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:15.742706    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2865be82-0924-44fa-b215-28622b0957d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c5f4bd6-9vh7q\" (UID: \"2865be82-0924-44fa-b215-28622b0957d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c5f4bd6-9vh7q"
Oct 30 19:01:15.748463 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:15.748436    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tdjj\" (UniqueName: \"kubernetes.io/projected/2865be82-0924-44fa-b215-28622b0957d3-kube-api-access-5tdjj\") pod \"maas-api-revoke-sub-0c5f4bd6-9vh7q\" (UID: \"2865be82-0924-44fa-b215-28622b0957d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c5f4bd6-9vh7q"
Oct 30 19:01:15.823881 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:15.823800    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c5f4bd6-9vh7q"
Oct 30 19:01:16.113851 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:01:16.113812    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2865be82_0924_44fa_b215_28622b0957d3.slice/crio-3c144d26d1e9c975c2a6242cf7f6a896433e007831e90a6eb02215bdf3006704 WatchSource:0}: Error finding container 3c144d26d1e9c975c2a6242cf7f6a896433e007831e90a6eb02215bdf3006704: Status 404 returned error can't find the container with id 3c144d26d1e9c975c2a6242cf7f6a896433e007831e90a6eb02215bdf3006704
Oct 30 19:01:16.114773 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:16.114745    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c5f4bd6-9vh7q"]
Oct 30 19:01:16.356663 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:16.356632    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c5f4bd6-9vh7q" event={"ID":"2865be82-0924-44fa-b215-28622b0957d3","Type":"ContainerStarted","Data":"3c144d26d1e9c975c2a6242cf7f6a896433e007831e90a6eb02215bdf3006704"}
Oct 30 19:01:17.417116 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:17.417060    4525 generic.go:356] "Generic (PLEG): container finished" podID="2865be82-0924-44fa-b215-28622b0957d3" containerID="15a05a403aa48a93970925ee05c193139594e86595bc6b8d4429aaa68f79d782" exitCode=0
Oct 30 19:01:17.418867 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:17.418839    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c5f4bd6-9vh7q" event={"ID":"2865be82-0924-44fa-b215-28622b0957d3","Type":"ContainerDied","Data":"15a05a403aa48a93970925ee05c193139594e86595bc6b8d4429aaa68f79d782"}
Oct 30 19:01:17.790405 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:01:17.790359    4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d49l" containerName="extractor"
Oct 30 19:01:18.065889 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:18.065773    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb" podUID="12ebc8b9-c794-47b5-b87b-a63c4bd993e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:18.072879 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:18.072840    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-6a327a/praxis-route-6a327a-kserve-5cb7997949-4vzjq" podUID="0dcb92f5-5de5-4c91-87cc-27764bea9683" containerName="main" probeResult="failure" output="Get \"https://10.133.0.149:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:18.152704 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:18.151817    4525 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj" podUID="84c99c49-1d60-4176-bda4-cad29c7d801c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:18.182919 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:18.182869    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj" podUID="84c99c49-1d60-4176-bda4-cad29c7d801c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:18.184339 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:18.184308    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr" podUID="82bbbf85-ed93-48d2-9f17-49ac1f31c0a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:18.188110 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:18.188079    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts" podUID="be6d4969-285b-4da0-883e-4847f2cbf3af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Oct 30 19:01:18.274429 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:18.274379    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw" podUID="ac6852fd-01be-4d69-8c9b-0c591e50b889" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 19:01:19.155200 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:19.112848    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-6a327a/praxis-route-6a327a-kserve-5cb7997949-4vzjq" podUID="0dcb92f5-5de5-4c91-87cc-27764bea9683" containerName="main" probeResult="failure" output="Get \"https://10.133.0.149:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:19.155200 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:19.113304    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-6a327a/praxis-route-6a327a-kserve-5cb7997949-4vzjq"
Oct 30 19:01:19.210733 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:19.208548    4525 scope.go:122] "RemoveContainer" containerID="a2f75a78c298b0d507653d2da7cc549b9dd7c8b53e00df6576a6926cf4d8d92e"
Oct 30 19:01:19.260138 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:19.259408    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-6a327a/praxis-route-6a327a-kserve-5cb7997949-4vzjq"
Oct 30 19:01:19.287623 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:19.287276    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c5f4bd6-9vh7q"
Oct 30 19:01:19.354721 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:19.347114    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2865be82-0924-44fa-b215-28622b0957d3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2865be82-0924-44fa-b215-28622b0957d3-maas-api-xks-ca\") pod \"2865be82-0924-44fa-b215-28622b0957d3\" (UID: \"2865be82-0924-44fa-b215-28622b0957d3\") "
Oct 30 19:01:19.359847 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:19.359804    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2865be82-0924-44fa-b215-28622b0957d3-kube-api-access-5tdjj\" (UniqueName: \"kubernetes.io/projected/2865be82-0924-44fa-b215-28622b0957d3-kube-api-access-5tdjj\") pod \"2865be82-0924-44fa-b215-28622b0957d3\" (UID: \"2865be82-0924-44fa-b215-28622b0957d3\") "
Oct 30 19:01:19.360026 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:19.359853    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2865be82-0924-44fa-b215-28622b0957d3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2865be82-0924-44fa-b215-28622b0957d3-maas-api-openshift-ca\") pod \"2865be82-0924-44fa-b215-28622b0957d3\" (UID: \"2865be82-0924-44fa-b215-28622b0957d3\") "
Oct 30 19:01:19.360813 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:19.360786    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2865be82-0924-44fa-b215-28622b0957d3-maas-api-openshift-ca" pod "2865be82-0924-44fa-b215-28622b0957d3" (UID: "2865be82-0924-44fa-b215-28622b0957d3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:01:19.370702 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:19.369928    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2865be82-0924-44fa-b215-28622b0957d3-kube-api-access-5tdjj" pod "2865be82-0924-44fa-b215-28622b0957d3" (UID: "2865be82-0924-44fa-b215-28622b0957d3"). InnerVolumeSpecName "kube-api-access-5tdjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:01:19.381765 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:19.380652    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2865be82-0924-44fa-b215-28622b0957d3-maas-api-xks-ca" pod "2865be82-0924-44fa-b215-28622b0957d3" (UID: "2865be82-0924-44fa-b215-28622b0957d3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:01:19.406912 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:19.406862    4525 scope.go:122] "RemoveContainer" containerID="ce496fee7c64949d9029dae1b4578aeb0dabc73e935b99346ba6d1f8b8ec16e5"
Oct 30 19:01:19.431697 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:19.431648    4525 scope.go:122] "RemoveContainer" containerID="0c58e4acbdb96c73029cfdc6fbba135f28079ebdbdc1ef94568faaa3a70eb860"
Oct 30 19:01:19.462858 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:19.462660    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2865be82-0924-44fa-b215-28622b0957d3-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:19.462858 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:19.462725    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tdjj\" (UniqueName: \"kubernetes.io/projected/2865be82-0924-44fa-b215-28622b0957d3-kube-api-access-5tdjj\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:19.462858 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:19.462742    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2865be82-0924-44fa-b215-28622b0957d3-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:19.527620 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:19.527594    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c5f4bd6-9vh7q"
Oct 30 19:01:19.527791 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:19.527618    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c5f4bd6-9vh7q" event={"ID":"2865be82-0924-44fa-b215-28622b0957d3","Type":"ContainerDied","Data":"3c144d26d1e9c975c2a6242cf7f6a896433e007831e90a6eb02215bdf3006704"}
Oct 30 19:01:19.527791 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:19.527692    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c144d26d1e9c975c2a6242cf7f6a896433e007831e90a6eb02215bdf3006704"
Oct 30 19:01:19.745137 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:19.745056    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdc7e4a4-cnzvg"]
Oct 30 19:01:19.745692 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:19.745665    4525 cpu_manager.go:414] "containerMap: removing container" podUID="2865be82-0924-44fa-b215-28622b0957d3" containerName="revoke-keys"
Oct 30 19:01:19.745755 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:19.745696    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2865be82-0924-44fa-b215-28622b0957d3" containerName="revoke-keys"
Oct 30 19:01:19.745824 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:19.745815    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="2865be82-0924-44fa-b215-28622b0957d3" containerName="revoke-keys"
Oct 30 19:01:19.751046 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:19.751026    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdc7e4a4-cnzvg"
Oct 30 19:01:19.755131 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:19.755104    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94hcn\""
Oct 30 19:01:19.755262 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:19.755173    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:01:19.759032 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:19.759006    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdc7e4a4-cnzvg"]
Oct 30 19:01:19.767612 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:19.767590    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac9d22c9-c98b-4857-bd09-6c55f8d79667-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fdc7e4a4-cnzvg\" (UID: \"ac9d22c9-c98b-4857-bd09-6c55f8d79667\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdc7e4a4-cnzvg"
Oct 30 19:01:19.767917 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:19.767898    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac9d22c9-c98b-4857-bd09-6c55f8d79667-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fdc7e4a4-cnzvg\" (UID: \"ac9d22c9-c98b-4857-bd09-6c55f8d79667\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdc7e4a4-cnzvg"
Oct 30 19:01:19.768166 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:19.767945    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8jt7\" (UniqueName: \"kubernetes.io/projected/ac9d22c9-c98b-4857-bd09-6c55f8d79667-kube-api-access-v8jt7\") pod \"maas-api-revoke-sub-fdc7e4a4-cnzvg\" (UID: \"ac9d22c9-c98b-4857-bd09-6c55f8d79667\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdc7e4a4-cnzvg"
Oct 30 19:01:19.869759 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:19.869724    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac9d22c9-c98b-4857-bd09-6c55f8d79667-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fdc7e4a4-cnzvg\" (UID: \"ac9d22c9-c98b-4857-bd09-6c55f8d79667\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdc7e4a4-cnzvg"
Oct 30 19:01:19.869935 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:19.869778    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8jt7\" (UniqueName: \"kubernetes.io/projected/ac9d22c9-c98b-4857-bd09-6c55f8d79667-kube-api-access-v8jt7\") pod \"maas-api-revoke-sub-fdc7e4a4-cnzvg\" (UID: \"ac9d22c9-c98b-4857-bd09-6c55f8d79667\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdc7e4a4-cnzvg"
Oct 30 19:01:19.869935 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:19.869871    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac9d22c9-c98b-4857-bd09-6c55f8d79667-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fdc7e4a4-cnzvg\" (UID: \"ac9d22c9-c98b-4857-bd09-6c55f8d79667\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdc7e4a4-cnzvg"
Oct 30 19:01:19.870341 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:19.870322    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac9d22c9-c98b-4857-bd09-6c55f8d79667-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fdc7e4a4-cnzvg\" (UID: \"ac9d22c9-c98b-4857-bd09-6c55f8d79667\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdc7e4a4-cnzvg"
Oct 30 19:01:19.872332 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:19.872312    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac9d22c9-c98b-4857-bd09-6c55f8d79667-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fdc7e4a4-cnzvg\" (UID: \"ac9d22c9-c98b-4857-bd09-6c55f8d79667\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdc7e4a4-cnzvg"
Oct 30 19:01:19.878263 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:19.878241    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8jt7\" (UniqueName: \"kubernetes.io/projected/ac9d22c9-c98b-4857-bd09-6c55f8d79667-kube-api-access-v8jt7\") pod \"maas-api-revoke-sub-fdc7e4a4-cnzvg\" (UID: \"ac9d22c9-c98b-4857-bd09-6c55f8d79667\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdc7e4a4-cnzvg"
Oct 30 19:01:20.070795 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:20.070721    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdc7e4a4-cnzvg"
Oct 30 19:01:20.352399 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:01:20.352363    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac9d22c9_c98b_4857_bd09_6c55f8d79667.slice/crio-cca54ed400796ea1fb4be97745cf8840e25f49c5d86c90945ecd545907f0fd0e WatchSource:0}: Error finding container cca54ed400796ea1fb4be97745cf8840e25f49c5d86c90945ecd545907f0fd0e: Status 404 returned error can't find the container with id cca54ed400796ea1fb4be97745cf8840e25f49c5d86c90945ecd545907f0fd0e
Oct 30 19:01:20.352768 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:20.352489    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdc7e4a4-cnzvg"]
Oct 30 19:01:20.538449 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:20.538411    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdc7e4a4-cnzvg" event={"ID":"ac9d22c9-c98b-4857-bd09-6c55f8d79667","Type":"ContainerStarted","Data":"cca54ed400796ea1fb4be97745cf8840e25f49c5d86c90945ecd545907f0fd0e"}
Oct 30 19:01:21.548285 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:21.548200    4525 generic.go:356] "Generic (PLEG): container finished" podID="ac9d22c9-c98b-4857-bd09-6c55f8d79667" containerID="d6e3d41bacb177aa6e14ac9966710b901a3708bdbfae153d48a48ee14d727789" exitCode=0
Oct 30 19:01:21.548646 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:21.548287    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdc7e4a4-cnzvg" event={"ID":"ac9d22c9-c98b-4857-bd09-6c55f8d79667","Type":"ContainerDied","Data":"d6e3d41bacb177aa6e14ac9966710b901a3708bdbfae153d48a48ee14d727789"}
Oct 30 19:01:22.714425 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:22.714395    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdc7e4a4-cnzvg"
Oct 30 19:01:22.810210 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:22.810132    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac9d22c9-c98b-4857-bd09-6c55f8d79667-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac9d22c9-c98b-4857-bd09-6c55f8d79667-maas-api-openshift-ca\") pod \"ac9d22c9-c98b-4857-bd09-6c55f8d79667\" (UID: \"ac9d22c9-c98b-4857-bd09-6c55f8d79667\") "
Oct 30 19:01:22.810370 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:22.810245    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac9d22c9-c98b-4857-bd09-6c55f8d79667-kube-api-access-v8jt7\" (UniqueName: \"kubernetes.io/projected/ac9d22c9-c98b-4857-bd09-6c55f8d79667-kube-api-access-v8jt7\") pod \"ac9d22c9-c98b-4857-bd09-6c55f8d79667\" (UID: \"ac9d22c9-c98b-4857-bd09-6c55f8d79667\") "
Oct 30 19:01:22.810370 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:22.810283    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac9d22c9-c98b-4857-bd09-6c55f8d79667-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac9d22c9-c98b-4857-bd09-6c55f8d79667-maas-api-xks-ca\") pod \"ac9d22c9-c98b-4857-bd09-6c55f8d79667\" (UID: \"ac9d22c9-c98b-4857-bd09-6c55f8d79667\") "
Oct 30 19:01:22.810510 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:22.810471    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac9d22c9-c98b-4857-bd09-6c55f8d79667-maas-api-openshift-ca" pod "ac9d22c9-c98b-4857-bd09-6c55f8d79667" (UID: "ac9d22c9-c98b-4857-bd09-6c55f8d79667"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:01:22.810991 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:22.810965    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac9d22c9-c98b-4857-bd09-6c55f8d79667-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:22.812585 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:22.812557    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac9d22c9-c98b-4857-bd09-6c55f8d79667-maas-api-xks-ca" pod "ac9d22c9-c98b-4857-bd09-6c55f8d79667" (UID: "ac9d22c9-c98b-4857-bd09-6c55f8d79667"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:01:22.812720 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:22.812597    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac9d22c9-c98b-4857-bd09-6c55f8d79667-kube-api-access-v8jt7" pod "ac9d22c9-c98b-4857-bd09-6c55f8d79667" (UID: "ac9d22c9-c98b-4857-bd09-6c55f8d79667"). InnerVolumeSpecName "kube-api-access-v8jt7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:01:22.912646 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:22.912603    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8jt7\" (UniqueName: \"kubernetes.io/projected/ac9d22c9-c98b-4857-bd09-6c55f8d79667-kube-api-access-v8jt7\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:22.912646 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:22.912639    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac9d22c9-c98b-4857-bd09-6c55f8d79667-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:23.349998 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:23.349955    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b701f1a6-xglvp"]
Oct 30 19:01:23.350573 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:23.350557    4525 cpu_manager.go:414] "containerMap: removing container" podUID="ac9d22c9-c98b-4857-bd09-6c55f8d79667" containerName="revoke-keys"
Oct 30 19:01:23.350618 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:23.350576    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac9d22c9-c98b-4857-bd09-6c55f8d79667" containerName="revoke-keys"
Oct 30 19:01:23.350732 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:23.350671    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac9d22c9-c98b-4857-bd09-6c55f8d79667" containerName="revoke-keys"
Oct 30 19:01:23.354377 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:23.354360    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b701f1a6-xglvp"
Oct 30 19:01:23.361985 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:23.361957    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b701f1a6-xglvp"]
Oct 30 19:01:23.420076 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:23.420042    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa9ee8b1-cd03-459f-9ba7-e86b8dd0c5dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b701f1a6-xglvp\" (UID: \"aa9ee8b1-cd03-459f-9ba7-e86b8dd0c5dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b701f1a6-xglvp"
Oct 30 19:01:23.420251 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:23.420196    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qld89\" (UniqueName: \"kubernetes.io/projected/aa9ee8b1-cd03-459f-9ba7-e86b8dd0c5dd-kube-api-access-qld89\") pod \"maas-api-revoke-sub-b701f1a6-xglvp\" (UID: \"aa9ee8b1-cd03-459f-9ba7-e86b8dd0c5dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b701f1a6-xglvp"
Oct 30 19:01:23.420301 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:23.420251    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa9ee8b1-cd03-459f-9ba7-e86b8dd0c5dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b701f1a6-xglvp\" (UID: \"aa9ee8b1-cd03-459f-9ba7-e86b8dd0c5dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b701f1a6-xglvp"
Oct 30 19:01:23.521319 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:23.521286    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa9ee8b1-cd03-459f-9ba7-e86b8dd0c5dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b701f1a6-xglvp\" (UID: \"aa9ee8b1-cd03-459f-9ba7-e86b8dd0c5dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b701f1a6-xglvp"
Oct 30 19:01:23.521489 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:23.521408    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa9ee8b1-cd03-459f-9ba7-e86b8dd0c5dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b701f1a6-xglvp\" (UID: \"aa9ee8b1-cd03-459f-9ba7-e86b8dd0c5dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b701f1a6-xglvp"
Oct 30 19:01:23.521590 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:23.521573    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qld89\" (UniqueName: \"kubernetes.io/projected/aa9ee8b1-cd03-459f-9ba7-e86b8dd0c5dd-kube-api-access-qld89\") pod \"maas-api-revoke-sub-b701f1a6-xglvp\" (UID: \"aa9ee8b1-cd03-459f-9ba7-e86b8dd0c5dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b701f1a6-xglvp"
Oct 30 19:01:23.522229 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:23.522191    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa9ee8b1-cd03-459f-9ba7-e86b8dd0c5dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b701f1a6-xglvp\" (UID: \"aa9ee8b1-cd03-459f-9ba7-e86b8dd0c5dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b701f1a6-xglvp"
Oct 30 19:01:23.523916 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:23.523895    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa9ee8b1-cd03-459f-9ba7-e86b8dd0c5dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b701f1a6-xglvp\" (UID: \"aa9ee8b1-cd03-459f-9ba7-e86b8dd0c5dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b701f1a6-xglvp"
Oct 30 19:01:23.529744 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:23.529722    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qld89\" (UniqueName: \"kubernetes.io/projected/aa9ee8b1-cd03-459f-9ba7-e86b8dd0c5dd-kube-api-access-qld89\") pod \"maas-api-revoke-sub-b701f1a6-xglvp\" (UID: \"aa9ee8b1-cd03-459f-9ba7-e86b8dd0c5dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b701f1a6-xglvp"
Oct 30 19:01:23.567007 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:23.566984    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdc7e4a4-cnzvg"
Oct 30 19:01:23.567110 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:23.567019    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdc7e4a4-cnzvg" event={"ID":"ac9d22c9-c98b-4857-bd09-6c55f8d79667","Type":"ContainerDied","Data":"cca54ed400796ea1fb4be97745cf8840e25f49c5d86c90945ecd545907f0fd0e"}
Oct 30 19:01:23.567110 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:23.567065    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cca54ed400796ea1fb4be97745cf8840e25f49c5d86c90945ecd545907f0fd0e"
Oct 30 19:01:23.671698 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:23.671656    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b701f1a6-xglvp"
Oct 30 19:01:23.961483 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:23.961394    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b701f1a6-xglvp"]
Oct 30 19:01:24.576675 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:24.576575    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b701f1a6-xglvp" event={"ID":"aa9ee8b1-cd03-459f-9ba7-e86b8dd0c5dd","Type":"ContainerStarted","Data":"a1a69b5630659aa7bba6c90b110cbf32204c266d43c744608e47e6e22116c0d6"}
Oct 30 19:01:24.576675 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:24.576635    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b701f1a6-xglvp" event={"ID":"aa9ee8b1-cd03-459f-9ba7-e86b8dd0c5dd","Type":"ContainerStarted","Data":"0cdc93e25a46ecdcf79563d12201a4f21d0729ca35b4a6b261e2b02f6f3360f3"}
Oct 30 19:01:24.600234 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:24.600179    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b701f1a6-xglvp" podStartSLOduration=1.600165787 podStartE2EDuration="1.600165787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:01:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:01:24.5979502 +0000 UTC m=+1507.223620163" watchObservedRunningTime="2026-10-30 19:01:24.600165787 +0000 UTC m=+1507.225835750"
Oct 30 19:01:25.587064 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:25.587030    4525 generic.go:356] "Generic (PLEG): container finished" podID="aa9ee8b1-cd03-459f-9ba7-e86b8dd0c5dd" containerID="a1a69b5630659aa7bba6c90b110cbf32204c266d43c744608e47e6e22116c0d6" exitCode=0
Oct 30 19:01:25.587513 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:25.587122    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b701f1a6-xglvp" event={"ID":"aa9ee8b1-cd03-459f-9ba7-e86b8dd0c5dd","Type":"ContainerDied","Data":"a1a69b5630659aa7bba6c90b110cbf32204c266d43c744608e47e6e22116c0d6"}
Oct 30 19:01:26.744843 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:26.744820    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b701f1a6-xglvp"
Oct 30 19:01:26.889588 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:26.889556    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa9ee8b1-cd03-459f-9ba7-e86b8dd0c5dd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa9ee8b1-cd03-459f-9ba7-e86b8dd0c5dd-maas-api-xks-ca\") pod \"aa9ee8b1-cd03-459f-9ba7-e86b8dd0c5dd\" (UID: \"aa9ee8b1-cd03-459f-9ba7-e86b8dd0c5dd\") "
Oct 30 19:01:26.889812 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:26.889632    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa9ee8b1-cd03-459f-9ba7-e86b8dd0c5dd-kube-api-access-qld89\" (UniqueName: \"kubernetes.io/projected/aa9ee8b1-cd03-459f-9ba7-e86b8dd0c5dd-kube-api-access-qld89\") pod \"aa9ee8b1-cd03-459f-9ba7-e86b8dd0c5dd\" (UID: \"aa9ee8b1-cd03-459f-9ba7-e86b8dd0c5dd\") "
Oct 30 19:01:26.889812 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:26.889671    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa9ee8b1-cd03-459f-9ba7-e86b8dd0c5dd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa9ee8b1-cd03-459f-9ba7-e86b8dd0c5dd-maas-api-openshift-ca\") pod \"aa9ee8b1-cd03-459f-9ba7-e86b8dd0c5dd\" (UID: \"aa9ee8b1-cd03-459f-9ba7-e86b8dd0c5dd\") "
Oct 30 19:01:26.890128 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:26.890099    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa9ee8b1-cd03-459f-9ba7-e86b8dd0c5dd-maas-api-openshift-ca" pod "aa9ee8b1-cd03-459f-9ba7-e86b8dd0c5dd" (UID: "aa9ee8b1-cd03-459f-9ba7-e86b8dd0c5dd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:01:26.890244 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:26.890228    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa9ee8b1-cd03-459f-9ba7-e86b8dd0c5dd-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:26.891957 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:26.891926    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa9ee8b1-cd03-459f-9ba7-e86b8dd0c5dd-kube-api-access-qld89" pod "aa9ee8b1-cd03-459f-9ba7-e86b8dd0c5dd" (UID: "aa9ee8b1-cd03-459f-9ba7-e86b8dd0c5dd"). InnerVolumeSpecName "kube-api-access-qld89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:01:26.892035 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:26.891981    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa9ee8b1-cd03-459f-9ba7-e86b8dd0c5dd-maas-api-xks-ca" pod "aa9ee8b1-cd03-459f-9ba7-e86b8dd0c5dd" (UID: "aa9ee8b1-cd03-459f-9ba7-e86b8dd0c5dd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:01:26.991338 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:26.991295    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa9ee8b1-cd03-459f-9ba7-e86b8dd0c5dd-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:26.991338 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:26.991331    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qld89\" (UniqueName: \"kubernetes.io/projected/aa9ee8b1-cd03-459f-9ba7-e86b8dd0c5dd-kube-api-access-qld89\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:27.605070 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:27.605041    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b701f1a6-xglvp"
Oct 30 19:01:27.605246 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:27.605082    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b701f1a6-xglvp" event={"ID":"aa9ee8b1-cd03-459f-9ba7-e86b8dd0c5dd","Type":"ContainerDied","Data":"0cdc93e25a46ecdcf79563d12201a4f21d0729ca35b4a6b261e2b02f6f3360f3"}
Oct 30 19:01:27.605246 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:27.605126    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0cdc93e25a46ecdcf79563d12201a4f21d0729ca35b4a6b261e2b02f6f3360f3"
Oct 30 19:01:28.181759 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:28.181721    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6a327a-z5pnw"]
Oct 30 19:01:28.187703 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:28.186911    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6a327a-z5pnw"]
Oct 30 19:01:29.608061 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:29.608017    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67f662f4-cbe1-4762-9678-220149cc2eb2" path="/var/lib/kubelet/pods/67f662f4-cbe1-4762-9678-220149cc2eb2/volumes"
Oct 30 19:01:30.121786 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:30.090782    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb" podUID="12ebc8b9-c794-47b5-b87b-a63c4bd993e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:30.223735 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:30.194546    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj" podUID="84c99c49-1d60-4176-bda4-cad29c7d801c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:30.233710 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:30.202552    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr" podUID="82bbbf85-ed93-48d2-9f17-49ac1f31c0a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:30.259709 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:30.202589    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts" podUID="be6d4969-285b-4da0-883e-4847f2cbf3af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Oct 30 19:01:30.263814 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:30.263762    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-6a327a/praxis-route-6a327a-kserve-5cb7997949-4vzjq" podUID="0dcb92f5-5de5-4c91-87cc-27764bea9683" containerName="main" probeResult="failure" output="Get \"https://10.133.0.149:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:30.265020 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:30.264984    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw" podUID="ac6852fd-01be-4d69-8c9b-0c591e50b889" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:30.385800 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:30.379036    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t" podUID="424fca14-6d8a-4663-86ec-0c91165f34ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 19:01:30.522020 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:30.520783    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv" podUID="595230e3-55f3-43eb-9a7b-6d89250af9b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:31.117910 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:31.117869    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb" podUID="12ebc8b9-c794-47b5-b87b-a63c4bd993e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 19:01:31.129707 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:31.122452    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb"
Oct 30 19:01:31.263714 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:31.256890    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr" podUID="82bbbf85-ed93-48d2-9f17-49ac1f31c0a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:31.263714 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:31.257333    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj" podUID="84c99c49-1d60-4176-bda4-cad29c7d801c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:31.263714 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:31.257661    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr"
Oct 30 19:01:31.263714 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:31.257781    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj"
Oct 30 19:01:31.264483 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:31.264447    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts" podUID="be6d4969-285b-4da0-883e-4847f2cbf3af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:31.268706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:31.265375    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts"
Oct 30 19:01:31.284914 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:31.272779    4525 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv" podUID="595230e3-55f3-43eb-9a7b-6d89250af9b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": context deadline exceeded"
Oct 30 19:01:31.284914 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:31.273434    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-6a327a/praxis-route-6a327a-kserve-5cb7997949-4vzjq" podUID="0dcb92f5-5de5-4c91-87cc-27764bea9683" containerName="main" probeResult="failure" output="Get \"https://10.133.0.149:8000/ready\": context deadline exceeded"
Oct 30 19:01:31.284914 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:31.273762    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-6a327a/praxis-route-6a327a-kserve-5cb7997949-4vzjq"
Oct 30 19:01:31.297009 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:31.296816    4525 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-praxis-a-6a327a/praxis-route-6a327a-kserve-5cb7997949-4vzjq" podUID="0dcb92f5-5de5-4c91-87cc-27764bea9683" containerName="main" probeResult="failure" output="Get \"https://10.133.0.149:8000/health\": context deadline exceeded"
Oct 30 19:01:31.359950 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:31.359902    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw" podUID="ac6852fd-01be-4d69-8c9b-0c591e50b889" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 19:01:31.360596 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:31.360571    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw"
Oct 30 19:01:31.379500 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:31.379397    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t" podUID="424fca14-6d8a-4663-86ec-0c91165f34ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:31.408709 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:31.408658    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t"
Oct 30 19:01:31.941098 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:31.941056    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv" podUID="595230e3-55f3-43eb-9a7b-6d89250af9b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 19:01:32.038573 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:32.038541    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv"
Oct 30 19:01:32.073539 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:32.073511    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb"
Oct 30 19:01:32.077553 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:32.077534    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj"
Oct 30 19:01:32.077998 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:32.077983    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t"
Oct 30 19:01:32.098204 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:32.095625    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr"
Oct 30 19:01:32.098204 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:32.095696    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw"
Oct 30 19:01:32.099709 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:32.099481    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts"
Oct 30 19:01:32.105991 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:32.105030    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-6a327a/praxis-route-6a327a-kserve-5cb7997949-4vzjq"
Oct 30 19:01:32.111462 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:32.111436    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv"
Oct 30 19:01:32.853092 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:32.853054    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44a8360d-nl2qd"]
Oct 30 19:01:32.853733 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:32.853716    4525 cpu_manager.go:414] "containerMap: removing container" podUID="aa9ee8b1-cd03-459f-9ba7-e86b8dd0c5dd" containerName="revoke-keys"
Oct 30 19:01:32.853790 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:32.853735    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa9ee8b1-cd03-459f-9ba7-e86b8dd0c5dd" containerName="revoke-keys"
Oct 30 19:01:32.853865 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:32.853856    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa9ee8b1-cd03-459f-9ba7-e86b8dd0c5dd" containerName="revoke-keys"
Oct 30 19:01:32.857788 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:32.857768    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44a8360d-nl2qd"
Oct 30 19:01:32.861493 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:32.861403    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:01:32.862150 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:32.862129    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94hcn\""
Oct 30 19:01:32.866501 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:32.866476    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44a8360d-nl2qd"]
Oct 30 19:01:32.950101 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:32.950064    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea492c81-b0a8-4973-9e81-0bfb8d8c1c10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44a8360d-nl2qd\" (UID: \"ea492c81-b0a8-4973-9e81-0bfb8d8c1c10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44a8360d-nl2qd"
Oct 30 19:01:32.950280 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:32.950234    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea492c81-b0a8-4973-9e81-0bfb8d8c1c10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44a8360d-nl2qd\" (UID: \"ea492c81-b0a8-4973-9e81-0bfb8d8c1c10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44a8360d-nl2qd"
Oct 30 19:01:32.950361 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:32.950346    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68ntm\" (UniqueName: \"kubernetes.io/projected/ea492c81-b0a8-4973-9e81-0bfb8d8c1c10-kube-api-access-68ntm\") pod \"maas-api-revoke-sub-44a8360d-nl2qd\" (UID: \"ea492c81-b0a8-4973-9e81-0bfb8d8c1c10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44a8360d-nl2qd"
Oct 30 19:01:33.051156 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:33.051119    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68ntm\" (UniqueName: \"kubernetes.io/projected/ea492c81-b0a8-4973-9e81-0bfb8d8c1c10-kube-api-access-68ntm\") pod \"maas-api-revoke-sub-44a8360d-nl2qd\" (UID: \"ea492c81-b0a8-4973-9e81-0bfb8d8c1c10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44a8360d-nl2qd"
Oct 30 19:01:33.051307 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:33.051228    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea492c81-b0a8-4973-9e81-0bfb8d8c1c10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44a8360d-nl2qd\" (UID: \"ea492c81-b0a8-4973-9e81-0bfb8d8c1c10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44a8360d-nl2qd"
Oct 30 19:01:33.051367 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:33.051351    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea492c81-b0a8-4973-9e81-0bfb8d8c1c10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44a8360d-nl2qd\" (UID: \"ea492c81-b0a8-4973-9e81-0bfb8d8c1c10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44a8360d-nl2qd"
Oct 30 19:01:33.052063 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:33.052037    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea492c81-b0a8-4973-9e81-0bfb8d8c1c10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44a8360d-nl2qd\" (UID: \"ea492c81-b0a8-4973-9e81-0bfb8d8c1c10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44a8360d-nl2qd"
Oct 30 19:01:33.053843 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:33.053815    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea492c81-b0a8-4973-9e81-0bfb8d8c1c10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44a8360d-nl2qd\" (UID: \"ea492c81-b0a8-4973-9e81-0bfb8d8c1c10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44a8360d-nl2qd"
Oct 30 19:01:33.060545 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:33.060517    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68ntm\" (UniqueName: \"kubernetes.io/projected/ea492c81-b0a8-4973-9e81-0bfb8d8c1c10-kube-api-access-68ntm\") pod \"maas-api-revoke-sub-44a8360d-nl2qd\" (UID: \"ea492c81-b0a8-4973-9e81-0bfb8d8c1c10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44a8360d-nl2qd"
Oct 30 19:01:33.177595 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:33.177562    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44a8360d-nl2qd"
Oct 30 19:01:33.461251 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:01:33.461223    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea492c81_b0a8_4973_9e81_0bfb8d8c1c10.slice/crio-4e351b22878ec609d7f985e4be53ceba6c63b907b70685bed447b483ccb38602 WatchSource:0}: Error finding container 4e351b22878ec609d7f985e4be53ceba6c63b907b70685bed447b483ccb38602: Status 404 returned error can't find the container with id 4e351b22878ec609d7f985e4be53ceba6c63b907b70685bed447b483ccb38602
Oct 30 19:01:33.463073 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:33.463047    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44a8360d-nl2qd"]
Oct 30 19:01:33.629638 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:33.629603    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-6a327a/praxis-route-6a327a-kserve-5cb7997949-4vzjq"]
Oct 30 19:01:34.076377 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:34.076284    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44a8360d-nl2qd" event={"ID":"ea492c81-b0a8-4973-9e81-0bfb8d8c1c10","Type":"ContainerStarted","Data":"17141dcbd8999ddd4d7fad1bc31f4f8fe0474c633eeb13c4fcbafce242d926b7"}
Oct 30 19:01:34.078996 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:34.078917    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44a8360d-nl2qd" event={"ID":"ea492c81-b0a8-4973-9e81-0bfb8d8c1c10","Type":"ContainerStarted","Data":"4e351b22878ec609d7f985e4be53ceba6c63b907b70685bed447b483ccb38602"}
Oct 30 19:01:34.104760 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:34.104711    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44a8360d-nl2qd" podStartSLOduration=2.104666756 podStartE2EDuration="2.104666756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:01:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:01:34.102555224 +0000 UTC m=+1516.728225183" watchObservedRunningTime="2026-10-30 19:01:34.104666756 +0000 UTC m=+1516.730336746"
Oct 30 19:01:34.166933 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:34.166905    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf304416-z29q5"]
Oct 30 19:01:34.171545 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:34.171520    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf304416-z29q5"
Oct 30 19:01:34.180514 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:34.180483    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf304416-z29q5"]
Oct 30 19:01:34.266137 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:34.266104    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ff9d08b-2785-438e-a818-d217f24f247c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf304416-z29q5\" (UID: \"5ff9d08b-2785-438e-a818-d217f24f247c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf304416-z29q5"
Oct 30 19:01:34.266137 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:34.266149    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdgsx\" (UniqueName: \"kubernetes.io/projected/5ff9d08b-2785-438e-a818-d217f24f247c-kube-api-access-pdgsx\") pod \"maas-api-revoke-sub-bf304416-z29q5\" (UID: \"5ff9d08b-2785-438e-a818-d217f24f247c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf304416-z29q5"
Oct 30 19:01:34.266409 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:34.266390    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ff9d08b-2785-438e-a818-d217f24f247c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf304416-z29q5\" (UID: \"5ff9d08b-2785-438e-a818-d217f24f247c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf304416-z29q5"
Oct 30 19:01:34.367326 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:34.367283    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ff9d08b-2785-438e-a818-d217f24f247c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf304416-z29q5\" (UID: \"5ff9d08b-2785-438e-a818-d217f24f247c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf304416-z29q5"
Oct 30 19:01:34.367501 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:34.367346    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdgsx\" (UniqueName: \"kubernetes.io/projected/5ff9d08b-2785-438e-a818-d217f24f247c-kube-api-access-pdgsx\") pod \"maas-api-revoke-sub-bf304416-z29q5\" (UID: \"5ff9d08b-2785-438e-a818-d217f24f247c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf304416-z29q5"
Oct 30 19:01:34.367581 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:34.367561    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ff9d08b-2785-438e-a818-d217f24f247c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf304416-z29q5\" (UID: \"5ff9d08b-2785-438e-a818-d217f24f247c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf304416-z29q5"
Oct 30 19:01:34.368043 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:34.368019    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ff9d08b-2785-438e-a818-d217f24f247c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf304416-z29q5\" (UID: \"5ff9d08b-2785-438e-a818-d217f24f247c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf304416-z29q5"
Oct 30 19:01:34.370125 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:34.370099    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ff9d08b-2785-438e-a818-d217f24f247c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf304416-z29q5\" (UID: \"5ff9d08b-2785-438e-a818-d217f24f247c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf304416-z29q5"
Oct 30 19:01:34.375932 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:34.375908    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdgsx\" (UniqueName: \"kubernetes.io/projected/5ff9d08b-2785-438e-a818-d217f24f247c-kube-api-access-pdgsx\") pod \"maas-api-revoke-sub-bf304416-z29q5\" (UID: \"5ff9d08b-2785-438e-a818-d217f24f247c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf304416-z29q5"
Oct 30 19:01:34.495140 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:34.495111    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf304416-z29q5"
Oct 30 19:01:34.783807 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:01:34.783777    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ff9d08b_2785_438e_a818_d217f24f247c.slice/crio-5a16f15bbf6f98fd72b709a5b3323ad17109217d0cfd599d5c20ba9aaa15359f WatchSource:0}: Error finding container 5a16f15bbf6f98fd72b709a5b3323ad17109217d0cfd599d5c20ba9aaa15359f: Status 404 returned error can't find the container with id 5a16f15bbf6f98fd72b709a5b3323ad17109217d0cfd599d5c20ba9aaa15359f
Oct 30 19:01:34.783939 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:34.783905    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf304416-z29q5"]
Oct 30 19:01:34.812424 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:34.812398    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6555186c-9phwp"]
Oct 30 19:01:34.817967 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:34.817949    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6555186c-9phwp"
Oct 30 19:01:34.828393 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:34.828369    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6555186c-9phwp"]
Oct 30 19:01:34.873026 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:34.872997    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7cda5cb-59d6-4ca3-b308-401c7c3de9af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6555186c-9phwp\" (UID: \"d7cda5cb-59d6-4ca3-b308-401c7c3de9af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6555186c-9phwp"
Oct 30 19:01:34.873190 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:34.873060    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7cda5cb-59d6-4ca3-b308-401c7c3de9af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6555186c-9phwp\" (UID: \"d7cda5cb-59d6-4ca3-b308-401c7c3de9af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6555186c-9phwp"
Oct 30 19:01:34.873234 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:34.873217    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wgvc\" (UniqueName: \"kubernetes.io/projected/d7cda5cb-59d6-4ca3-b308-401c7c3de9af-kube-api-access-9wgvc\") pod \"maas-api-revoke-sub-6555186c-9phwp\" (UID: \"d7cda5cb-59d6-4ca3-b308-401c7c3de9af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6555186c-9phwp"
Oct 30 19:01:34.974200 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:34.974162    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7cda5cb-59d6-4ca3-b308-401c7c3de9af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6555186c-9phwp\" (UID: \"d7cda5cb-59d6-4ca3-b308-401c7c3de9af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6555186c-9phwp"
Oct 30 19:01:34.974403 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:34.974267    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7cda5cb-59d6-4ca3-b308-401c7c3de9af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6555186c-9phwp\" (UID: \"d7cda5cb-59d6-4ca3-b308-401c7c3de9af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6555186c-9phwp"
Oct 30 19:01:34.974403 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:34.974387    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wgvc\" (UniqueName: \"kubernetes.io/projected/d7cda5cb-59d6-4ca3-b308-401c7c3de9af-kube-api-access-9wgvc\") pod \"maas-api-revoke-sub-6555186c-9phwp\" (UID: \"d7cda5cb-59d6-4ca3-b308-401c7c3de9af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6555186c-9phwp"
Oct 30 19:01:34.974927 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:34.974906    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7cda5cb-59d6-4ca3-b308-401c7c3de9af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6555186c-9phwp\" (UID: \"d7cda5cb-59d6-4ca3-b308-401c7c3de9af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6555186c-9phwp"
Oct 30 19:01:34.976663 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:34.976633    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7cda5cb-59d6-4ca3-b308-401c7c3de9af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6555186c-9phwp\" (UID: \"d7cda5cb-59d6-4ca3-b308-401c7c3de9af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6555186c-9phwp"
Oct 30 19:01:34.982671 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:34.982650    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wgvc\" (UniqueName: \"kubernetes.io/projected/d7cda5cb-59d6-4ca3-b308-401c7c3de9af-kube-api-access-9wgvc\") pod \"maas-api-revoke-sub-6555186c-9phwp\" (UID: \"d7cda5cb-59d6-4ca3-b308-401c7c3de9af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6555186c-9phwp"
Oct 30 19:01:35.090200 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:35.090110    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf304416-z29q5" event={"ID":"5ff9d08b-2785-438e-a818-d217f24f247c","Type":"ContainerStarted","Data":"5a16f15bbf6f98fd72b709a5b3323ad17109217d0cfd599d5c20ba9aaa15359f"}
Oct 30 19:01:35.093216 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:35.093190    4525 generic.go:356] "Generic (PLEG): container finished" podID="ea492c81-b0a8-4973-9e81-0bfb8d8c1c10" containerID="17141dcbd8999ddd4d7fad1bc31f4f8fe0474c633eeb13c4fcbafce242d926b7" exitCode=0
Oct 30 19:01:35.093346 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:35.093256    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44a8360d-nl2qd" event={"ID":"ea492c81-b0a8-4973-9e81-0bfb8d8c1c10","Type":"ContainerDied","Data":"17141dcbd8999ddd4d7fad1bc31f4f8fe0474c633eeb13c4fcbafce242d926b7"}
Oct 30 19:01:35.145095 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:35.145057    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6555186c-9phwp"
Oct 30 19:01:35.433533 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:01:35.433505    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7cda5cb_59d6_4ca3_b308_401c7c3de9af.slice/crio-9ac4e94312ca41201670a541abc4cbe1b456fd18b0ae549ebf648236ad71a54e WatchSource:0}: Error finding container 9ac4e94312ca41201670a541abc4cbe1b456fd18b0ae549ebf648236ad71a54e: Status 404 returned error can't find the container with id 9ac4e94312ca41201670a541abc4cbe1b456fd18b0ae549ebf648236ad71a54e
Oct 30 19:01:35.433772 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:35.433749    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6555186c-9phwp"]
Oct 30 19:01:36.104008 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:36.103968    4525 generic.go:356] "Generic (PLEG): container finished" podID="5ff9d08b-2785-438e-a818-d217f24f247c" containerID="3403f2bca61b8b74b244abb2b28473ae39f48739527ca4cc26232755be53e9c9" exitCode=0
Oct 30 19:01:36.104502 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:36.104054    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf304416-z29q5" event={"ID":"5ff9d08b-2785-438e-a818-d217f24f247c","Type":"ContainerDied","Data":"3403f2bca61b8b74b244abb2b28473ae39f48739527ca4cc26232755be53e9c9"}
Oct 30 19:01:36.106047 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:36.106018    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6555186c-9phwp" event={"ID":"d7cda5cb-59d6-4ca3-b308-401c7c3de9af","Type":"ContainerStarted","Data":"335e35fb4b1342a9a2103c25ec31815838310b69db68139b24b4183a4191750d"}
Oct 30 19:01:36.106197 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:36.106076    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6555186c-9phwp" event={"ID":"d7cda5cb-59d6-4ca3-b308-401c7c3de9af","Type":"ContainerStarted","Data":"9ac4e94312ca41201670a541abc4cbe1b456fd18b0ae549ebf648236ad71a54e"}
Oct 30 19:01:36.142002 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:36.141948    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6555186c-9phwp" podStartSLOduration=2.141933405 podStartE2EDuration="2.141933405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:01:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:01:36.138672071 +0000 UTC m=+1518.764342034" watchObservedRunningTime="2026-10-30 19:01:36.141933405 +0000 UTC m=+1518.767603396"
Oct 30 19:01:36.269075 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:36.269052    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44a8360d-nl2qd"
Oct 30 19:01:36.299849 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:36.299814    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea492c81-b0a8-4973-9e81-0bfb8d8c1c10-kube-api-access-68ntm\" (UniqueName: \"kubernetes.io/projected/ea492c81-b0a8-4973-9e81-0bfb8d8c1c10-kube-api-access-68ntm\") pod \"ea492c81-b0a8-4973-9e81-0bfb8d8c1c10\" (UID: \"ea492c81-b0a8-4973-9e81-0bfb8d8c1c10\") "
Oct 30 19:01:36.300020 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:36.299915    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea492c81-b0a8-4973-9e81-0bfb8d8c1c10-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea492c81-b0a8-4973-9e81-0bfb8d8c1c10-maas-api-openshift-ca\") pod \"ea492c81-b0a8-4973-9e81-0bfb8d8c1c10\" (UID: \"ea492c81-b0a8-4973-9e81-0bfb8d8c1c10\") "
Oct 30 19:01:36.300092 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:36.300073    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea492c81-b0a8-4973-9e81-0bfb8d8c1c10-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea492c81-b0a8-4973-9e81-0bfb8d8c1c10-maas-api-xks-ca\") pod \"ea492c81-b0a8-4973-9e81-0bfb8d8c1c10\" (UID: \"ea492c81-b0a8-4973-9e81-0bfb8d8c1c10\") "
Oct 30 19:01:36.302454 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:36.302389    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea492c81-b0a8-4973-9e81-0bfb8d8c1c10-maas-api-openshift-ca" pod "ea492c81-b0a8-4973-9e81-0bfb8d8c1c10" (UID: "ea492c81-b0a8-4973-9e81-0bfb8d8c1c10"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:01:36.303528 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:36.303492    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea492c81-b0a8-4973-9e81-0bfb8d8c1c10-maas-api-xks-ca" pod "ea492c81-b0a8-4973-9e81-0bfb8d8c1c10" (UID: "ea492c81-b0a8-4973-9e81-0bfb8d8c1c10"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:01:36.305225 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:36.305193    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea492c81-b0a8-4973-9e81-0bfb8d8c1c10-kube-api-access-68ntm" pod "ea492c81-b0a8-4973-9e81-0bfb8d8c1c10" (UID: "ea492c81-b0a8-4973-9e81-0bfb8d8c1c10"). InnerVolumeSpecName "kube-api-access-68ntm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:01:36.402621 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:36.402580    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea492c81-b0a8-4973-9e81-0bfb8d8c1c10-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:36.402621 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:36.402616    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68ntm\" (UniqueName: \"kubernetes.io/projected/ea492c81-b0a8-4973-9e81-0bfb8d8c1c10-kube-api-access-68ntm\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:36.402621 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:36.402627    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea492c81-b0a8-4973-9e81-0bfb8d8c1c10-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:37.116453 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:37.116413    4525 generic.go:356] "Generic (PLEG): container finished" podID="d7cda5cb-59d6-4ca3-b308-401c7c3de9af" containerID="335e35fb4b1342a9a2103c25ec31815838310b69db68139b24b4183a4191750d" exitCode=0
Oct 30 19:01:37.116979 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:37.116568    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6555186c-9phwp" event={"ID":"d7cda5cb-59d6-4ca3-b308-401c7c3de9af","Type":"ContainerDied","Data":"335e35fb4b1342a9a2103c25ec31815838310b69db68139b24b4183a4191750d"}
Oct 30 19:01:37.119049 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:37.119014    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44a8360d-nl2qd"
Oct 30 19:01:37.121278 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:37.120137    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44a8360d-nl2qd" event={"ID":"ea492c81-b0a8-4973-9e81-0bfb8d8c1c10","Type":"ContainerDied","Data":"4e351b22878ec609d7f985e4be53ceba6c63b907b70685bed447b483ccb38602"}
Oct 30 19:01:37.121278 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:37.120184    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e351b22878ec609d7f985e4be53ceba6c63b907b70685bed447b483ccb38602"
Oct 30 19:01:37.336860 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:37.336836    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf304416-z29q5"
Oct 30 19:01:37.415617 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:37.415578    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ff9d08b-2785-438e-a818-d217f24f247c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ff9d08b-2785-438e-a818-d217f24f247c-maas-api-xks-ca\") pod \"5ff9d08b-2785-438e-a818-d217f24f247c\" (UID: \"5ff9d08b-2785-438e-a818-d217f24f247c\") "
Oct 30 19:01:37.415837 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:37.415710    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ff9d08b-2785-438e-a818-d217f24f247c-kube-api-access-pdgsx\" (UniqueName: \"kubernetes.io/projected/5ff9d08b-2785-438e-a818-d217f24f247c-kube-api-access-pdgsx\") pod \"5ff9d08b-2785-438e-a818-d217f24f247c\" (UID: \"5ff9d08b-2785-438e-a818-d217f24f247c\") "
Oct 30 19:01:37.415837 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:37.415783    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ff9d08b-2785-438e-a818-d217f24f247c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ff9d08b-2785-438e-a818-d217f24f247c-maas-api-openshift-ca\") pod \"5ff9d08b-2785-438e-a818-d217f24f247c\" (UID: \"5ff9d08b-2785-438e-a818-d217f24f247c\") "
Oct 30 19:01:37.416174 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:37.416148    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ff9d08b-2785-438e-a818-d217f24f247c-maas-api-openshift-ca" pod "5ff9d08b-2785-438e-a818-d217f24f247c" (UID: "5ff9d08b-2785-438e-a818-d217f24f247c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:01:37.416474 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:37.416451    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ff9d08b-2785-438e-a818-d217f24f247c-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:37.417934 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:37.417911    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ff9d08b-2785-438e-a818-d217f24f247c-kube-api-access-pdgsx" pod "5ff9d08b-2785-438e-a818-d217f24f247c" (UID: "5ff9d08b-2785-438e-a818-d217f24f247c"). InnerVolumeSpecName "kube-api-access-pdgsx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:01:37.417934 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:37.417916    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ff9d08b-2785-438e-a818-d217f24f247c-maas-api-xks-ca" pod "5ff9d08b-2785-438e-a818-d217f24f247c" (UID: "5ff9d08b-2785-438e-a818-d217f24f247c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:01:37.518288 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:37.518242    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdgsx\" (UniqueName: \"kubernetes.io/projected/5ff9d08b-2785-438e-a818-d217f24f247c-kube-api-access-pdgsx\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:37.518288 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:37.518280    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ff9d08b-2785-438e-a818-d217f24f247c-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:38.178708 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:38.175787    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf304416-z29q5" event={"ID":"5ff9d08b-2785-438e-a818-d217f24f247c","Type":"ContainerDied","Data":"5a16f15bbf6f98fd72b709a5b3323ad17109217d0cfd599d5c20ba9aaa15359f"}
Oct 30 19:01:38.178708 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:38.175857    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a16f15bbf6f98fd72b709a5b3323ad17109217d0cfd599d5c20ba9aaa15359f"
Oct 30 19:01:38.178708 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:38.177806    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf304416-z29q5"
Oct 30 19:01:38.357965 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:38.357942    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6555186c-9phwp"
Oct 30 19:01:38.433444 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:38.433359    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7cda5cb-59d6-4ca3-b308-401c7c3de9af-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7cda5cb-59d6-4ca3-b308-401c7c3de9af-maas-api-xks-ca\") pod \"d7cda5cb-59d6-4ca3-b308-401c7c3de9af\" (UID: \"d7cda5cb-59d6-4ca3-b308-401c7c3de9af\") "
Oct 30 19:01:38.433585 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:38.433449    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7cda5cb-59d6-4ca3-b308-401c7c3de9af-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7cda5cb-59d6-4ca3-b308-401c7c3de9af-maas-api-openshift-ca\") pod \"d7cda5cb-59d6-4ca3-b308-401c7c3de9af\" (UID: \"d7cda5cb-59d6-4ca3-b308-401c7c3de9af\") "
Oct 30 19:01:38.433667 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:38.433648    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7cda5cb-59d6-4ca3-b308-401c7c3de9af-kube-api-access-9wgvc\" (UniqueName: \"kubernetes.io/projected/d7cda5cb-59d6-4ca3-b308-401c7c3de9af-kube-api-access-9wgvc\") pod \"d7cda5cb-59d6-4ca3-b308-401c7c3de9af\" (UID: \"d7cda5cb-59d6-4ca3-b308-401c7c3de9af\") "
Oct 30 19:01:38.433913 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:38.433883    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7cda5cb-59d6-4ca3-b308-401c7c3de9af-maas-api-openshift-ca" pod "d7cda5cb-59d6-4ca3-b308-401c7c3de9af" (UID: "d7cda5cb-59d6-4ca3-b308-401c7c3de9af"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:01:38.434195 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:38.434176    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7cda5cb-59d6-4ca3-b308-401c7c3de9af-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:38.435743 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:38.435721    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7cda5cb-59d6-4ca3-b308-401c7c3de9af-maas-api-xks-ca" pod "d7cda5cb-59d6-4ca3-b308-401c7c3de9af" (UID: "d7cda5cb-59d6-4ca3-b308-401c7c3de9af"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:01:38.435831 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:38.435804    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7cda5cb-59d6-4ca3-b308-401c7c3de9af-kube-api-access-9wgvc" pod "d7cda5cb-59d6-4ca3-b308-401c7c3de9af" (UID: "d7cda5cb-59d6-4ca3-b308-401c7c3de9af"). InnerVolumeSpecName "kube-api-access-9wgvc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:01:38.535370 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:38.535336    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wgvc\" (UniqueName: \"kubernetes.io/projected/d7cda5cb-59d6-4ca3-b308-401c7c3de9af-kube-api-access-9wgvc\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:38.535370 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:38.535366    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7cda5cb-59d6-4ca3-b308-401c7c3de9af-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:38.704182 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:38.704102    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55276386-slqmv"]
Oct 30 19:01:38.704642 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:38.704629    4525 cpu_manager.go:414] "containerMap: removing container" podUID="d7cda5cb-59d6-4ca3-b308-401c7c3de9af" containerName="revoke-keys"
Oct 30 19:01:38.704722 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:38.704644    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7cda5cb-59d6-4ca3-b308-401c7c3de9af" containerName="revoke-keys"
Oct 30 19:01:38.704722 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:38.704692    4525 cpu_manager.go:414] "containerMap: removing container" podUID="ea492c81-b0a8-4973-9e81-0bfb8d8c1c10" containerName="revoke-keys"
Oct 30 19:01:38.704722 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:38.704698    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea492c81-b0a8-4973-9e81-0bfb8d8c1c10" containerName="revoke-keys"
Oct 30 19:01:38.704722 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:38.704711    4525 cpu_manager.go:414] "containerMap: removing container" podUID="5ff9d08b-2785-438e-a818-d217f24f247c" containerName="revoke-keys"
Oct 30 19:01:38.704722 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:38.704716    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ff9d08b-2785-438e-a818-d217f24f247c" containerName="revoke-keys"
Oct 30 19:01:38.704884 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:38.704810    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7cda5cb-59d6-4ca3-b308-401c7c3de9af" containerName="revoke-keys"
Oct 30 19:01:38.704884 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:38.704824    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea492c81-b0a8-4973-9e81-0bfb8d8c1c10" containerName="revoke-keys"
Oct 30 19:01:38.704884 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:38.704831    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ff9d08b-2785-438e-a818-d217f24f247c" containerName="revoke-keys"
Oct 30 19:01:38.707992 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:38.707974    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55276386-slqmv"
Oct 30 19:01:38.716714 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:38.716649    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55276386-slqmv"]
Oct 30 19:01:38.737598 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:38.737566    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1c2894f-597a-4d4a-a247-863056689d5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55276386-slqmv\" (UID: \"c1c2894f-597a-4d4a-a247-863056689d5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55276386-slqmv"
Oct 30 19:01:38.737788 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:38.737655    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5x4zk\" (UniqueName: \"kubernetes.io/projected/c1c2894f-597a-4d4a-a247-863056689d5c-kube-api-access-5x4zk\") pod \"maas-api-revoke-sub-55276386-slqmv\" (UID: \"c1c2894f-597a-4d4a-a247-863056689d5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55276386-slqmv"
Oct 30 19:01:38.737869 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:38.737848    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1c2894f-597a-4d4a-a247-863056689d5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55276386-slqmv\" (UID: \"c1c2894f-597a-4d4a-a247-863056689d5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55276386-slqmv"
Oct 30 19:01:38.839080 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:38.839045    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1c2894f-597a-4d4a-a247-863056689d5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55276386-slqmv\" (UID: \"c1c2894f-597a-4d4a-a247-863056689d5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55276386-slqmv"
Oct 30 19:01:38.839254 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:38.839105    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5x4zk\" (UniqueName: \"kubernetes.io/projected/c1c2894f-597a-4d4a-a247-863056689d5c-kube-api-access-5x4zk\") pod \"maas-api-revoke-sub-55276386-slqmv\" (UID: \"c1c2894f-597a-4d4a-a247-863056689d5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55276386-slqmv"
Oct 30 19:01:38.839254 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:38.839155    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1c2894f-597a-4d4a-a247-863056689d5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55276386-slqmv\" (UID: \"c1c2894f-597a-4d4a-a247-863056689d5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55276386-slqmv"
Oct 30 19:01:38.839788 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:38.839764    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1c2894f-597a-4d4a-a247-863056689d5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55276386-slqmv\" (UID: \"c1c2894f-597a-4d4a-a247-863056689d5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55276386-slqmv"
Oct 30 19:01:38.841545 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:38.841519    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1c2894f-597a-4d4a-a247-863056689d5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55276386-slqmv\" (UID: \"c1c2894f-597a-4d4a-a247-863056689d5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55276386-slqmv"
Oct 30 19:01:38.846962 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:38.846941    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5x4zk\" (UniqueName: \"kubernetes.io/projected/c1c2894f-597a-4d4a-a247-863056689d5c-kube-api-access-5x4zk\") pod \"maas-api-revoke-sub-55276386-slqmv\" (UID: \"c1c2894f-597a-4d4a-a247-863056689d5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55276386-slqmv"
Oct 30 19:01:39.039756 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:39.039622    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55276386-slqmv"
Oct 30 19:01:39.191122 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:39.191079    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6555186c-9phwp" event={"ID":"d7cda5cb-59d6-4ca3-b308-401c7c3de9af","Type":"ContainerDied","Data":"9ac4e94312ca41201670a541abc4cbe1b456fd18b0ae549ebf648236ad71a54e"}
Oct 30 19:01:39.191659 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:39.191141    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6555186c-9phwp"
Oct 30 19:01:39.191659 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:39.191161    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ac4e94312ca41201670a541abc4cbe1b456fd18b0ae549ebf648236ad71a54e"
Oct 30 19:01:39.361122 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:01:39.361088    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1c2894f_597a_4d4a_a247_863056689d5c.slice/crio-c4fd3560b310a3dab010b9977281ce35e0471394f7dbca97b8025f6c7e54fc88 WatchSource:0}: Error finding container c4fd3560b310a3dab010b9977281ce35e0471394f7dbca97b8025f6c7e54fc88: Status 404 returned error can't find the container with id c4fd3560b310a3dab010b9977281ce35e0471394f7dbca97b8025f6c7e54fc88
Oct 30 19:01:39.361455 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:39.361437    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55276386-slqmv"]
Oct 30 19:01:40.201575 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:40.201539    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55276386-slqmv" event={"ID":"c1c2894f-597a-4d4a-a247-863056689d5c","Type":"ContainerStarted","Data":"88ba3292d9cf75f275029cedd83c48e22836f50c28a7b3b554dc305ab000e4c8"}
Oct 30 19:01:40.202002 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:40.201595    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55276386-slqmv" event={"ID":"c1c2894f-597a-4d4a-a247-863056689d5c","Type":"ContainerStarted","Data":"c4fd3560b310a3dab010b9977281ce35e0471394f7dbca97b8025f6c7e54fc88"}
Oct 30 19:01:40.220261 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:40.220202    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55276386-slqmv" podStartSLOduration=2.220186033 podStartE2EDuration="2.220186033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:01:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:01:40.218285902 +0000 UTC m=+1522.843955870" watchObservedRunningTime="2026-10-30 19:01:40.220186033 +0000 UTC m=+1522.845856013"
Oct 30 19:01:41.211863 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:41.211825    4525 generic.go:356] "Generic (PLEG): container finished" podID="c1c2894f-597a-4d4a-a247-863056689d5c" containerID="88ba3292d9cf75f275029cedd83c48e22836f50c28a7b3b554dc305ab000e4c8" exitCode=0
Oct 30 19:01:41.212267 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:41.211907    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55276386-slqmv" event={"ID":"c1c2894f-597a-4d4a-a247-863056689d5c","Type":"ContainerDied","Data":"88ba3292d9cf75f275029cedd83c48e22836f50c28a7b3b554dc305ab000e4c8"}
Oct 30 19:01:42.378127 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:42.378098    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55276386-slqmv"
Oct 30 19:01:42.499383 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:42.499290    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1c2894f-597a-4d4a-a247-863056689d5c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1c2894f-597a-4d4a-a247-863056689d5c-maas-api-xks-ca\") pod \"c1c2894f-597a-4d4a-a247-863056689d5c\" (UID: \"c1c2894f-597a-4d4a-a247-863056689d5c\") "
Oct 30 19:01:42.499383 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:42.499339    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1c2894f-597a-4d4a-a247-863056689d5c-kube-api-access-5x4zk\" (UniqueName: \"kubernetes.io/projected/c1c2894f-597a-4d4a-a247-863056689d5c-kube-api-access-5x4zk\") pod \"c1c2894f-597a-4d4a-a247-863056689d5c\" (UID: \"c1c2894f-597a-4d4a-a247-863056689d5c\") "
Oct 30 19:01:42.499626 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:42.499487    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1c2894f-597a-4d4a-a247-863056689d5c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1c2894f-597a-4d4a-a247-863056689d5c-maas-api-openshift-ca\") pod \"c1c2894f-597a-4d4a-a247-863056689d5c\" (UID: \"c1c2894f-597a-4d4a-a247-863056689d5c\") "
Oct 30 19:01:42.499863 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:42.499829    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1c2894f-597a-4d4a-a247-863056689d5c-maas-api-openshift-ca" pod "c1c2894f-597a-4d4a-a247-863056689d5c" (UID: "c1c2894f-597a-4d4a-a247-863056689d5c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:01:42.500439 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:42.500404    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1c2894f-597a-4d4a-a247-863056689d5c-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:42.501922 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:42.501872    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1c2894f-597a-4d4a-a247-863056689d5c-maas-api-xks-ca" pod "c1c2894f-597a-4d4a-a247-863056689d5c" (UID: "c1c2894f-597a-4d4a-a247-863056689d5c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:01:42.502381 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:42.502360    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1c2894f-597a-4d4a-a247-863056689d5c-kube-api-access-5x4zk" pod "c1c2894f-597a-4d4a-a247-863056689d5c" (UID: "c1c2894f-597a-4d4a-a247-863056689d5c"). InnerVolumeSpecName "kube-api-access-5x4zk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:01:42.601377 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:42.601337    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1c2894f-597a-4d4a-a247-863056689d5c-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:42.601377 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:42.601374    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5x4zk\" (UniqueName: \"kubernetes.io/projected/c1c2894f-597a-4d4a-a247-863056689d5c-kube-api-access-5x4zk\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:43.332146 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:43.330729    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55276386-slqmv" event={"ID":"c1c2894f-597a-4d4a-a247-863056689d5c","Type":"ContainerDied","Data":"c4fd3560b310a3dab010b9977281ce35e0471394f7dbca97b8025f6c7e54fc88"}
Oct 30 19:01:43.332146 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:43.330799    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c4fd3560b310a3dab010b9977281ce35e0471394f7dbca97b8025f6c7e54fc88"
Oct 30 19:01:43.332146 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:43.330874    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55276386-slqmv"
Oct 30 19:01:46.086707 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:46.085784    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw" podUID="ac6852fd-01be-4d69-8c9b-0c591e50b889" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:46.086707 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:46.086249    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj" podUID="84c99c49-1d60-4176-bda4-cad29c7d801c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 19:01:46.090704 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:46.087872    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t" podUID="424fca14-6d8a-4663-86ec-0c91165f34ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:46.090704 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:46.088530    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr" podUID="82bbbf85-ed93-48d2-9f17-49ac1f31c0a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:46.103707 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:46.101759    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts" podUID="be6d4969-285b-4da0-883e-4847f2cbf3af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:46.112220 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:46.109837    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-6a327a/praxis-route-6a327a-kserve-5cb7997949-4vzjq" podUID="0dcb92f5-5de5-4c91-87cc-27764bea9683" containerName="main" probeResult="failure" output="Get \"https://10.133.0.149:8000/ready\": context deadline exceeded"
Oct 30 19:01:46.112470 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:46.112442    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv" podUID="595230e3-55f3-43eb-9a7b-6d89250af9b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 19:01:46.162707 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:46.154779    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb" podUID="12ebc8b9-c794-47b5-b87b-a63c4bd993e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:46.930841 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:46.930785    4525 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb" podUID="12ebc8b9-c794-47b5-b87b-a63c4bd993e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:47.132708 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:47.119794    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv" podUID="595230e3-55f3-43eb-9a7b-6d89250af9b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:47.144806 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:47.144759    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-6a327a/praxis-route-6a327a-kserve-5cb7997949-4vzjq" podUID="0dcb92f5-5de5-4c91-87cc-27764bea9683" containerName="main" probeResult="failure" output="Get \"https://10.133.0.149:8000/ready\": context deadline exceeded"
Oct 30 19:01:47.177707 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:47.152852    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv"
Oct 30 19:01:47.344922 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:47.344822    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-6a327a/praxis-route-6a327a-kserve-5cb7997949-4vzjq"
Oct 30 19:01:47.371770 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:47.300314    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts" podUID="be6d4969-285b-4da0-883e-4847f2cbf3af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:47.372305 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:47.300336    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw" podUID="ac6852fd-01be-4d69-8c9b-0c591e50b889" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 19:01:47.416707 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:47.384417    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts"
Oct 30 19:01:47.451847 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:47.451816    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw"
Oct 30 19:01:47.476852 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:47.300397    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr" podUID="82bbbf85-ed93-48d2-9f17-49ac1f31c0a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 30 19:01:47.493751 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:47.300260    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb" podUID="12ebc8b9-c794-47b5-b87b-a63c4bd993e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:47.494389 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:47.300356    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj" podUID="84c99c49-1d60-4176-bda4-cad29c7d801c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 19:01:47.510710 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:47.502023    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb"
Oct 30 19:01:47.511000 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:47.510978    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj"
Oct 30 19:01:47.539789 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:47.391380    4525 patch_prober.go:29] interesting pod/e2e-worker-w3-2889dd-openshift-default-84768d69f7-gtdr6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:01:47.540068 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:47.540036    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-gtdr6" podUID="01a5c107-dcf4-4ca5-8c9d-57c40814206d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:47.558743 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:47.300376    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t" podUID="424fca14-6d8a-4663-86ec-0c91165f34ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:47.572173 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:47.572140    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t"
Oct 30 19:01:47.685699 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:01:47.685646    4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d49l" containerName="extractor"
Oct 30 19:01:47.797999 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:47.796982    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr"
Oct 30 19:01:47.797999 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:47.797044    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr"
Oct 30 19:01:47.797999 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:47.797074    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-6a327a/praxis-route-6a327a-kserve-5cb7997949-4vzjq"
Oct 30 19:01:47.797999 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:47.797104    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb"
Oct 30 19:01:47.797999 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:47.797135    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts"
Oct 30 19:01:47.797999 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:47.797162    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj"
Oct 30 19:01:47.797999 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:47.797192    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t"
Oct 30 19:01:47.797999 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:47.797222    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv"
Oct 30 19:01:47.801232 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:47.801196    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw"
Oct 30 19:01:47.802788 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:47.802757    4525 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr" podUID="82bbbf85-ed93-48d2-9f17-49ac1f31c0a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:48.691510 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:48.691475    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-praxis-a-6a327a/praxis-route-6a327a-kserve-5cb7997949-4vzjq" podUID="0dcb92f5-5de5-4c91-87cc-27764bea9683" containerName="main" containerID="cri-o://435d9aad2e4f57d0efe0cd6430c36f20c19b46aaf3151783674d1a6fa269b77f" gracePeriod=45
Oct 30 19:01:48.980447 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:48.980421    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-6a327a/praxis-route-6a327a-kserve-5cb7997949-4vzjq"
Oct 30 19:01:49.131580 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:49.131539    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0dcb92f5-5de5-4c91-87cc-27764bea9683-tls-certs\" (UniqueName: \"kubernetes.io/secret/0dcb92f5-5de5-4c91-87cc-27764bea9683-tls-certs\") pod \"0dcb92f5-5de5-4c91-87cc-27764bea9683\" (UID: \"0dcb92f5-5de5-4c91-87cc-27764bea9683\") "
Oct 30 19:01:49.131801 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:49.131650    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dcb92f5-5de5-4c91-87cc-27764bea9683-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dcb92f5-5de5-4c91-87cc-27764bea9683-tmp-dir\") pod \"0dcb92f5-5de5-4c91-87cc-27764bea9683\" (UID: \"0dcb92f5-5de5-4c91-87cc-27764bea9683\") "
Oct 30 19:01:49.131865 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:49.131852    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dcb92f5-5de5-4c91-87cc-27764bea9683-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0dcb92f5-5de5-4c91-87cc-27764bea9683-dshm\") pod \"0dcb92f5-5de5-4c91-87cc-27764bea9683\" (UID: \"0dcb92f5-5de5-4c91-87cc-27764bea9683\") "
Oct 30 19:01:49.131913 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:49.131895    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dcb92f5-5de5-4c91-87cc-27764bea9683-tmp-dir" pod "0dcb92f5-5de5-4c91-87cc-27764bea9683" (UID: "0dcb92f5-5de5-4c91-87cc-27764bea9683"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:01:49.131985 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:49.131961    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dcb92f5-5de5-4c91-87cc-27764bea9683-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0dcb92f5-5de5-4c91-87cc-27764bea9683-model-cache\") pod \"0dcb92f5-5de5-4c91-87cc-27764bea9683\" (UID: \"0dcb92f5-5de5-4c91-87cc-27764bea9683\") "
Oct 30 19:01:49.132169 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:49.131997    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dcb92f5-5de5-4c91-87cc-27764bea9683-kube-api-access-9b8xw\" (UniqueName: \"kubernetes.io/projected/0dcb92f5-5de5-4c91-87cc-27764bea9683-kube-api-access-9b8xw\") pod \"0dcb92f5-5de5-4c91-87cc-27764bea9683\" (UID: \"0dcb92f5-5de5-4c91-87cc-27764bea9683\") "
Oct 30 19:01:49.132169 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:49.132028    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dcb92f5-5de5-4c91-87cc-27764bea9683-home\" (UniqueName: \"kubernetes.io/empty-dir/0dcb92f5-5de5-4c91-87cc-27764bea9683-home\") pod \"0dcb92f5-5de5-4c91-87cc-27764bea9683\" (UID: \"0dcb92f5-5de5-4c91-87cc-27764bea9683\") "
Oct 30 19:01:49.132458 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:49.132431    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dcb92f5-5de5-4c91-87cc-27764bea9683-home" pod "0dcb92f5-5de5-4c91-87cc-27764bea9683" (UID: "0dcb92f5-5de5-4c91-87cc-27764bea9683"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:01:49.132566 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:49.132342    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dcb92f5-5de5-4c91-87cc-27764bea9683-model-cache" pod "0dcb92f5-5de5-4c91-87cc-27764bea9683" (UID: "0dcb92f5-5de5-4c91-87cc-27764bea9683"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:01:49.132748 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:49.132727    4525 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dcb92f5-5de5-4c91-87cc-27764bea9683-tmp-dir\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:49.132807 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:49.132757    4525 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0dcb92f5-5de5-4c91-87cc-27764bea9683-model-cache\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:49.132807 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:49.132775    4525 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0dcb92f5-5de5-4c91-87cc-27764bea9683-home\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:49.134140 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:49.134114    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0dcb92f5-5de5-4c91-87cc-27764bea9683-tls-certs" pod "0dcb92f5-5de5-4c91-87cc-27764bea9683" (UID: "0dcb92f5-5de5-4c91-87cc-27764bea9683"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:01:49.134229 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:49.134174    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dcb92f5-5de5-4c91-87cc-27764bea9683-dshm" pod "0dcb92f5-5de5-4c91-87cc-27764bea9683" (UID: "0dcb92f5-5de5-4c91-87cc-27764bea9683"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:01:49.134271 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:49.134230    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dcb92f5-5de5-4c91-87cc-27764bea9683-kube-api-access-9b8xw" pod "0dcb92f5-5de5-4c91-87cc-27764bea9683" (UID: "0dcb92f5-5de5-4c91-87cc-27764bea9683"). InnerVolumeSpecName "kube-api-access-9b8xw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:01:49.238764 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:49.238644    4525 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0dcb92f5-5de5-4c91-87cc-27764bea9683-dshm\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:49.238995 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:49.238977    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9b8xw\" (UniqueName: \"kubernetes.io/projected/0dcb92f5-5de5-4c91-87cc-27764bea9683-kube-api-access-9b8xw\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:49.239105 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:49.239091    4525 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0dcb92f5-5de5-4c91-87cc-27764bea9683-tls-certs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:49.625532 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:49.625495    4525 generic.go:356] "Generic (PLEG): container finished" podID="0dcb92f5-5de5-4c91-87cc-27764bea9683" containerID="435d9aad2e4f57d0efe0cd6430c36f20c19b46aaf3151783674d1a6fa269b77f" exitCode=0
Oct 30 19:01:49.625757 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:49.625546    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-6a327a/praxis-route-6a327a-kserve-5cb7997949-4vzjq"
Oct 30 19:01:49.625757 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:49.625576    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-6a327a/praxis-route-6a327a-kserve-5cb7997949-4vzjq" event={"ID":"0dcb92f5-5de5-4c91-87cc-27764bea9683","Type":"ContainerDied","Data":"435d9aad2e4f57d0efe0cd6430c36f20c19b46aaf3151783674d1a6fa269b77f"}
Oct 30 19:01:49.625757 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:49.625618    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-6a327a/praxis-route-6a327a-kserve-5cb7997949-4vzjq" event={"ID":"0dcb92f5-5de5-4c91-87cc-27764bea9683","Type":"ContainerDied","Data":"a200cee14afdcb421767e20d4a6b78ca738c155d91c0ffe77c6b9e33cbc14a5e"}
Oct 30 19:01:49.625757 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:49.625643    4525 scope.go:122] "RemoveContainer" containerID="435d9aad2e4f57d0efe0cd6430c36f20c19b46aaf3151783674d1a6fa269b77f"
Oct 30 19:01:49.639360 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:49.639341    4525 scope.go:122] "RemoveContainer" containerID="435d9aad2e4f57d0efe0cd6430c36f20c19b46aaf3151783674d1a6fa269b77f"
Oct 30 19:01:49.640506 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:01:49.640472    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"435d9aad2e4f57d0efe0cd6430c36f20c19b46aaf3151783674d1a6fa269b77f\": container with ID starting with 435d9aad2e4f57d0efe0cd6430c36f20c19b46aaf3151783674d1a6fa269b77f not found: ID does not exist" containerID="435d9aad2e4f57d0efe0cd6430c36f20c19b46aaf3151783674d1a6fa269b77f"
Oct 30 19:01:49.640611 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:49.640504    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"435d9aad2e4f57d0efe0cd6430c36f20c19b46aaf3151783674d1a6fa269b77f"} err="failed to get container status \"435d9aad2e4f57d0efe0cd6430c36f20c19b46aaf3151783674d1a6fa269b77f\": rpc error: code = NotFound desc = could not find container \"435d9aad2e4f57d0efe0cd6430c36f20c19b46aaf3151783674d1a6fa269b77f\": container with ID starting with 435d9aad2e4f57d0efe0cd6430c36f20c19b46aaf3151783674d1a6fa269b77f not found: ID does not exist"
Oct 30 19:01:49.645879 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:49.645858    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-6a327a/praxis-route-6a327a-kserve-5cb7997949-4vzjq"]
Oct 30 19:01:49.650298 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:49.650281    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-praxis-a-6a327a/praxis-route-6a327a-kserve-5cb7997949-4vzjq"]
Oct 30 19:01:50.198035 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:50.197997    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06319aae-h4mgm"]
Oct 30 19:01:50.198557 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:50.198543    4525 cpu_manager.go:414] "containerMap: removing container" podUID="c1c2894f-597a-4d4a-a247-863056689d5c" containerName="revoke-keys"
Oct 30 19:01:50.198615 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:50.198559    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1c2894f-597a-4d4a-a247-863056689d5c" containerName="revoke-keys"
Oct 30 19:01:50.198615 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:50.198585    4525 cpu_manager.go:414] "containerMap: removing container" podUID="0dcb92f5-5de5-4c91-87cc-27764bea9683" containerName="main"
Oct 30 19:01:50.198615 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:50.198590    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dcb92f5-5de5-4c91-87cc-27764bea9683" containerName="main"
Oct 30 19:01:50.198752 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:50.198674    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dcb92f5-5de5-4c91-87cc-27764bea9683" containerName="main"
Oct 30 19:01:50.198752 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:50.198703    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1c2894f-597a-4d4a-a247-863056689d5c" containerName="revoke-keys"
Oct 30 19:01:50.204271 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:50.204248    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06319aae-h4mgm"
Oct 30 19:01:50.208159 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:50.208124    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:01:50.209087 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:50.209052    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94hcn\""
Oct 30 19:01:50.212190 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:50.212167    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06319aae-h4mgm"]
Oct 30 19:01:50.290298 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:50.290219    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/334444c1-d724-4938-b69e-331a3cacefd2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06319aae-h4mgm\" (UID: \"334444c1-d724-4938-b69e-331a3cacefd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06319aae-h4mgm"
Oct 30 19:01:50.290741 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:50.290717    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/334444c1-d724-4938-b69e-331a3cacefd2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06319aae-h4mgm\" (UID: \"334444c1-d724-4938-b69e-331a3cacefd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06319aae-h4mgm"
Oct 30 19:01:50.291014 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:50.290998    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sdgs\" (UniqueName: \"kubernetes.io/projected/334444c1-d724-4938-b69e-331a3cacefd2-kube-api-access-5sdgs\") pod \"maas-api-revoke-sub-06319aae-h4mgm\" (UID: \"334444c1-d724-4938-b69e-331a3cacefd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06319aae-h4mgm"
Oct 30 19:01:50.392603 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:50.392556    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/334444c1-d724-4938-b69e-331a3cacefd2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06319aae-h4mgm\" (UID: \"334444c1-d724-4938-b69e-331a3cacefd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06319aae-h4mgm"
Oct 30 19:01:50.392853 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:50.392636    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/334444c1-d724-4938-b69e-331a3cacefd2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06319aae-h4mgm\" (UID: \"334444c1-d724-4938-b69e-331a3cacefd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06319aae-h4mgm"
Oct 30 19:01:50.392853 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:50.392787    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sdgs\" (UniqueName: \"kubernetes.io/projected/334444c1-d724-4938-b69e-331a3cacefd2-kube-api-access-5sdgs\") pod \"maas-api-revoke-sub-06319aae-h4mgm\" (UID: \"334444c1-d724-4938-b69e-331a3cacefd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06319aae-h4mgm"
Oct 30 19:01:50.393348 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:50.393319    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/334444c1-d724-4938-b69e-331a3cacefd2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06319aae-h4mgm\" (UID: \"334444c1-d724-4938-b69e-331a3cacefd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06319aae-h4mgm"
Oct 30 19:01:50.395277 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:50.395252    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/334444c1-d724-4938-b69e-331a3cacefd2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06319aae-h4mgm\" (UID: \"334444c1-d724-4938-b69e-331a3cacefd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06319aae-h4mgm"
Oct 30 19:01:50.402637 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:50.402611    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sdgs\" (UniqueName: \"kubernetes.io/projected/334444c1-d724-4938-b69e-331a3cacefd2-kube-api-access-5sdgs\") pod \"maas-api-revoke-sub-06319aae-h4mgm\" (UID: \"334444c1-d724-4938-b69e-331a3cacefd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06319aae-h4mgm"
Oct 30 19:01:50.523058 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:50.522970    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06319aae-h4mgm"
Oct 30 19:01:50.811769 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:50.811695    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06319aae-h4mgm"]
Oct 30 19:01:51.546836 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:51.546799    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0dcb92f5-5de5-4c91-87cc-27764bea9683" path="/var/lib/kubelet/pods/0dcb92f5-5de5-4c91-87cc-27764bea9683/volumes"
Oct 30 19:01:51.648589 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:51.648549    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06319aae-h4mgm" event={"ID":"334444c1-d724-4938-b69e-331a3cacefd2","Type":"ContainerStarted","Data":"aff0bc3b44c94c5eaf2f85e12c58c5aa12e11f570e8cad741bc94433e6a5bd86"}
Oct 30 19:01:51.648811 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:51.648611    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06319aae-h4mgm" event={"ID":"334444c1-d724-4938-b69e-331a3cacefd2","Type":"ContainerStarted","Data":"70d8110a910097f83b0bfaae30ea214b9f5c5bd9a5ff5fb51c57af85e0a2ba39"}
Oct 30 19:01:51.684193 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:51.684145    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06319aae-h4mgm" podStartSLOduration=1.6841314170000001 podStartE2EDuration="1.684131417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:01:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:01:51.68123278 +0000 UTC m=+1534.306902745" watchObservedRunningTime="2026-10-30 19:01:51.684131417 +0000 UTC m=+1534.309801381"
Oct 30 19:01:52.716289 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:52.715815    4525 generic.go:356] "Generic (PLEG): container finished" podID="334444c1-d724-4938-b69e-331a3cacefd2" containerID="aff0bc3b44c94c5eaf2f85e12c58c5aa12e11f570e8cad741bc94433e6a5bd86" exitCode=0
Oct 30 19:01:52.716289 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:52.716148    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06319aae-h4mgm" event={"ID":"334444c1-d724-4938-b69e-331a3cacefd2","Type":"ContainerDied","Data":"aff0bc3b44c94c5eaf2f85e12c58c5aa12e11f570e8cad741bc94433e6a5bd86"}
Oct 30 19:01:53.899367 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:53.899341    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06319aae-h4mgm"
Oct 30 19:01:53.992604 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:53.992498    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/334444c1-d724-4938-b69e-331a3cacefd2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/334444c1-d724-4938-b69e-331a3cacefd2-maas-api-openshift-ca\") pod \"334444c1-d724-4938-b69e-331a3cacefd2\" (UID: \"334444c1-d724-4938-b69e-331a3cacefd2\") "
Oct 30 19:01:53.992826 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:53.992725    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/334444c1-d724-4938-b69e-331a3cacefd2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/334444c1-d724-4938-b69e-331a3cacefd2-maas-api-xks-ca\") pod \"334444c1-d724-4938-b69e-331a3cacefd2\" (UID: \"334444c1-d724-4938-b69e-331a3cacefd2\") "
Oct 30 19:01:53.992826 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:53.992787    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/334444c1-d724-4938-b69e-331a3cacefd2-kube-api-access-5sdgs\" (UniqueName: \"kubernetes.io/projected/334444c1-d724-4938-b69e-331a3cacefd2-kube-api-access-5sdgs\") pod \"334444c1-d724-4938-b69e-331a3cacefd2\" (UID: \"334444c1-d724-4938-b69e-331a3cacefd2\") "
Oct 30 19:01:53.992967 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:53.992923    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/334444c1-d724-4938-b69e-331a3cacefd2-maas-api-openshift-ca" pod "334444c1-d724-4938-b69e-331a3cacefd2" (UID: "334444c1-d724-4938-b69e-331a3cacefd2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:01:53.993329 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:53.993308    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/334444c1-d724-4938-b69e-331a3cacefd2-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:53.994957 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:53.994930    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/334444c1-d724-4938-b69e-331a3cacefd2-maas-api-xks-ca" pod "334444c1-d724-4938-b69e-331a3cacefd2" (UID: "334444c1-d724-4938-b69e-331a3cacefd2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:01:53.995075 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:53.995043    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/334444c1-d724-4938-b69e-331a3cacefd2-kube-api-access-5sdgs" pod "334444c1-d724-4938-b69e-331a3cacefd2" (UID: "334444c1-d724-4938-b69e-331a3cacefd2"). InnerVolumeSpecName "kube-api-access-5sdgs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:01:54.095069 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:54.095034    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/334444c1-d724-4938-b69e-331a3cacefd2-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:54.095069 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:54.095064    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sdgs\" (UniqueName: \"kubernetes.io/projected/334444c1-d724-4938-b69e-331a3cacefd2-kube-api-access-5sdgs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:54.739460 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:54.739423    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06319aae-h4mgm"
Oct 30 19:01:54.739460 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:54.739451    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06319aae-h4mgm" event={"ID":"334444c1-d724-4938-b69e-331a3cacefd2","Type":"ContainerDied","Data":"70d8110a910097f83b0bfaae30ea214b9f5c5bd9a5ff5fb51c57af85e0a2ba39"}
Oct 30 19:01:54.739696 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:54.739497    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70d8110a910097f83b0bfaae30ea214b9f5c5bd9a5ff5fb51c57af85e0a2ba39"
Oct 30 19:01:55.033837 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:55.033728    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-248f423d-bxtt8"]
Oct 30 19:01:55.034375 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:55.034359    4525 cpu_manager.go:414] "containerMap: removing container" podUID="334444c1-d724-4938-b69e-331a3cacefd2" containerName="revoke-keys"
Oct 30 19:01:55.034465 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:55.034377    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="334444c1-d724-4938-b69e-331a3cacefd2" containerName="revoke-keys"
Oct 30 19:01:55.034465 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:55.034464    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="334444c1-d724-4938-b69e-331a3cacefd2" containerName="revoke-keys"
Oct 30 19:01:55.038172 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:55.038156    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-248f423d-bxtt8"
Oct 30 19:01:55.044925 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:55.044834    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:01:55.045076 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:55.045037    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94hcn\""
Oct 30 19:01:55.047315 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:55.047292    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-248f423d-bxtt8"]
Oct 30 19:01:55.110213 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:55.110181    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1aa70b7e-acd9-48f1-bd02-0db2ea932d98-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-248f423d-bxtt8\" (UID: \"1aa70b7e-acd9-48f1-bd02-0db2ea932d98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-248f423d-bxtt8"
Oct 30 19:01:55.110398 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:55.110301    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk7jg\" (UniqueName: \"kubernetes.io/projected/1aa70b7e-acd9-48f1-bd02-0db2ea932d98-kube-api-access-hk7jg\") pod \"maas-api-revoke-sub-248f423d-bxtt8\" (UID: \"1aa70b7e-acd9-48f1-bd02-0db2ea932d98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-248f423d-bxtt8"
Oct 30 19:01:55.110398 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:55.110375    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1aa70b7e-acd9-48f1-bd02-0db2ea932d98-maas-api-xks-ca\") pod \"maas-api-revoke-sub-248f423d-bxtt8\" (UID: \"1aa70b7e-acd9-48f1-bd02-0db2ea932d98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-248f423d-bxtt8"
Oct 30 19:01:55.225381 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:55.225320    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1aa70b7e-acd9-48f1-bd02-0db2ea932d98-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-248f423d-bxtt8\" (UID: \"1aa70b7e-acd9-48f1-bd02-0db2ea932d98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-248f423d-bxtt8"
Oct 30 19:01:55.225786 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:55.225760    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk7jg\" (UniqueName: \"kubernetes.io/projected/1aa70b7e-acd9-48f1-bd02-0db2ea932d98-kube-api-access-hk7jg\") pod \"maas-api-revoke-sub-248f423d-bxtt8\" (UID: \"1aa70b7e-acd9-48f1-bd02-0db2ea932d98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-248f423d-bxtt8"
Oct 30 19:01:55.226041 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:55.226024    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1aa70b7e-acd9-48f1-bd02-0db2ea932d98-maas-api-xks-ca\") pod \"maas-api-revoke-sub-248f423d-bxtt8\" (UID: \"1aa70b7e-acd9-48f1-bd02-0db2ea932d98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-248f423d-bxtt8"
Oct 30 19:01:55.226667 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:55.226643    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1aa70b7e-acd9-48f1-bd02-0db2ea932d98-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-248f423d-bxtt8\" (UID: \"1aa70b7e-acd9-48f1-bd02-0db2ea932d98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-248f423d-bxtt8"
Oct 30 19:01:55.230574 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:55.230547    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1aa70b7e-acd9-48f1-bd02-0db2ea932d98-maas-api-xks-ca\") pod \"maas-api-revoke-sub-248f423d-bxtt8\" (UID: \"1aa70b7e-acd9-48f1-bd02-0db2ea932d98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-248f423d-bxtt8"
Oct 30 19:01:55.236449 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:55.236415    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk7jg\" (UniqueName: \"kubernetes.io/projected/1aa70b7e-acd9-48f1-bd02-0db2ea932d98-kube-api-access-hk7jg\") pod \"maas-api-revoke-sub-248f423d-bxtt8\" (UID: \"1aa70b7e-acd9-48f1-bd02-0db2ea932d98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-248f423d-bxtt8"
Oct 30 19:01:55.354853 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:55.354829    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-248f423d-bxtt8"
Oct 30 19:01:55.638208 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:01:55.638183    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aa70b7e_acd9_48f1_bd02_0db2ea932d98.slice/crio-6ab05a76af98c6d68198d771a73072aa2ea87b26fa5377b5a3bbe19dff085e1f WatchSource:0}: Error finding container 6ab05a76af98c6d68198d771a73072aa2ea87b26fa5377b5a3bbe19dff085e1f: Status 404 returned error can't find the container with id 6ab05a76af98c6d68198d771a73072aa2ea87b26fa5377b5a3bbe19dff085e1f
Oct 30 19:01:55.639957 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:55.639930    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-248f423d-bxtt8"]
Oct 30 19:01:55.748367 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:55.748331    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-248f423d-bxtt8" event={"ID":"1aa70b7e-acd9-48f1-bd02-0db2ea932d98","Type":"ContainerStarted","Data":"6ab05a76af98c6d68198d771a73072aa2ea87b26fa5377b5a3bbe19dff085e1f"}
Oct 30 19:01:56.758759 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:56.758656    4525 generic.go:356] "Generic (PLEG): container finished" podID="1aa70b7e-acd9-48f1-bd02-0db2ea932d98" containerID="503c49b2f1e7e8e2d248795849299ca3d0654f870502ace4d3820fd1275799b8" exitCode=0
Oct 30 19:01:56.759173 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:56.758766    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-248f423d-bxtt8" event={"ID":"1aa70b7e-acd9-48f1-bd02-0db2ea932d98","Type":"ContainerDied","Data":"503c49b2f1e7e8e2d248795849299ca3d0654f870502ace4d3820fd1275799b8"}
Oct 30 19:01:57.968203 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:57.968176    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-248f423d-bxtt8"
Oct 30 19:01:58.097279 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:58.097193    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1aa70b7e-acd9-48f1-bd02-0db2ea932d98-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1aa70b7e-acd9-48f1-bd02-0db2ea932d98-maas-api-xks-ca\") pod \"1aa70b7e-acd9-48f1-bd02-0db2ea932d98\" (UID: \"1aa70b7e-acd9-48f1-bd02-0db2ea932d98\") "
Oct 30 19:01:58.097462 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:58.097416    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1aa70b7e-acd9-48f1-bd02-0db2ea932d98-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1aa70b7e-acd9-48f1-bd02-0db2ea932d98-maas-api-openshift-ca\") pod \"1aa70b7e-acd9-48f1-bd02-0db2ea932d98\" (UID: \"1aa70b7e-acd9-48f1-bd02-0db2ea932d98\") "
Oct 30 19:01:58.097536 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:58.097497    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1aa70b7e-acd9-48f1-bd02-0db2ea932d98-kube-api-access-hk7jg\" (UniqueName: \"kubernetes.io/projected/1aa70b7e-acd9-48f1-bd02-0db2ea932d98-kube-api-access-hk7jg\") pod \"1aa70b7e-acd9-48f1-bd02-0db2ea932d98\" (UID: \"1aa70b7e-acd9-48f1-bd02-0db2ea932d98\") "
Oct 30 19:01:58.097752 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:58.097723    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1aa70b7e-acd9-48f1-bd02-0db2ea932d98-maas-api-openshift-ca" pod "1aa70b7e-acd9-48f1-bd02-0db2ea932d98" (UID: "1aa70b7e-acd9-48f1-bd02-0db2ea932d98"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:01:58.098067 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:58.097949    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1aa70b7e-acd9-48f1-bd02-0db2ea932d98-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:58.099498 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:58.099467    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1aa70b7e-acd9-48f1-bd02-0db2ea932d98-maas-api-xks-ca" pod "1aa70b7e-acd9-48f1-bd02-0db2ea932d98" (UID: "1aa70b7e-acd9-48f1-bd02-0db2ea932d98"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:01:58.099609 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:58.099550    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1aa70b7e-acd9-48f1-bd02-0db2ea932d98-kube-api-access-hk7jg" pod "1aa70b7e-acd9-48f1-bd02-0db2ea932d98" (UID: "1aa70b7e-acd9-48f1-bd02-0db2ea932d98"). InnerVolumeSpecName "kube-api-access-hk7jg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:01:58.199429 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:58.199394    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1aa70b7e-acd9-48f1-bd02-0db2ea932d98-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:58.199429 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:58.199422    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hk7jg\" (UniqueName: \"kubernetes.io/projected/1aa70b7e-acd9-48f1-bd02-0db2ea932d98-kube-api-access-hk7jg\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:58.799445 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:58.799396    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-248f423d-bxtt8" event={"ID":"1aa70b7e-acd9-48f1-bd02-0db2ea932d98","Type":"ContainerDied","Data":"6ab05a76af98c6d68198d771a73072aa2ea87b26fa5377b5a3bbe19dff085e1f"}
Oct 30 19:01:58.799623 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:58.799503    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ab05a76af98c6d68198d771a73072aa2ea87b26fa5377b5a3bbe19dff085e1f"
Oct 30 19:01:58.799623 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:01:58.799576    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-248f423d-bxtt8"
Oct 30 19:02:01.782368 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:01.782332    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-clvlk"]
Oct 30 19:02:01.783295 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:01.783266    4525 cpu_manager.go:414] "containerMap: removing container" podUID="1aa70b7e-acd9-48f1-bd02-0db2ea932d98" containerName="revoke-keys"
Oct 30 19:02:01.783295 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:01.783296    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1aa70b7e-acd9-48f1-bd02-0db2ea932d98" containerName="revoke-keys"
Oct 30 19:02:01.783479 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:01.783463    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="1aa70b7e-acd9-48f1-bd02-0db2ea932d98" containerName="revoke-keys"
Oct 30 19:02:01.789699 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:01.787516    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-clvlk"
Oct 30 19:02:01.801705 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:01.797295    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-clvlk"]
Oct 30 19:02:01.801705 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:01.798496    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-c2e3f9-openshift-default-dockercfg-r9f7b\""
Oct 30 19:02:01.960201 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:01.960163    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e14cca94-de16-4d65-b71a-9ece97424174-istio-data\") pod \"e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-clvlk\" (UID: \"e14cca94-de16-4d65-b71a-9ece97424174\") " pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-clvlk"
Oct 30 19:02:01.960201 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:01.960204    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e14cca94-de16-4d65-b71a-9ece97424174-credential-socket\") pod \"e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-clvlk\" (UID: \"e14cca94-de16-4d65-b71a-9ece97424174\") " pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-clvlk"
Oct 30 19:02:01.960415 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:01.960242    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpp2h\" (UniqueName: \"kubernetes.io/projected/e14cca94-de16-4d65-b71a-9ece97424174-kube-api-access-qpp2h\") pod \"e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-clvlk\" (UID: \"e14cca94-de16-4d65-b71a-9ece97424174\") " pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-clvlk"
Oct 30 19:02:01.960415 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:01.960372    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e14cca94-de16-4d65-b71a-9ece97424174-istio-envoy\") pod \"e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-clvlk\" (UID: \"e14cca94-de16-4d65-b71a-9ece97424174\") " pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-clvlk"
Oct 30 19:02:01.960415 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:01.960412    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e14cca94-de16-4d65-b71a-9ece97424174-istio-token\") pod \"e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-clvlk\" (UID: \"e14cca94-de16-4d65-b71a-9ece97424174\") " pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-clvlk"
Oct 30 19:02:01.960518 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:01.960438    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e14cca94-de16-4d65-b71a-9ece97424174-workload-socket\") pod \"e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-clvlk\" (UID: \"e14cca94-de16-4d65-b71a-9ece97424174\") " pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-clvlk"
Oct 30 19:02:01.960580 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:01.960556    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e14cca94-de16-4d65-b71a-9ece97424174-istio-podinfo\") pod \"e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-clvlk\" (UID: \"e14cca94-de16-4d65-b71a-9ece97424174\") " pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-clvlk"
Oct 30 19:02:01.960630 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:01.960619    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e14cca94-de16-4d65-b71a-9ece97424174-istiod-ca-cert\") pod \"e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-clvlk\" (UID: \"e14cca94-de16-4d65-b71a-9ece97424174\") " pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-clvlk"
Oct 30 19:02:01.960741 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:01.960724    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e14cca94-de16-4d65-b71a-9ece97424174-workload-certs\") pod \"e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-clvlk\" (UID: \"e14cca94-de16-4d65-b71a-9ece97424174\") " pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-clvlk"
Oct 30 19:02:02.061898 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:02.061806    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e14cca94-de16-4d65-b71a-9ece97424174-istiod-ca-cert\") pod \"e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-clvlk\" (UID: \"e14cca94-de16-4d65-b71a-9ece97424174\") " pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-clvlk"
Oct 30 19:02:02.061898 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:02.061881    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e14cca94-de16-4d65-b71a-9ece97424174-workload-certs\") pod \"e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-clvlk\" (UID: \"e14cca94-de16-4d65-b71a-9ece97424174\") " pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-clvlk"
Oct 30 19:02:02.062096 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:02.061947    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e14cca94-de16-4d65-b71a-9ece97424174-istio-data\") pod \"e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-clvlk\" (UID: \"e14cca94-de16-4d65-b71a-9ece97424174\") " pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-clvlk"
Oct 30 19:02:02.062096 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:02.061969    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e14cca94-de16-4d65-b71a-9ece97424174-credential-socket\") pod \"e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-clvlk\" (UID: \"e14cca94-de16-4d65-b71a-9ece97424174\") " pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-clvlk"
Oct 30 19:02:02.062096 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:02.062007    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpp2h\" (UniqueName: \"kubernetes.io/projected/e14cca94-de16-4d65-b71a-9ece97424174-kube-api-access-qpp2h\") pod \"e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-clvlk\" (UID: \"e14cca94-de16-4d65-b71a-9ece97424174\") " pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-clvlk"
Oct 30 19:02:02.062096 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:02.062081    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e14cca94-de16-4d65-b71a-9ece97424174-istio-envoy\") pod \"e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-clvlk\" (UID: \"e14cca94-de16-4d65-b71a-9ece97424174\") " pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-clvlk"
Oct 30 19:02:02.062253 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:02.062115    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e14cca94-de16-4d65-b71a-9ece97424174-istio-token\") pod \"e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-clvlk\" (UID: \"e14cca94-de16-4d65-b71a-9ece97424174\") " pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-clvlk"
Oct 30 19:02:02.062253 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:02.062146    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e14cca94-de16-4d65-b71a-9ece97424174-workload-socket\") pod \"e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-clvlk\" (UID: \"e14cca94-de16-4d65-b71a-9ece97424174\") " pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-clvlk"
Oct 30 19:02:02.062253 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:02.062219    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e14cca94-de16-4d65-b71a-9ece97424174-istio-podinfo\") pod \"e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-clvlk\" (UID: \"e14cca94-de16-4d65-b71a-9ece97424174\") " pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-clvlk"
Oct 30 19:02:02.062407 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:02.062300    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e14cca94-de16-4d65-b71a-9ece97424174-istio-data\") pod \"e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-clvlk\" (UID: \"e14cca94-de16-4d65-b71a-9ece97424174\") " pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-clvlk"
Oct 30 19:02:02.062407 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:02.062334    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e14cca94-de16-4d65-b71a-9ece97424174-workload-certs\") pod \"e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-clvlk\" (UID: \"e14cca94-de16-4d65-b71a-9ece97424174\") " pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-clvlk"
Oct 30 19:02:02.062542 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:02.062517    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e14cca94-de16-4d65-b71a-9ece97424174-credential-socket\") pod \"e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-clvlk\" (UID: \"e14cca94-de16-4d65-b71a-9ece97424174\") " pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-clvlk"
Oct 30 19:02:02.062631 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:02.062600    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e14cca94-de16-4d65-b71a-9ece97424174-istiod-ca-cert\") pod \"e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-clvlk\" (UID: \"e14cca94-de16-4d65-b71a-9ece97424174\") " pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-clvlk"
Oct 30 19:02:02.062721 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:02.062634    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e14cca94-de16-4d65-b71a-9ece97424174-workload-socket\") pod \"e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-clvlk\" (UID: \"e14cca94-de16-4d65-b71a-9ece97424174\") " pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-clvlk"
Oct 30 19:02:02.064716 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:02.064654    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e14cca94-de16-4d65-b71a-9ece97424174-istio-envoy\") pod \"e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-clvlk\" (UID: \"e14cca94-de16-4d65-b71a-9ece97424174\") " pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-clvlk"
Oct 30 19:02:02.064856 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:02.064841    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e14cca94-de16-4d65-b71a-9ece97424174-istio-podinfo\") pod \"e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-clvlk\" (UID: \"e14cca94-de16-4d65-b71a-9ece97424174\") " pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-clvlk"
Oct 30 19:02:02.070098 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:02.070073    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e14cca94-de16-4d65-b71a-9ece97424174-istio-token\") pod \"e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-clvlk\" (UID: \"e14cca94-de16-4d65-b71a-9ece97424174\") " pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-clvlk"
Oct 30 19:02:02.070184 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:02.070140    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpp2h\" (UniqueName: \"kubernetes.io/projected/e14cca94-de16-4d65-b71a-9ece97424174-kube-api-access-qpp2h\") pod \"e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-clvlk\" (UID: \"e14cca94-de16-4d65-b71a-9ece97424174\") " pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-clvlk"
Oct 30 19:02:02.132283 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:02.131284    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-clvlk"
Oct 30 19:02:02.552618 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:02:02.552583    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode14cca94_de16_4d65_b71a_9ece97424174.slice/crio-0075f3b7c884c36045ca13159bc0b9f395bdcd346ceb7206da73f9037134521c WatchSource:0}: Error finding container 0075f3b7c884c36045ca13159bc0b9f395bdcd346ceb7206da73f9037134521c: Status 404 returned error can't find the container with id 0075f3b7c884c36045ca13159bc0b9f395bdcd346ceb7206da73f9037134521c
Oct 30 19:02:02.553195 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:02.553168    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-clvlk"]
Oct 30 19:02:02.555142 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:02.555111    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:02:02.555212 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:02.555175    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:02:02.555212 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:02.555203    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:02:02.905708 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:02.900744    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-clvlk" event={"ID":"e14cca94-de16-4d65-b71a-9ece97424174","Type":"ContainerStarted","Data":"60d23ddc31d508eaa5eb17f954bf9625d80aeae99677e0445494225b7535eb60"}
Oct 30 19:02:02.905708 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:02.900814    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-clvlk" event={"ID":"e14cca94-de16-4d65-b71a-9ece97424174","Type":"ContainerStarted","Data":"0075f3b7c884c36045ca13159bc0b9f395bdcd346ceb7206da73f9037134521c"}
Oct 30 19:02:02.970627 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:02.970560    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-clvlk" podStartSLOduration=1.970537709 podStartE2EDuration="1.970537709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:02:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:02:02.960178822 +0000 UTC m=+1545.585848788" watchObservedRunningTime="2026-10-30 19:02:02.970537709 +0000 UTC m=+1545.596207674"
Oct 30 19:02:03.135593 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:03.135538    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-clvlk"
Oct 30 19:02:03.147193 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:03.147162    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-clvlk"
Oct 30 19:02:03.923749 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:03.923230    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-clvlk"
Oct 30 19:02:03.924624 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:03.924545    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-clvlk"
Oct 30 19:02:07.101606 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:07.101570    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm"]
Oct 30 19:02:07.106534 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:07.106503    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm"
Oct 30 19:02:07.125888 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:07.125834    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-baa504-openshift-default-dockercfg-tbb42\""
Oct 30 19:02:07.134050 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:07.134017    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm"]
Oct 30 19:02:07.167714 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:07.167645    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7xvt\" (UniqueName: \"kubernetes.io/projected/6ccf27c7-f6d3-457b-9ff8-69b54f9b9287-kube-api-access-l7xvt\") pod \"e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm\" (UID: \"6ccf27c7-f6d3-457b-9ff8-69b54f9b9287\") " pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm"
Oct 30 19:02:07.168023 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:07.167993    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ccf27c7-f6d3-457b-9ff8-69b54f9b9287-istio-envoy\") pod \"e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm\" (UID: \"6ccf27c7-f6d3-457b-9ff8-69b54f9b9287\") " pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm"
Oct 30 19:02:07.168126 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:07.168099    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ccf27c7-f6d3-457b-9ff8-69b54f9b9287-istiod-ca-cert\") pod \"e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm\" (UID: \"6ccf27c7-f6d3-457b-9ff8-69b54f9b9287\") " pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm"
Oct 30 19:02:07.168184 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:07.168133    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ccf27c7-f6d3-457b-9ff8-69b54f9b9287-credential-socket\") pod \"e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm\" (UID: \"6ccf27c7-f6d3-457b-9ff8-69b54f9b9287\") " pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm"
Oct 30 19:02:07.168252 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:07.168231    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ccf27c7-f6d3-457b-9ff8-69b54f9b9287-workload-certs\") pod \"e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm\" (UID: \"6ccf27c7-f6d3-457b-9ff8-69b54f9b9287\") " pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm"
Oct 30 19:02:07.168510 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:07.168482    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ccf27c7-f6d3-457b-9ff8-69b54f9b9287-istio-data\") pod \"e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm\" (UID: \"6ccf27c7-f6d3-457b-9ff8-69b54f9b9287\") " pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm"
Oct 30 19:02:07.168606 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:07.168579    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ccf27c7-f6d3-457b-9ff8-69b54f9b9287-istio-podinfo\") pod \"e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm\" (UID: \"6ccf27c7-f6d3-457b-9ff8-69b54f9b9287\") " pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm"
Oct 30 19:02:07.168709 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:07.168618    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ccf27c7-f6d3-457b-9ff8-69b54f9b9287-workload-socket\") pod \"e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm\" (UID: \"6ccf27c7-f6d3-457b-9ff8-69b54f9b9287\") " pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm"
Oct 30 19:02:07.168709 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:07.168655    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ccf27c7-f6d3-457b-9ff8-69b54f9b9287-istio-token\") pod \"e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm\" (UID: \"6ccf27c7-f6d3-457b-9ff8-69b54f9b9287\") " pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm"
Oct 30 19:02:07.275020 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:07.274982    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ccf27c7-f6d3-457b-9ff8-69b54f9b9287-istio-envoy\") pod \"e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm\" (UID: \"6ccf27c7-f6d3-457b-9ff8-69b54f9b9287\") " pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm"
Oct 30 19:02:07.275204 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:07.275052    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ccf27c7-f6d3-457b-9ff8-69b54f9b9287-istiod-ca-cert\") pod \"e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm\" (UID: \"6ccf27c7-f6d3-457b-9ff8-69b54f9b9287\") " pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm"
Oct 30 19:02:07.275204 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:07.275089    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ccf27c7-f6d3-457b-9ff8-69b54f9b9287-credential-socket\") pod \"e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm\" (UID: \"6ccf27c7-f6d3-457b-9ff8-69b54f9b9287\") " pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm"
Oct 30 19:02:07.275204 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:07.275160    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ccf27c7-f6d3-457b-9ff8-69b54f9b9287-workload-certs\") pod \"e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm\" (UID: \"6ccf27c7-f6d3-457b-9ff8-69b54f9b9287\") " pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm"
Oct 30 19:02:07.275406 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:07.275232    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ccf27c7-f6d3-457b-9ff8-69b54f9b9287-istio-data\") pod \"e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm\" (UID: \"6ccf27c7-f6d3-457b-9ff8-69b54f9b9287\") " pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm"
Oct 30 19:02:07.275406 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:07.275332    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ccf27c7-f6d3-457b-9ff8-69b54f9b9287-istio-podinfo\") pod \"e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm\" (UID: \"6ccf27c7-f6d3-457b-9ff8-69b54f9b9287\") " pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm"
Oct 30 19:02:07.275406 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:07.275371    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ccf27c7-f6d3-457b-9ff8-69b54f9b9287-workload-socket\") pod \"e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm\" (UID: \"6ccf27c7-f6d3-457b-9ff8-69b54f9b9287\") " pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm"
Oct 30 19:02:07.275588 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:07.275417    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ccf27c7-f6d3-457b-9ff8-69b54f9b9287-istio-token\") pod \"e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm\" (UID: \"6ccf27c7-f6d3-457b-9ff8-69b54f9b9287\") " pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm"
Oct 30 19:02:07.275588 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:07.275498    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7xvt\" (UniqueName: \"kubernetes.io/projected/6ccf27c7-f6d3-457b-9ff8-69b54f9b9287-kube-api-access-l7xvt\") pod \"e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm\" (UID: \"6ccf27c7-f6d3-457b-9ff8-69b54f9b9287\") " pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm"
Oct 30 19:02:07.275964 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:07.275937    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ccf27c7-f6d3-457b-9ff8-69b54f9b9287-workload-certs\") pod \"e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm\" (UID: \"6ccf27c7-f6d3-457b-9ff8-69b54f9b9287\") " pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm"
Oct 30 19:02:07.276464 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:07.276435    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ccf27c7-f6d3-457b-9ff8-69b54f9b9287-credential-socket\") pod \"e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm\" (UID: \"6ccf27c7-f6d3-457b-9ff8-69b54f9b9287\") " pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm"
Oct 30 19:02:07.276909 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:07.276866    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ccf27c7-f6d3-457b-9ff8-69b54f9b9287-istio-data\") pod \"e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm\" (UID: \"6ccf27c7-f6d3-457b-9ff8-69b54f9b9287\") " pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm"
Oct 30 19:02:07.277251 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:07.276900    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ccf27c7-f6d3-457b-9ff8-69b54f9b9287-workload-socket\") pod \"e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm\" (UID: \"6ccf27c7-f6d3-457b-9ff8-69b54f9b9287\") " pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm"
Oct 30 19:02:07.279597 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:07.279567    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ccf27c7-f6d3-457b-9ff8-69b54f9b9287-istiod-ca-cert\") pod \"e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm\" (UID: \"6ccf27c7-f6d3-457b-9ff8-69b54f9b9287\") " pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm"
Oct 30 19:02:07.281521 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:07.281495    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ccf27c7-f6d3-457b-9ff8-69b54f9b9287-istio-envoy\") pod \"e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm\" (UID: \"6ccf27c7-f6d3-457b-9ff8-69b54f9b9287\") " pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm"
Oct 30 19:02:07.281806 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:07.281783    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ccf27c7-f6d3-457b-9ff8-69b54f9b9287-istio-podinfo\") pod \"e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm\" (UID: \"6ccf27c7-f6d3-457b-9ff8-69b54f9b9287\") " pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm"
Oct 30 19:02:07.291256 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:07.291224    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ccf27c7-f6d3-457b-9ff8-69b54f9b9287-istio-token\") pod \"e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm\" (UID: \"6ccf27c7-f6d3-457b-9ff8-69b54f9b9287\") " pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm"
Oct 30 19:02:07.291641 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:07.291613    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7xvt\" (UniqueName: \"kubernetes.io/projected/6ccf27c7-f6d3-457b-9ff8-69b54f9b9287-kube-api-access-l7xvt\") pod \"e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm\" (UID: \"6ccf27c7-f6d3-457b-9ff8-69b54f9b9287\") " pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm"
Oct 30 19:02:07.437875 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:07.437838    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm"
Oct 30 19:02:07.747917 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:02:07.747884    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ccf27c7_f6d3_457b_9ff8_69b54f9b9287.slice/crio-1d1aa37a8a7180af40c5bf727f24d6847461386ca170b40f1269f527fbdcebac WatchSource:0}: Error finding container 1d1aa37a8a7180af40c5bf727f24d6847461386ca170b40f1269f527fbdcebac: Status 404 returned error can't find the container with id 1d1aa37a8a7180af40c5bf727f24d6847461386ca170b40f1269f527fbdcebac
Oct 30 19:02:07.750253 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:07.750227    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm"]
Oct 30 19:02:07.750647 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:07.750613    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:02:07.750753 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:07.750734    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:02:07.750831 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:07.750785    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:02:08.001032 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:08.000937    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm" event={"ID":"6ccf27c7-f6d3-457b-9ff8-69b54f9b9287","Type":"ContainerStarted","Data":"325128f3e14fb21cd06d956ada021baba82277fcd141493de35a31bccde645a0"}
Oct 30 19:02:08.001032 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:08.001001    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm" event={"ID":"6ccf27c7-f6d3-457b-9ff8-69b54f9b9287","Type":"ContainerStarted","Data":"1d1aa37a8a7180af40c5bf727f24d6847461386ca170b40f1269f527fbdcebac"}
Oct 30 19:02:08.025557 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:08.025496    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm" podStartSLOduration=1.025481756 podStartE2EDuration="1.025481756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:02:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:02:08.023144881 +0000 UTC m=+1550.648814848" watchObservedRunningTime="2026-10-30 19:02:08.025481756 +0000 UTC m=+1550.651151720"
Oct 30 19:02:08.438748 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:08.438712    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm"
Oct 30 19:02:08.444611 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:08.444583    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm"
Oct 30 19:02:09.084705 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:09.080539    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm"
Oct 30 19:02:09.101705 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:09.095971    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm"
Oct 30 19:02:15.050355 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:15.049140    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dhv28"]
Oct 30 19:02:15.082745 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:15.082666    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dhv28"]
Oct 30 19:02:15.082913 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:15.082803    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dhv28"
Oct 30 19:02:15.130935 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:15.130897    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd3970a9-8e49-456f-b7c7-b6af7320dfab-credential-socket\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-dhv28\" (UID: \"bd3970a9-8e49-456f-b7c7-b6af7320dfab\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dhv28"
Oct 30 19:02:15.131108 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:15.130974    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd3970a9-8e49-456f-b7c7-b6af7320dfab-istio-envoy\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-dhv28\" (UID: \"bd3970a9-8e49-456f-b7c7-b6af7320dfab\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dhv28"
Oct 30 19:02:15.131108 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:15.131071    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd3970a9-8e49-456f-b7c7-b6af7320dfab-istio-token\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-dhv28\" (UID: \"bd3970a9-8e49-456f-b7c7-b6af7320dfab\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dhv28"
Oct 30 19:02:15.131192 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:15.131113    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd3970a9-8e49-456f-b7c7-b6af7320dfab-workload-certs\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-dhv28\" (UID: \"bd3970a9-8e49-456f-b7c7-b6af7320dfab\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dhv28"
Oct 30 19:02:15.131192 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:15.131132    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd3970a9-8e49-456f-b7c7-b6af7320dfab-workload-socket\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-dhv28\" (UID: \"bd3970a9-8e49-456f-b7c7-b6af7320dfab\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dhv28"
Oct 30 19:02:15.131261 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:15.131218    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd3970a9-8e49-456f-b7c7-b6af7320dfab-istio-data\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-dhv28\" (UID: \"bd3970a9-8e49-456f-b7c7-b6af7320dfab\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dhv28"
Oct 30 19:02:15.131301 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:15.131286    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd3970a9-8e49-456f-b7c7-b6af7320dfab-istio-podinfo\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-dhv28\" (UID: \"bd3970a9-8e49-456f-b7c7-b6af7320dfab\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dhv28"
Oct 30 19:02:15.131357 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:15.131339    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg7c5\" (UniqueName: \"kubernetes.io/projected/bd3970a9-8e49-456f-b7c7-b6af7320dfab-kube-api-access-vg7c5\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-dhv28\" (UID: \"bd3970a9-8e49-456f-b7c7-b6af7320dfab\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dhv28"
Oct 30 19:02:15.131405 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:15.131393    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd3970a9-8e49-456f-b7c7-b6af7320dfab-istiod-ca-cert\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-dhv28\" (UID: \"bd3970a9-8e49-456f-b7c7-b6af7320dfab\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dhv28"
Oct 30 19:02:15.232616 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:15.232578    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg7c5\" (UniqueName: \"kubernetes.io/projected/bd3970a9-8e49-456f-b7c7-b6af7320dfab-kube-api-access-vg7c5\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-dhv28\" (UID: \"bd3970a9-8e49-456f-b7c7-b6af7320dfab\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dhv28"
Oct 30 19:02:15.232616 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:15.232620    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd3970a9-8e49-456f-b7c7-b6af7320dfab-istiod-ca-cert\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-dhv28\" (UID: \"bd3970a9-8e49-456f-b7c7-b6af7320dfab\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dhv28"
Oct 30 19:02:15.232851 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:15.232730    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd3970a9-8e49-456f-b7c7-b6af7320dfab-credential-socket\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-dhv28\" (UID: \"bd3970a9-8e49-456f-b7c7-b6af7320dfab\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dhv28"
Oct 30 19:02:15.232851 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:15.232788    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd3970a9-8e49-456f-b7c7-b6af7320dfab-istio-envoy\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-dhv28\" (UID: \"bd3970a9-8e49-456f-b7c7-b6af7320dfab\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dhv28"
Oct 30 19:02:15.232851 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:15.232834    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd3970a9-8e49-456f-b7c7-b6af7320dfab-istio-token\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-dhv28\" (UID: \"bd3970a9-8e49-456f-b7c7-b6af7320dfab\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dhv28"
Oct 30 19:02:15.233009 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:15.232863    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd3970a9-8e49-456f-b7c7-b6af7320dfab-workload-certs\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-dhv28\" (UID: \"bd3970a9-8e49-456f-b7c7-b6af7320dfab\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dhv28"
Oct 30 19:02:15.233009 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:15.232891    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd3970a9-8e49-456f-b7c7-b6af7320dfab-workload-socket\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-dhv28\" (UID: \"bd3970a9-8e49-456f-b7c7-b6af7320dfab\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dhv28"
Oct 30 19:02:15.233009 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:15.232928    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd3970a9-8e49-456f-b7c7-b6af7320dfab-istio-data\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-dhv28\" (UID: \"bd3970a9-8e49-456f-b7c7-b6af7320dfab\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dhv28"
Oct 30 19:02:15.233009 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:15.232985    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd3970a9-8e49-456f-b7c7-b6af7320dfab-istio-podinfo\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-dhv28\" (UID: \"bd3970a9-8e49-456f-b7c7-b6af7320dfab\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dhv28"
Oct 30 19:02:15.233239 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:15.233164    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd3970a9-8e49-456f-b7c7-b6af7320dfab-credential-socket\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-dhv28\" (UID: \"bd3970a9-8e49-456f-b7c7-b6af7320dfab\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dhv28"
Oct 30 19:02:15.233316 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:15.233251    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd3970a9-8e49-456f-b7c7-b6af7320dfab-workload-certs\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-dhv28\" (UID: \"bd3970a9-8e49-456f-b7c7-b6af7320dfab\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dhv28"
Oct 30 19:02:15.233416 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:15.233392    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd3970a9-8e49-456f-b7c7-b6af7320dfab-workload-socket\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-dhv28\" (UID: \"bd3970a9-8e49-456f-b7c7-b6af7320dfab\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dhv28"
Oct 30 19:02:15.233483 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:15.233447    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd3970a9-8e49-456f-b7c7-b6af7320dfab-istiod-ca-cert\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-dhv28\" (UID: \"bd3970a9-8e49-456f-b7c7-b6af7320dfab\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dhv28"
Oct 30 19:02:15.233546 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:15.233520    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd3970a9-8e49-456f-b7c7-b6af7320dfab-istio-data\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-dhv28\" (UID: \"bd3970a9-8e49-456f-b7c7-b6af7320dfab\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dhv28"
Oct 30 19:02:15.235302 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:15.235275    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd3970a9-8e49-456f-b7c7-b6af7320dfab-istio-envoy\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-dhv28\" (UID: \"bd3970a9-8e49-456f-b7c7-b6af7320dfab\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dhv28"
Oct 30 19:02:15.235572 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:15.235556    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd3970a9-8e49-456f-b7c7-b6af7320dfab-istio-podinfo\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-dhv28\" (UID: \"bd3970a9-8e49-456f-b7c7-b6af7320dfab\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dhv28"
Oct 30 19:02:15.241076 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:15.241057    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd3970a9-8e49-456f-b7c7-b6af7320dfab-istio-token\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-dhv28\" (UID: \"bd3970a9-8e49-456f-b7c7-b6af7320dfab\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dhv28"
Oct 30 19:02:15.241424 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:15.241407    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg7c5\" (UniqueName: \"kubernetes.io/projected/bd3970a9-8e49-456f-b7c7-b6af7320dfab-kube-api-access-vg7c5\") pod \"e2e-worker-w4-0458df-openshift-default-7c86d79f79-dhv28\" (UID: \"bd3970a9-8e49-456f-b7c7-b6af7320dfab\") " pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dhv28"
Oct 30 19:02:15.401199 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:15.401168    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dhv28"
Oct 30 19:02:15.689429 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:02:15.689403    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd3970a9_8e49_456f_b7c7_b6af7320dfab.slice/crio-06166f2f56488c3c9ba3c28cc0b565a2937e61e6995419e97d605498dca091f5 WatchSource:0}: Error finding container 06166f2f56488c3c9ba3c28cc0b565a2937e61e6995419e97d605498dca091f5: Status 404 returned error can't find the container with id 06166f2f56488c3c9ba3c28cc0b565a2937e61e6995419e97d605498dca091f5
Oct 30 19:02:15.698723 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:15.690874    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dhv28"]
Oct 30 19:02:15.698723 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:15.694225    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:02:15.698723 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:15.694307    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:02:15.698723 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:15.694344    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:02:15.745821 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:15.745788    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944"]
Oct 30 19:02:15.752009 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:15.751981    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944"
Oct 30 19:02:15.756974 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:15.756923    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944"]
Oct 30 19:02:15.764345 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:15.764317    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-ed012e-openshift-default-dockercfg-ls79r\""
Oct 30 19:02:15.853705 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:15.845674    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92a5cb37-4cd6-466c-bf3b-740d3573da07-istio-podinfo\") pod \"e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944\" (UID: \"92a5cb37-4cd6-466c-bf3b-740d3573da07\") " pod="openshift-ingress/e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944"
Oct 30 19:02:15.853705 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:15.845779    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92a5cb37-4cd6-466c-bf3b-740d3573da07-credential-socket\") pod \"e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944\" (UID: \"92a5cb37-4cd6-466c-bf3b-740d3573da07\") " pod="openshift-ingress/e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944"
Oct 30 19:02:15.853705 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:15.846395    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92a5cb37-4cd6-466c-bf3b-740d3573da07-istio-envoy\") pod \"e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944\" (UID: \"92a5cb37-4cd6-466c-bf3b-740d3573da07\") " pod="openshift-ingress/e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944"
Oct 30 19:02:15.853705 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:15.846426    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92a5cb37-4cd6-466c-bf3b-740d3573da07-istio-data\") pod \"e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944\" (UID: \"92a5cb37-4cd6-466c-bf3b-740d3573da07\") " pod="openshift-ingress/e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944"
Oct 30 19:02:15.853705 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:15.846476    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhkdd\" (UniqueName: \"kubernetes.io/projected/92a5cb37-4cd6-466c-bf3b-740d3573da07-kube-api-access-zhkdd\") pod \"e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944\" (UID: \"92a5cb37-4cd6-466c-bf3b-740d3573da07\") " pod="openshift-ingress/e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944"
Oct 30 19:02:15.853705 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:15.846594    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92a5cb37-4cd6-466c-bf3b-740d3573da07-istiod-ca-cert\") pod \"e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944\" (UID: \"92a5cb37-4cd6-466c-bf3b-740d3573da07\") " pod="openshift-ingress/e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944"
Oct 30 19:02:15.853705 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:15.846729    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92a5cb37-4cd6-466c-bf3b-740d3573da07-workload-socket\") pod \"e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944\" (UID: \"92a5cb37-4cd6-466c-bf3b-740d3573da07\") " pod="openshift-ingress/e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944"
Oct 30 19:02:15.853705 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:15.846822    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92a5cb37-4cd6-466c-bf3b-740d3573da07-workload-certs\") pod \"e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944\" (UID: \"92a5cb37-4cd6-466c-bf3b-740d3573da07\") " pod="openshift-ingress/e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944"
Oct 30 19:02:15.853705 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:15.846849    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92a5cb37-4cd6-466c-bf3b-740d3573da07-istio-token\") pod \"e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944\" (UID: \"92a5cb37-4cd6-466c-bf3b-740d3573da07\") " pod="openshift-ingress/e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944"
Oct 30 19:02:15.858574 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:15.857702    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn"]
Oct 30 19:02:15.864149 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:15.864131    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn"
Oct 30 19:02:15.882947 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:15.882921    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-ef6be7-openshift-default-dockercfg-rpwj4\""
Oct 30 19:02:15.885420 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:15.885392    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn"]
Oct 30 19:02:15.950642 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:15.950553    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e77e530a-8e80-4371-840e-b2345bf68449-workload-certs\") pod \"e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn\" (UID: \"e77e530a-8e80-4371-840e-b2345bf68449\") " pod="openshift-ingress/e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn"
Oct 30 19:02:15.951018 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:15.950997    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e77e530a-8e80-4371-840e-b2345bf68449-workload-socket\") pod \"e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn\" (UID: \"e77e530a-8e80-4371-840e-b2345bf68449\") " pod="openshift-ingress/e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn"
Oct 30 19:02:15.951204 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:15.951187    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92a5cb37-4cd6-466c-bf3b-740d3573da07-istiod-ca-cert\") pod \"e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944\" (UID: \"92a5cb37-4cd6-466c-bf3b-740d3573da07\") " pod="openshift-ingress/e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944"
Oct 30 19:02:15.952172 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:15.952149    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92a5cb37-4cd6-466c-bf3b-740d3573da07-workload-socket\") pod \"e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944\" (UID: \"92a5cb37-4cd6-466c-bf3b-740d3573da07\") " pod="openshift-ingress/e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944"
Oct 30 19:02:15.952856 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:15.952828    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e77e530a-8e80-4371-840e-b2345bf68449-istio-podinfo\") pod \"e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn\" (UID: \"e77e530a-8e80-4371-840e-b2345bf68449\") " pod="openshift-ingress/e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn"
Oct 30 19:02:15.953075 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:15.953059    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92a5cb37-4cd6-466c-bf3b-740d3573da07-workload-certs\") pod \"e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944\" (UID: \"92a5cb37-4cd6-466c-bf3b-740d3573da07\") " pod="openshift-ingress/e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944"
Oct 30 19:02:15.953538 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:15.953521    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92a5cb37-4cd6-466c-bf3b-740d3573da07-istio-token\") pod \"e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944\" (UID: \"92a5cb37-4cd6-466c-bf3b-740d3573da07\") " pod="openshift-ingress/e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944"
Oct 30 19:02:15.954732 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:15.954711    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92a5cb37-4cd6-466c-bf3b-740d3573da07-istio-podinfo\") pod \"e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944\" (UID: \"92a5cb37-4cd6-466c-bf3b-740d3573da07\") " pod="openshift-ingress/e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944"
Oct 30 19:02:15.955339 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:15.955302    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e77e530a-8e80-4371-840e-b2345bf68449-istio-data\") pod \"e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn\" (UID: \"e77e530a-8e80-4371-840e-b2345bf68449\") " pod="openshift-ingress/e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn"
Oct 30 19:02:15.955561 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:15.955543    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92a5cb37-4cd6-466c-bf3b-740d3573da07-credential-socket\") pod \"e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944\" (UID: \"92a5cb37-4cd6-466c-bf3b-740d3573da07\") " pod="openshift-ingress/e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944"
Oct 30 19:02:15.956052 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:15.956035    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e77e530a-8e80-4371-840e-b2345bf68449-credential-socket\") pod \"e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn\" (UID: \"e77e530a-8e80-4371-840e-b2345bf68449\") " pod="openshift-ingress/e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn"
Oct 30 19:02:15.956292 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:15.956273    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e77e530a-8e80-4371-840e-b2345bf68449-istio-envoy\") pod \"e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn\" (UID: \"e77e530a-8e80-4371-840e-b2345bf68449\") " pod="openshift-ingress/e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn"
Oct 30 19:02:15.956475 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:15.956461    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e77e530a-8e80-4371-840e-b2345bf68449-istiod-ca-cert\") pod \"e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn\" (UID: \"e77e530a-8e80-4371-840e-b2345bf68449\") " pod="openshift-ingress/e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn"
Oct 30 19:02:15.956650 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:15.956635    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92a5cb37-4cd6-466c-bf3b-740d3573da07-istio-envoy\") pod \"e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944\" (UID: \"92a5cb37-4cd6-466c-bf3b-740d3573da07\") " pod="openshift-ingress/e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944"
Oct 30 19:02:15.957125 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:15.957107    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92a5cb37-4cd6-466c-bf3b-740d3573da07-istio-data\") pod \"e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944\" (UID: \"92a5cb37-4cd6-466c-bf3b-740d3573da07\") " pod="openshift-ingress/e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944"
Oct 30 19:02:15.957556 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:15.957540    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e77e530a-8e80-4371-840e-b2345bf68449-istio-token\") pod \"e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn\" (UID: \"e77e530a-8e80-4371-840e-b2345bf68449\") " pod="openshift-ingress/e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn"
Oct 30 19:02:15.957765 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:15.957747    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhkdd\" (UniqueName: \"kubernetes.io/projected/92a5cb37-4cd6-466c-bf3b-740d3573da07-kube-api-access-zhkdd\") pod \"e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944\" (UID: \"92a5cb37-4cd6-466c-bf3b-740d3573da07\") " pod="openshift-ingress/e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944"
Oct 30 19:02:15.958209 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:15.958189    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcb67\" (UniqueName: \"kubernetes.io/projected/e77e530a-8e80-4371-840e-b2345bf68449-kube-api-access-zcb67\") pod \"e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn\" (UID: \"e77e530a-8e80-4371-840e-b2345bf68449\") " pod="openshift-ingress/e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn"
Oct 30 19:02:15.958477 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:15.957490    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92a5cb37-4cd6-466c-bf3b-740d3573da07-istio-data\") pod \"e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944\" (UID: \"92a5cb37-4cd6-466c-bf3b-740d3573da07\") " pod="openshift-ingress/e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944"
Oct 30 19:02:15.958598 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:15.953469    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92a5cb37-4cd6-466c-bf3b-740d3573da07-workload-certs\") pod \"e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944\" (UID: \"92a5cb37-4cd6-466c-bf3b-740d3573da07\") " pod="openshift-ingress/e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944"
Oct 30 19:02:15.958733 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:15.952069    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92a5cb37-4cd6-466c-bf3b-740d3573da07-istiod-ca-cert\") pod \"e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944\" (UID: \"92a5cb37-4cd6-466c-bf3b-740d3573da07\") " pod="openshift-ingress/e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944"
Oct 30 19:02:15.958833 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:15.955968    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92a5cb37-4cd6-466c-bf3b-740d3573da07-credential-socket\") pod \"e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944\" (UID: \"92a5cb37-4cd6-466c-bf3b-740d3573da07\") " pod="openshift-ingress/e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944"
Oct 30 19:02:15.958924 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:15.952605    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92a5cb37-4cd6-466c-bf3b-740d3573da07-workload-socket\") pod \"e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944\" (UID: \"92a5cb37-4cd6-466c-bf3b-740d3573da07\") " pod="openshift-ingress/e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944"
Oct 30 19:02:15.962240 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:15.962199    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92a5cb37-4cd6-466c-bf3b-740d3573da07-istio-envoy\") pod \"e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944\" (UID: \"92a5cb37-4cd6-466c-bf3b-740d3573da07\") " pod="openshift-ingress/e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944"
Oct 30 19:02:15.964586 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:15.964556    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92a5cb37-4cd6-466c-bf3b-740d3573da07-istio-token\") pod \"e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944\" (UID: \"92a5cb37-4cd6-466c-bf3b-740d3573da07\") " pod="openshift-ingress/e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944"
Oct 30 19:02:15.969149 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:15.968247    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92a5cb37-4cd6-466c-bf3b-740d3573da07-istio-podinfo\") pod \"e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944\" (UID: \"92a5cb37-4cd6-466c-bf3b-740d3573da07\") " pod="openshift-ingress/e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944"
Oct 30 19:02:15.975092 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:15.975045    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhkdd\" (UniqueName: \"kubernetes.io/projected/92a5cb37-4cd6-466c-bf3b-740d3573da07-kube-api-access-zhkdd\") pod \"e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944\" (UID: \"92a5cb37-4cd6-466c-bf3b-740d3573da07\") " pod="openshift-ingress/e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944"
Oct 30 19:02:16.059428 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:16.059384    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e77e530a-8e80-4371-840e-b2345bf68449-credential-socket\") pod \"e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn\" (UID: \"e77e530a-8e80-4371-840e-b2345bf68449\") " pod="openshift-ingress/e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn"
Oct 30 19:02:16.059959 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:16.059450    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e77e530a-8e80-4371-840e-b2345bf68449-istio-envoy\") pod \"e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn\" (UID: \"e77e530a-8e80-4371-840e-b2345bf68449\") " pod="openshift-ingress/e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn"
Oct 30 19:02:16.059959 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:16.059485    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e77e530a-8e80-4371-840e-b2345bf68449-istiod-ca-cert\") pod \"e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn\" (UID: \"e77e530a-8e80-4371-840e-b2345bf68449\") " pod="openshift-ingress/e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn"
Oct 30 19:02:16.059959 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:16.059520    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e77e530a-8e80-4371-840e-b2345bf68449-istio-token\") pod \"e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn\" (UID: \"e77e530a-8e80-4371-840e-b2345bf68449\") " pod="openshift-ingress/e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn"
Oct 30 19:02:16.059959 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:16.059561    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcb67\" (UniqueName: \"kubernetes.io/projected/e77e530a-8e80-4371-840e-b2345bf68449-kube-api-access-zcb67\") pod \"e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn\" (UID: \"e77e530a-8e80-4371-840e-b2345bf68449\") " pod="openshift-ingress/e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn"
Oct 30 19:02:16.059959 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:16.059594    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e77e530a-8e80-4371-840e-b2345bf68449-workload-certs\") pod \"e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn\" (UID: \"e77e530a-8e80-4371-840e-b2345bf68449\") " pod="openshift-ingress/e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn"
Oct 30 19:02:16.059959 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:16.059662    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e77e530a-8e80-4371-840e-b2345bf68449-workload-socket\") pod \"e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn\" (UID: \"e77e530a-8e80-4371-840e-b2345bf68449\") " pod="openshift-ingress/e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn"
Oct 30 19:02:16.059959 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:16.059777    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e77e530a-8e80-4371-840e-b2345bf68449-istio-podinfo\") pod \"e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn\" (UID: \"e77e530a-8e80-4371-840e-b2345bf68449\") " pod="openshift-ingress/e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn"
Oct 30 19:02:16.059959 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:16.059900    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e77e530a-8e80-4371-840e-b2345bf68449-istio-data\") pod \"e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn\" (UID: \"e77e530a-8e80-4371-840e-b2345bf68449\") " pod="openshift-ingress/e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn"
Oct 30 19:02:16.060420 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:16.060165    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e77e530a-8e80-4371-840e-b2345bf68449-workload-certs\") pod \"e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn\" (UID: \"e77e530a-8e80-4371-840e-b2345bf68449\") " pod="openshift-ingress/e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn"
Oct 30 19:02:16.060420 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:16.060255    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e77e530a-8e80-4371-840e-b2345bf68449-istio-data\") pod \"e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn\" (UID: \"e77e530a-8e80-4371-840e-b2345bf68449\") " pod="openshift-ingress/e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn"
Oct 30 19:02:16.060420 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:16.060284    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e77e530a-8e80-4371-840e-b2345bf68449-istiod-ca-cert\") pod \"e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn\" (UID: \"e77e530a-8e80-4371-840e-b2345bf68449\") " pod="openshift-ingress/e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn"
Oct 30 19:02:16.060710 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:16.060654    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e77e530a-8e80-4371-840e-b2345bf68449-workload-socket\") pod \"e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn\" (UID: \"e77e530a-8e80-4371-840e-b2345bf68449\") " pod="openshift-ingress/e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn"
Oct 30 19:02:16.060710 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:16.060880    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e77e530a-8e80-4371-840e-b2345bf68449-credential-socket\") pod \"e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn\" (UID: \"e77e530a-8e80-4371-840e-b2345bf68449\") " pod="openshift-ingress/e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn"
Oct 30 19:02:16.063627 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:16.063424    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e77e530a-8e80-4371-840e-b2345bf68449-istio-podinfo\") pod \"e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn\" (UID: \"e77e530a-8e80-4371-840e-b2345bf68449\") " pod="openshift-ingress/e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn"
Oct 30 19:02:16.065093 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:16.065066    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e77e530a-8e80-4371-840e-b2345bf68449-istio-envoy\") pod \"e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn\" (UID: \"e77e530a-8e80-4371-840e-b2345bf68449\") " pod="openshift-ingress/e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn"
Oct 30 19:02:16.072363 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:16.072209    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944"
Oct 30 19:02:16.075110 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:16.075086    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e77e530a-8e80-4371-840e-b2345bf68449-istio-token\") pod \"e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn\" (UID: \"e77e530a-8e80-4371-840e-b2345bf68449\") " pod="openshift-ingress/e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn"
Oct 30 19:02:16.078030 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:16.078005    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcb67\" (UniqueName: \"kubernetes.io/projected/e77e530a-8e80-4371-840e-b2345bf68449-kube-api-access-zcb67\") pod \"e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn\" (UID: \"e77e530a-8e80-4371-840e-b2345bf68449\") " pod="openshift-ingress/e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn"
Oct 30 19:02:16.140150 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:16.140119    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dhv28" event={"ID":"bd3970a9-8e49-456f-b7c7-b6af7320dfab","Type":"ContainerStarted","Data":"660496b207a9a7640e6fe57760be261b29f5df1864cbcbfea328fc946d68dcae"}
Oct 30 19:02:16.140515 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:16.140492    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dhv28" event={"ID":"bd3970a9-8e49-456f-b7c7-b6af7320dfab","Type":"ContainerStarted","Data":"06166f2f56488c3c9ba3c28cc0b565a2937e61e6995419e97d605498dca091f5"}
Oct 30 19:02:16.190557 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:16.190524    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn"
Oct 30 19:02:16.414708 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:16.414267    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dhv28"
Oct 30 19:02:16.421704 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:16.420486    4525 patch_prober.go:29] interesting pod/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dhv28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Oct 30 19:02:16.421704 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:16.420545    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dhv28" podUID="bd3970a9-8e49-456f-b7c7-b6af7320dfab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Oct 30 19:02:16.428006 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:16.426717    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dhv28" podStartSLOduration=1.426698447 podStartE2EDuration="1.426698447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:02:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:02:16.171284989 +0000 UTC m=+1558.796954976" watchObservedRunningTime="2026-10-30 19:02:16.426698447 +0000 UTC m=+1559.052368411"
Oct 30 19:02:16.428006 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:16.426795    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:02:16.428006 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:16.426897    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:02:16.428006 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:16.426964    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:02:16.428006 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:16.427043    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944"]
Oct 30 19:02:16.819299 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:16.819156    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn"]
Oct 30 19:02:16.821598 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:16.821562    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:02:16.821882 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:16.821842    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:02:16.822198 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:16.822152    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:02:17.254483 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:17.254444    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn" event={"ID":"e77e530a-8e80-4371-840e-b2345bf68449","Type":"ContainerStarted","Data":"bc1fc55950a91a7538ea3218d9447d112374babdbb61184d2633be1fa66bd903"}
Oct 30 19:02:17.254910 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:17.254536    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn" event={"ID":"e77e530a-8e80-4371-840e-b2345bf68449","Type":"ContainerStarted","Data":"00921869f6c1d6c2ecdca138d2be84ab793f54199fa37ea497455ed2a7cbe0bf"}
Oct 30 19:02:17.257100 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:17.257065    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944" event={"ID":"92a5cb37-4cd6-466c-bf3b-740d3573da07","Type":"ContainerStarted","Data":"2ff7347f12d88897fab512adef677fc96a7de0cae3bce0381e7b5ac746082424"}
Oct 30 19:02:17.257239 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:17.257126    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944" event={"ID":"92a5cb37-4cd6-466c-bf3b-740d3573da07","Type":"ContainerStarted","Data":"f1b4e445fb70c4a339b84efa71bd83fdf020173ece75d4a93046b4907a453994"}
Oct 30 19:02:17.339832 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:17.339761    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn" podStartSLOduration=2.339741342 podStartE2EDuration="2.339741342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:02:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:02:17.317069971 +0000 UTC m=+1559.942739950" watchObservedRunningTime="2026-10-30 19:02:17.339741342 +0000 UTC m=+1559.965411307"
Oct 30 19:02:17.340431 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:17.340388    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944" podStartSLOduration=2.340377882 podStartE2EDuration="2.340377882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:02:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:02:17.338398331 +0000 UTC m=+1559.964068320" watchObservedRunningTime="2026-10-30 19:02:17.340377882 +0000 UTC m=+1559.966047846"
Oct 30 19:02:17.402707 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:17.402225    4525 patch_prober.go:29] interesting pod/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dhv28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Oct 30 19:02:17.402707 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:17.402292    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dhv28" podUID="bd3970a9-8e49-456f-b7c7-b6af7320dfab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Oct 30 19:02:17.428292 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:02:17.428252    4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d49l" containerName="extractor"
Oct 30 19:02:18.073744 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:18.073705    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944"
Oct 30 19:02:18.110109 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:18.110073    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944"
Oct 30 19:02:18.197706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:18.193771    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn"
Oct 30 19:02:18.214853 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:18.211999    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn"
Oct 30 19:02:18.268874 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:18.268842    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944"
Oct 30 19:02:18.269418 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:18.268907    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn"
Oct 30 19:02:18.269739 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:18.269716    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944"
Oct 30 19:02:18.270423 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:18.270404    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn"
Oct 30 19:02:18.410845 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:18.410753    4525 patch_prober.go:29] interesting pod/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dhv28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Oct 30 19:02:18.411063 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:18.410851    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dhv28" podUID="bd3970a9-8e49-456f-b7c7-b6af7320dfab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Oct 30 19:02:19.405709 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:19.405121    4525 patch_prober.go:29] interesting pod/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dhv28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Oct 30 19:02:19.405709 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:19.405184    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dhv28" podUID="bd3970a9-8e49-456f-b7c7-b6af7320dfab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Oct 30 19:02:19.539435 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:19.539352    4525 scope.go:122] "RemoveContainer" containerID="9e0f03f6326cc853b605222094b1d29ab448b78977eb409a393a6d19402c9c55"
Oct 30 19:02:19.563501 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:19.563479    4525 scope.go:122] "RemoveContainer" containerID="5ea4ffe143407064eef7cfa43bdd3d88f589ab13edae31e3eb6a7c3bb2d2846b"
Oct 30 19:02:19.573625 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:19.573605    4525 scope.go:122] "RemoveContainer" containerID="77b8f76751f2ecfb673475d5c389695275b75f72ce3bdefc8f6a1e30a6c6d871"
Oct 30 19:02:20.405159 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:20.405122    4525 patch_prober.go:29] interesting pod/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dhv28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Oct 30 19:02:20.405429 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:20.405403    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dhv28" podUID="bd3970a9-8e49-456f-b7c7-b6af7320dfab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Oct 30 19:02:21.401781 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:21.401746    4525 patch_prober.go:29] interesting pod/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dhv28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Oct 30 19:02:21.402259 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:21.402228    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dhv28" podUID="bd3970a9-8e49-456f-b7c7-b6af7320dfab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Oct 30 19:02:22.402360 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:22.402322    4525 patch_prober.go:29] interesting pod/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dhv28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Oct 30 19:02:22.402782 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:22.402379    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dhv28" podUID="bd3970a9-8e49-456f-b7c7-b6af7320dfab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Oct 30 19:02:23.644316 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:23.644284    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dhv28"
Oct 30 19:02:23.644822 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:23.644670    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dhv28"
Oct 30 19:02:23.645595 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:23.645574    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dhv28"
Oct 30 19:02:23.743173 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:23.743130    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a014d1-kbqjs"]
Oct 30 19:02:23.750454 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:23.750409    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a014d1-kbqjs"
Oct 30 19:02:23.758510 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:23.754260    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:02:23.758510 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:23.754294    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94hcn\""
Oct 30 19:02:23.758510 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:23.754332    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpj8t\" (UniqueName: \"kubernetes.io/projected/8cf2e598-14de-4b14-9956-bb2531575375-kube-api-access-xpj8t\") pod \"maas-api-revoke-keys-e2e-praxis-b-a014d1-kbqjs\" (UID: \"8cf2e598-14de-4b14-9956-bb2531575375\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a014d1-kbqjs"
Oct 30 19:02:23.758510 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:23.754404    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cf2e598-14de-4b14-9956-bb2531575375-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-a014d1-kbqjs\" (UID: \"8cf2e598-14de-4b14-9956-bb2531575375\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a014d1-kbqjs"
Oct 30 19:02:23.758510 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:23.754582    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cf2e598-14de-4b14-9956-bb2531575375-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-a014d1-kbqjs\" (UID: \"8cf2e598-14de-4b14-9956-bb2531575375\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a014d1-kbqjs"
Oct 30 19:02:23.758510 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:23.757767    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a014d1-kbqjs"]
Oct 30 19:02:23.855756 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:23.855704    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cf2e598-14de-4b14-9956-bb2531575375-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-a014d1-kbqjs\" (UID: \"8cf2e598-14de-4b14-9956-bb2531575375\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a014d1-kbqjs"
Oct 30 19:02:23.855942 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:23.855877    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpj8t\" (UniqueName: \"kubernetes.io/projected/8cf2e598-14de-4b14-9956-bb2531575375-kube-api-access-xpj8t\") pod \"maas-api-revoke-keys-e2e-praxis-b-a014d1-kbqjs\" (UID: \"8cf2e598-14de-4b14-9956-bb2531575375\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a014d1-kbqjs"
Oct 30 19:02:23.855990 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:23.855966    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cf2e598-14de-4b14-9956-bb2531575375-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-a014d1-kbqjs\" (UID: \"8cf2e598-14de-4b14-9956-bb2531575375\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a014d1-kbqjs"
Oct 30 19:02:23.856385 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:23.856366    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cf2e598-14de-4b14-9956-bb2531575375-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-a014d1-kbqjs\" (UID: \"8cf2e598-14de-4b14-9956-bb2531575375\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a014d1-kbqjs"
Oct 30 19:02:23.858556 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:23.858531    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cf2e598-14de-4b14-9956-bb2531575375-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-a014d1-kbqjs\" (UID: \"8cf2e598-14de-4b14-9956-bb2531575375\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a014d1-kbqjs"
Oct 30 19:02:23.866046 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:23.866025    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpj8t\" (UniqueName: \"kubernetes.io/projected/8cf2e598-14de-4b14-9956-bb2531575375-kube-api-access-xpj8t\") pod \"maas-api-revoke-keys-e2e-praxis-b-a014d1-kbqjs\" (UID: \"8cf2e598-14de-4b14-9956-bb2531575375\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a014d1-kbqjs"
Oct 30 19:02:24.072494 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:24.072410    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a014d1-kbqjs"
Oct 30 19:02:24.379060 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:02:24.379022    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cf2e598_14de_4b14_9956_bb2531575375.slice/crio-29baba06a3ad7b340dc594f7db1ed67b17ffd33eab31674a0a112913cfaa0678 WatchSource:0}: Error finding container 29baba06a3ad7b340dc594f7db1ed67b17ffd33eab31674a0a112913cfaa0678: Status 404 returned error can't find the container with id 29baba06a3ad7b340dc594f7db1ed67b17ffd33eab31674a0a112913cfaa0678
Oct 30 19:02:24.380465 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:24.380436    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a014d1-kbqjs"]
Oct 30 19:02:25.385257 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:25.385222    4525 generic.go:356] "Generic (PLEG): container finished" podID="8cf2e598-14de-4b14-9956-bb2531575375" containerID="1e6098c8b0278137a665f8f0773c7690ea43ff38814dedb95e00d956f5607b23" exitCode=0
Oct 30 19:02:25.385741 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:25.385303    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a014d1-kbqjs" event={"ID":"8cf2e598-14de-4b14-9956-bb2531575375","Type":"ContainerDied","Data":"1e6098c8b0278137a665f8f0773c7690ea43ff38814dedb95e00d956f5607b23"}
Oct 30 19:02:25.385741 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:25.385367    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a014d1-kbqjs" event={"ID":"8cf2e598-14de-4b14-9956-bb2531575375","Type":"ContainerStarted","Data":"29baba06a3ad7b340dc594f7db1ed67b17ffd33eab31674a0a112913cfaa0678"}
Oct 30 19:02:26.557010 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:26.556983    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a014d1-kbqjs"
Oct 30 19:02:26.600047 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:26.600016    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cf2e598-14de-4b14-9956-bb2531575375-kube-api-access-xpj8t\" (UniqueName: \"kubernetes.io/projected/8cf2e598-14de-4b14-9956-bb2531575375-kube-api-access-xpj8t\") pod \"8cf2e598-14de-4b14-9956-bb2531575375\" (UID: \"8cf2e598-14de-4b14-9956-bb2531575375\") "
Oct 30 19:02:26.600047 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:26.600052    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cf2e598-14de-4b14-9956-bb2531575375-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cf2e598-14de-4b14-9956-bb2531575375-maas-api-openshift-ca\") pod \"8cf2e598-14de-4b14-9956-bb2531575375\" (UID: \"8cf2e598-14de-4b14-9956-bb2531575375\") "
Oct 30 19:02:26.600303 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:26.600164    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8cf2e598-14de-4b14-9956-bb2531575375-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cf2e598-14de-4b14-9956-bb2531575375-maas-api-xks-ca\") pod \"8cf2e598-14de-4b14-9956-bb2531575375\" (UID: \"8cf2e598-14de-4b14-9956-bb2531575375\") "
Oct 30 19:02:26.600558 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:26.600529    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cf2e598-14de-4b14-9956-bb2531575375-maas-api-openshift-ca" pod "8cf2e598-14de-4b14-9956-bb2531575375" (UID: "8cf2e598-14de-4b14-9956-bb2531575375"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:02:26.601502 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:26.601281    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cf2e598-14de-4b14-9956-bb2531575375-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:02:26.602520 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:26.602497    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cf2e598-14de-4b14-9956-bb2531575375-maas-api-xks-ca" pod "8cf2e598-14de-4b14-9956-bb2531575375" (UID: "8cf2e598-14de-4b14-9956-bb2531575375"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:02:26.602636 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:26.602597    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cf2e598-14de-4b14-9956-bb2531575375-kube-api-access-xpj8t" pod "8cf2e598-14de-4b14-9956-bb2531575375" (UID: "8cf2e598-14de-4b14-9956-bb2531575375"). InnerVolumeSpecName "kube-api-access-xpj8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:02:26.703088 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:26.703004    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cf2e598-14de-4b14-9956-bb2531575375-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:02:26.703088 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:26.703031    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpj8t\" (UniqueName: \"kubernetes.io/projected/8cf2e598-14de-4b14-9956-bb2531575375-kube-api-access-xpj8t\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:02:27.418865 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:27.416822    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a014d1-kbqjs" event={"ID":"8cf2e598-14de-4b14-9956-bb2531575375","Type":"ContainerDied","Data":"29baba06a3ad7b340dc594f7db1ed67b17ffd33eab31674a0a112913cfaa0678"}
Oct 30 19:02:27.418865 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:27.416926    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29baba06a3ad7b340dc594f7db1ed67b17ffd33eab31674a0a112913cfaa0678"
Oct 30 19:02:27.418865 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:27.417029    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a014d1-kbqjs"
Oct 30 19:02:35.334763 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:35.331013    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a014d1-kbqjs"]
Oct 30 19:02:35.339888 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:35.339834    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a014d1-kbqjs"]
Oct 30 19:02:35.545565 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:35.545530    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cf2e598-14de-4b14-9956-bb2531575375" path="/var/lib/kubelet/pods/8cf2e598-14de-4b14-9956-bb2531575375/volumes"
Oct 30 19:02:36.442707 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:36.442659    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh"]
Oct 30 19:02:36.444143 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:36.444114    4525 cpu_manager.go:414] "containerMap: removing container" podUID="8cf2e598-14de-4b14-9956-bb2531575375" containerName="revoke-keys"
Oct 30 19:02:36.444314 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:36.444301    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cf2e598-14de-4b14-9956-bb2531575375" containerName="revoke-keys"
Oct 30 19:02:36.444568 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:36.444555    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cf2e598-14de-4b14-9956-bb2531575375" containerName="revoke-keys"
Oct 30 19:02:36.451252 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:36.451229    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh"
Oct 30 19:02:36.452775 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:36.452753    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh"]
Oct 30 19:02:36.456614 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:36.455923    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-2d1c71-openshift-default-dockercfg-vq6d2\""
Oct 30 19:02:36.868898 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:36.682541    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b37419e-12f0-4e1e-8b2a-48c39c4c9e99-istio-token\") pod \"e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh\" (UID: \"8b37419e-12f0-4e1e-8b2a-48c39c4c9e99\") " pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh"
Oct 30 19:02:36.868898 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:36.707364    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b37419e-12f0-4e1e-8b2a-48c39c4c9e99-istio-data\") pod \"e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh\" (UID: \"8b37419e-12f0-4e1e-8b2a-48c39c4c9e99\") " pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh"
Oct 30 19:02:36.868898 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:36.707450    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b37419e-12f0-4e1e-8b2a-48c39c4c9e99-credential-socket\") pod \"e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh\" (UID: \"8b37419e-12f0-4e1e-8b2a-48c39c4c9e99\") " pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh"
Oct 30 19:02:36.868898 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:36.707492    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b37419e-12f0-4e1e-8b2a-48c39c4c9e99-istio-podinfo\") pod \"e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh\" (UID: \"8b37419e-12f0-4e1e-8b2a-48c39c4c9e99\") " pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh"
Oct 30 19:02:36.868898 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:36.707601    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b37419e-12f0-4e1e-8b2a-48c39c4c9e99-workload-certs\") pod \"e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh\" (UID: \"8b37419e-12f0-4e1e-8b2a-48c39c4c9e99\") " pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh"
Oct 30 19:02:36.868898 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:36.707647    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b37419e-12f0-4e1e-8b2a-48c39c4c9e99-workload-socket\") pod \"e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh\" (UID: \"8b37419e-12f0-4e1e-8b2a-48c39c4c9e99\") " pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh"
Oct 30 19:02:36.868898 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:36.707720    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b37419e-12f0-4e1e-8b2a-48c39c4c9e99-istio-envoy\") pod \"e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh\" (UID: \"8b37419e-12f0-4e1e-8b2a-48c39c4c9e99\") " pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh"
Oct 30 19:02:36.868898 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:36.707783    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b37419e-12f0-4e1e-8b2a-48c39c4c9e99-istiod-ca-cert\") pod \"e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh\" (UID: \"8b37419e-12f0-4e1e-8b2a-48c39c4c9e99\") " pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh"
Oct 30 19:02:36.868898 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:36.707982    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjxbz\" (UniqueName: \"kubernetes.io/projected/8b37419e-12f0-4e1e-8b2a-48c39c4c9e99-kube-api-access-zjxbz\") pod \"e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh\" (UID: \"8b37419e-12f0-4e1e-8b2a-48c39c4c9e99\") " pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh"
Oct 30 19:02:36.868898 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:36.867903    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b37419e-12f0-4e1e-8b2a-48c39c4c9e99-credential-socket\") pod \"e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh\" (UID: \"8b37419e-12f0-4e1e-8b2a-48c39c4c9e99\") " pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh"
Oct 30 19:02:36.868898 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:36.867950    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b37419e-12f0-4e1e-8b2a-48c39c4c9e99-istio-podinfo\") pod \"e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh\" (UID: \"8b37419e-12f0-4e1e-8b2a-48c39c4c9e99\") " pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh"
Oct 30 19:02:36.868898 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:36.868099    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b37419e-12f0-4e1e-8b2a-48c39c4c9e99-workload-certs\") pod \"e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh\" (UID: \"8b37419e-12f0-4e1e-8b2a-48c39c4c9e99\") " pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh"
Oct 30 19:02:36.868898 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:36.868137    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b37419e-12f0-4e1e-8b2a-48c39c4c9e99-workload-socket\") pod \"e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh\" (UID: \"8b37419e-12f0-4e1e-8b2a-48c39c4c9e99\") " pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh"
Oct 30 19:02:36.868898 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:36.868179    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b37419e-12f0-4e1e-8b2a-48c39c4c9e99-istio-envoy\") pod \"e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh\" (UID: \"8b37419e-12f0-4e1e-8b2a-48c39c4c9e99\") " pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh"
Oct 30 19:02:36.868898 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:36.868298    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b37419e-12f0-4e1e-8b2a-48c39c4c9e99-istiod-ca-cert\") pod \"e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh\" (UID: \"8b37419e-12f0-4e1e-8b2a-48c39c4c9e99\") " pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh"
Oct 30 19:02:36.868898 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:36.868429    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjxbz\" (UniqueName: \"kubernetes.io/projected/8b37419e-12f0-4e1e-8b2a-48c39c4c9e99-kube-api-access-zjxbz\") pod \"e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh\" (UID: \"8b37419e-12f0-4e1e-8b2a-48c39c4c9e99\") " pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh"
Oct 30 19:02:36.868898 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:36.868540    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b37419e-12f0-4e1e-8b2a-48c39c4c9e99-istio-token\") pod \"e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh\" (UID: \"8b37419e-12f0-4e1e-8b2a-48c39c4c9e99\") " pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh"
Oct 30 19:02:36.868898 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:36.868653    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b37419e-12f0-4e1e-8b2a-48c39c4c9e99-istio-data\") pod \"e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh\" (UID: \"8b37419e-12f0-4e1e-8b2a-48c39c4c9e99\") " pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh"
Oct 30 19:02:36.873827 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:36.870570    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b37419e-12f0-4e1e-8b2a-48c39c4c9e99-istio-data\") pod \"e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh\" (UID: \"8b37419e-12f0-4e1e-8b2a-48c39c4c9e99\") " pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh"
Oct 30 19:02:36.915864 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:36.902591    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b37419e-12f0-4e1e-8b2a-48c39c4c9e99-workload-certs\") pod \"e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh\" (UID: \"8b37419e-12f0-4e1e-8b2a-48c39c4c9e99\") " pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh"
Oct 30 19:02:36.956944 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:36.913289    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b37419e-12f0-4e1e-8b2a-48c39c4c9e99-credential-socket\") pod \"e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh\" (UID: \"8b37419e-12f0-4e1e-8b2a-48c39c4c9e99\") " pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh"
Oct 30 19:02:36.961105 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:36.922452    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b37419e-12f0-4e1e-8b2a-48c39c4c9e99-istiod-ca-cert\") pod \"e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh\" (UID: \"8b37419e-12f0-4e1e-8b2a-48c39c4c9e99\") " pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh"
Oct 30 19:02:36.961105 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:36.947910    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b37419e-12f0-4e1e-8b2a-48c39c4c9e99-istio-podinfo\") pod \"e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh\" (UID: \"8b37419e-12f0-4e1e-8b2a-48c39c4c9e99\") " pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh"
Oct 30 19:02:36.962700 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:36.961229    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b37419e-12f0-4e1e-8b2a-48c39c4c9e99-workload-socket\") pod \"e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh\" (UID: \"8b37419e-12f0-4e1e-8b2a-48c39c4c9e99\") " pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh"
Oct 30 19:02:36.962700 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:36.962469    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b37419e-12f0-4e1e-8b2a-48c39c4c9e99-istio-envoy\") pod \"e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh\" (UID: \"8b37419e-12f0-4e1e-8b2a-48c39c4c9e99\") " pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh"
Oct 30 19:02:36.966984 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:36.966953    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b37419e-12f0-4e1e-8b2a-48c39c4c9e99-istio-token\") pod \"e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh\" (UID: \"8b37419e-12f0-4e1e-8b2a-48c39c4c9e99\") " pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh"
Oct 30 19:02:36.968696 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:36.968651    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjxbz\" (UniqueName: \"kubernetes.io/projected/8b37419e-12f0-4e1e-8b2a-48c39c4c9e99-kube-api-access-zjxbz\") pod \"e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh\" (UID: \"8b37419e-12f0-4e1e-8b2a-48c39c4c9e99\") " pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh"
Oct 30 19:02:37.038239 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:37.037950    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh"
Oct 30 19:02:37.452744 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:02:37.452711    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b37419e_12f0_4e1e_8b2a_48c39c4c9e99.slice/crio-d593c3d545e83f9a933c45411de311309704637838fd777abbae289788524f4a WatchSource:0}: Error finding container d593c3d545e83f9a933c45411de311309704637838fd777abbae289788524f4a: Status 404 returned error can't find the container with id d593c3d545e83f9a933c45411de311309704637838fd777abbae289788524f4a
Oct 30 19:02:37.454286 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:37.454262    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh"]
Oct 30 19:02:37.455404 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:37.455368    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:02:37.455512 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:37.455444    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:02:37.455512 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:37.455491    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:02:38.062799 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:38.062738    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh" event={"ID":"8b37419e-12f0-4e1e-8b2a-48c39c4c9e99","Type":"ContainerStarted","Data":"61529cb0c57fb81b6f1ff254514dbb0a5ae3aa1ce6910eacfc8b06288ce15574"}
Oct 30 19:02:38.062969 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:38.062819    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh" event={"ID":"8b37419e-12f0-4e1e-8b2a-48c39c4c9e99","Type":"ContainerStarted","Data":"d593c3d545e83f9a933c45411de311309704637838fd777abbae289788524f4a"}
Oct 30 19:02:38.092451 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:38.092386    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh" podStartSLOduration=2.09236636 podStartE2EDuration="2.09236636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:02:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:02:38.086647697 +0000 UTC m=+1580.712317661" watchObservedRunningTime="2026-10-30 19:02:38.09236636 +0000 UTC m=+1580.718036324"
Oct 30 19:02:39.039632 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:39.039547    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh"
Oct 30 19:02:39.045475 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:39.045445    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh"
Oct 30 19:02:39.072139 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:39.072109    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh"
Oct 30 19:02:39.073067 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:39.073048    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh"
Oct 30 19:02:43.207659 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:43.207617    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-a014d1-openshift-default-566cdcf765-4c97j"]
Oct 30 19:02:43.208189 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:43.207865    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-a014d1-openshift-default-566cdcf765-4c97j" podUID="116d5ed7-fbe2-4a19-8985-6fc29c87912d" containerName="istio-proxy" containerID="cri-o://9e8bddf78828caf065396ba20548508f9b8ad9dd188fe59b4febb72343c4b0de" gracePeriod=30
Oct 30 19:02:45.162880 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:45.162836    4525 patch_prober.go:29] interesting pod/e2e-praxis-b-a014d1-openshift-default-566cdcf765-4c97j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:02:45.646374 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:45.162918    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-a014d1-openshift-default-566cdcf765-4c97j" podUID="116d5ed7-fbe2-4a19-8985-6fc29c87912d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:02:45.646374 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:45.581089    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-68b466-openshift-default-57cb5457fc-jgkhq"]
Oct 30 19:02:45.646374 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:45.603156    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-68b466-openshift-default-57cb5457fc-jgkhq"
Oct 30 19:02:45.646374 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:45.603037    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-68b466-openshift-default-57cb5457fc-jgkhq"]
Oct 30 19:02:45.646374 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:45.637483    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-68b466-openshift-default-dockercfg-vmnl8\""
Oct 30 19:02:45.772714 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:45.772303    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19f00f47-605d-4b19-9a1c-664de3661507-istio-envoy\") pod \"e2e-shared-b-w0-68b466-openshift-default-57cb5457fc-jgkhq\" (UID: \"19f00f47-605d-4b19-9a1c-664de3661507\") " pod="openshift-ingress/e2e-shared-b-w0-68b466-openshift-default-57cb5457fc-jgkhq"
Oct 30 19:02:45.772714 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:45.772369    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19f00f47-605d-4b19-9a1c-664de3661507-istio-podinfo\") pod \"e2e-shared-b-w0-68b466-openshift-default-57cb5457fc-jgkhq\" (UID: \"19f00f47-605d-4b19-9a1c-664de3661507\") " pod="openshift-ingress/e2e-shared-b-w0-68b466-openshift-default-57cb5457fc-jgkhq"
Oct 30 19:02:45.772714 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:45.772451    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19f00f47-605d-4b19-9a1c-664de3661507-istiod-ca-cert\") pod \"e2e-shared-b-w0-68b466-openshift-default-57cb5457fc-jgkhq\" (UID: \"19f00f47-605d-4b19-9a1c-664de3661507\") " pod="openshift-ingress/e2e-shared-b-w0-68b466-openshift-default-57cb5457fc-jgkhq"
Oct 30 19:02:45.772714 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:45.772485    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19f00f47-605d-4b19-9a1c-664de3661507-credential-socket\") pod \"e2e-shared-b-w0-68b466-openshift-default-57cb5457fc-jgkhq\" (UID: \"19f00f47-605d-4b19-9a1c-664de3661507\") " pod="openshift-ingress/e2e-shared-b-w0-68b466-openshift-default-57cb5457fc-jgkhq"
Oct 30 19:02:45.772714 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:45.772553    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19f00f47-605d-4b19-9a1c-664de3661507-workload-certs\") pod \"e2e-shared-b-w0-68b466-openshift-default-57cb5457fc-jgkhq\" (UID: \"19f00f47-605d-4b19-9a1c-664de3661507\") " pod="openshift-ingress/e2e-shared-b-w0-68b466-openshift-default-57cb5457fc-jgkhq"
Oct 30 19:02:45.772714 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:45.772633    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19f00f47-605d-4b19-9a1c-664de3661507-istio-token\") pod \"e2e-shared-b-w0-68b466-openshift-default-57cb5457fc-jgkhq\" (UID: \"19f00f47-605d-4b19-9a1c-664de3661507\") " pod="openshift-ingress/e2e-shared-b-w0-68b466-openshift-default-57cb5457fc-jgkhq"
Oct 30 19:02:45.780729 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:45.773393    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19f00f47-605d-4b19-9a1c-664de3661507-workload-socket\") pod \"e2e-shared-b-w0-68b466-openshift-default-57cb5457fc-jgkhq\" (UID: \"19f00f47-605d-4b19-9a1c-664de3661507\") " pod="openshift-ingress/e2e-shared-b-w0-68b466-openshift-default-57cb5457fc-jgkhq"
Oct 30 19:02:45.780729 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:45.773490    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prt54\" (UniqueName: \"kubernetes.io/projected/19f00f47-605d-4b19-9a1c-664de3661507-kube-api-access-prt54\") pod \"e2e-shared-b-w0-68b466-openshift-default-57cb5457fc-jgkhq\" (UID: \"19f00f47-605d-4b19-9a1c-664de3661507\") " pod="openshift-ingress/e2e-shared-b-w0-68b466-openshift-default-57cb5457fc-jgkhq"
Oct 30 19:02:45.780729 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:45.773526    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19f00f47-605d-4b19-9a1c-664de3661507-istio-data\") pod \"e2e-shared-b-w0-68b466-openshift-default-57cb5457fc-jgkhq\" (UID: \"19f00f47-605d-4b19-9a1c-664de3661507\") " pod="openshift-ingress/e2e-shared-b-w0-68b466-openshift-default-57cb5457fc-jgkhq"
Oct 30 19:02:45.876232 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:45.874904    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19f00f47-605d-4b19-9a1c-664de3661507-istio-podinfo\") pod \"e2e-shared-b-w0-68b466-openshift-default-57cb5457fc-jgkhq\" (UID: \"19f00f47-605d-4b19-9a1c-664de3661507\") " pod="openshift-ingress/e2e-shared-b-w0-68b466-openshift-default-57cb5457fc-jgkhq"
Oct 30 19:02:45.876232 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:45.874993    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19f00f47-605d-4b19-9a1c-664de3661507-istiod-ca-cert\") pod \"e2e-shared-b-w0-68b466-openshift-default-57cb5457fc-jgkhq\" (UID: \"19f00f47-605d-4b19-9a1c-664de3661507\") " pod="openshift-ingress/e2e-shared-b-w0-68b466-openshift-default-57cb5457fc-jgkhq"
Oct 30 19:02:45.876232 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:45.875024    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19f00f47-605d-4b19-9a1c-664de3661507-credential-socket\") pod \"e2e-shared-b-w0-68b466-openshift-default-57cb5457fc-jgkhq\" (UID: \"19f00f47-605d-4b19-9a1c-664de3661507\") " pod="openshift-ingress/e2e-shared-b-w0-68b466-openshift-default-57cb5457fc-jgkhq"
Oct 30 19:02:45.876232 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:45.875138    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19f00f47-605d-4b19-9a1c-664de3661507-workload-certs\") pod \"e2e-shared-b-w0-68b466-openshift-default-57cb5457fc-jgkhq\" (UID: \"19f00f47-605d-4b19-9a1c-664de3661507\") " pod="openshift-ingress/e2e-shared-b-w0-68b466-openshift-default-57cb5457fc-jgkhq"
Oct 30 19:02:45.876232 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:45.875253    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19f00f47-605d-4b19-9a1c-664de3661507-istio-token\") pod \"e2e-shared-b-w0-68b466-openshift-default-57cb5457fc-jgkhq\" (UID: \"19f00f47-605d-4b19-9a1c-664de3661507\") " pod="openshift-ingress/e2e-shared-b-w0-68b466-openshift-default-57cb5457fc-jgkhq"
Oct 30 19:02:45.876232 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:45.875357    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19f00f47-605d-4b19-9a1c-664de3661507-workload-socket\") pod \"e2e-shared-b-w0-68b466-openshift-default-57cb5457fc-jgkhq\" (UID: \"19f00f47-605d-4b19-9a1c-664de3661507\") " pod="openshift-ingress/e2e-shared-b-w0-68b466-openshift-default-57cb5457fc-jgkhq"
Oct 30 19:02:45.876232 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:45.875411    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prt54\" (UniqueName: \"kubernetes.io/projected/19f00f47-605d-4b19-9a1c-664de3661507-kube-api-access-prt54\") pod \"e2e-shared-b-w0-68b466-openshift-default-57cb5457fc-jgkhq\" (UID: \"19f00f47-605d-4b19-9a1c-664de3661507\") " pod="openshift-ingress/e2e-shared-b-w0-68b466-openshift-default-57cb5457fc-jgkhq"
Oct 30 19:02:45.876232 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:45.875442    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19f00f47-605d-4b19-9a1c-664de3661507-istio-data\") pod \"e2e-shared-b-w0-68b466-openshift-default-57cb5457fc-jgkhq\" (UID: \"19f00f47-605d-4b19-9a1c-664de3661507\") " pod="openshift-ingress/e2e-shared-b-w0-68b466-openshift-default-57cb5457fc-jgkhq"
Oct 30 19:02:45.876232 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:45.875654    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19f00f47-605d-4b19-9a1c-664de3661507-istio-envoy\") pod \"e2e-shared-b-w0-68b466-openshift-default-57cb5457fc-jgkhq\" (UID: \"19f00f47-605d-4b19-9a1c-664de3661507\") " pod="openshift-ingress/e2e-shared-b-w0-68b466-openshift-default-57cb5457fc-jgkhq"
Oct 30 19:02:45.904510 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:45.877931    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19f00f47-605d-4b19-9a1c-664de3661507-workload-certs\") pod \"e2e-shared-b-w0-68b466-openshift-default-57cb5457fc-jgkhq\" (UID: \"19f00f47-605d-4b19-9a1c-664de3661507\") " pod="openshift-ingress/e2e-shared-b-w0-68b466-openshift-default-57cb5457fc-jgkhq"
Oct 30 19:02:45.904510 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:45.885824    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19f00f47-605d-4b19-9a1c-664de3661507-istio-podinfo\") pod \"e2e-shared-b-w0-68b466-openshift-default-57cb5457fc-jgkhq\" (UID: \"19f00f47-605d-4b19-9a1c-664de3661507\") " pod="openshift-ingress/e2e-shared-b-w0-68b466-openshift-default-57cb5457fc-jgkhq"
Oct 30 19:02:45.904510 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:45.886331    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19f00f47-605d-4b19-9a1c-664de3661507-credential-socket\") pod \"e2e-shared-b-w0-68b466-openshift-default-57cb5457fc-jgkhq\" (UID: \"19f00f47-605d-4b19-9a1c-664de3661507\") " pod="openshift-ingress/e2e-shared-b-w0-68b466-openshift-default-57cb5457fc-jgkhq"
Oct 30 19:02:45.904510 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:45.886618    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19f00f47-605d-4b19-9a1c-664de3661507-workload-socket\") pod \"e2e-shared-b-w0-68b466-openshift-default-57cb5457fc-jgkhq\" (UID: \"19f00f47-605d-4b19-9a1c-664de3661507\") " pod="openshift-ingress/e2e-shared-b-w0-68b466-openshift-default-57cb5457fc-jgkhq"
Oct 30 19:02:45.904510 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:45.886742    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19f00f47-605d-4b19-9a1c-664de3661507-istiod-ca-cert\") pod \"e2e-shared-b-w0-68b466-openshift-default-57cb5457fc-jgkhq\" (UID: \"19f00f47-605d-4b19-9a1c-664de3661507\") " pod="openshift-ingress/e2e-shared-b-w0-68b466-openshift-default-57cb5457fc-jgkhq"
Oct 30 19:02:45.904510 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:45.887201    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19f00f47-605d-4b19-9a1c-664de3661507-istio-data\") pod \"e2e-shared-b-w0-68b466-openshift-default-57cb5457fc-jgkhq\" (UID: \"19f00f47-605d-4b19-9a1c-664de3661507\") " pod="openshift-ingress/e2e-shared-b-w0-68b466-openshift-default-57cb5457fc-jgkhq"
Oct 30 19:02:45.904510 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:45.903748    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prt54\" (UniqueName: \"kubernetes.io/projected/19f00f47-605d-4b19-9a1c-664de3661507-kube-api-access-prt54\") pod \"e2e-shared-b-w0-68b466-openshift-default-57cb5457fc-jgkhq\" (UID: \"19f00f47-605d-4b19-9a1c-664de3661507\") " pod="openshift-ingress/e2e-shared-b-w0-68b466-openshift-default-57cb5457fc-jgkhq"
Oct 30 19:02:45.908415 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:45.908341    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19f00f47-605d-4b19-9a1c-664de3661507-istio-envoy\") pod \"e2e-shared-b-w0-68b466-openshift-default-57cb5457fc-jgkhq\" (UID: \"19f00f47-605d-4b19-9a1c-664de3661507\") " pod="openshift-ingress/e2e-shared-b-w0-68b466-openshift-default-57cb5457fc-jgkhq"
Oct 30 19:02:45.909254 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:45.909227    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19f00f47-605d-4b19-9a1c-664de3661507-istio-token\") pod \"e2e-shared-b-w0-68b466-openshift-default-57cb5457fc-jgkhq\" (UID: \"19f00f47-605d-4b19-9a1c-664de3661507\") " pod="openshift-ingress/e2e-shared-b-w0-68b466-openshift-default-57cb5457fc-jgkhq"
Oct 30 19:02:45.992182 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:45.992121    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-68b466-openshift-default-57cb5457fc-jgkhq"
Oct 30 19:02:46.218856 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:46.218766    4525 generic.go:356] "Generic (PLEG): container finished" podID="e14cca94-de16-4d65-b71a-9ece97424174" containerID="60d23ddc31d508eaa5eb17f954bf9625d80aeae99677e0445494225b7535eb60" exitCode=0
Oct 30 19:02:46.218856 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:46.218841    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-clvlk" event={"ID":"e14cca94-de16-4d65-b71a-9ece97424174","Type":"ContainerDied","Data":"60d23ddc31d508eaa5eb17f954bf9625d80aeae99677e0445494225b7535eb60"}
Oct 30 19:02:46.219449 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:46.219345    4525 scope.go:122] "RemoveContainer" containerID="60d23ddc31d508eaa5eb17f954bf9625d80aeae99677e0445494225b7535eb60"
Oct 30 19:02:46.221499 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:46.221398    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:02:46.221499 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:46.221480    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:02:46.221755 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:46.221515    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:02:46.319825 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:46.315628    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-68b466-openshift-default-57cb5457fc-jgkhq"]
Oct 30 19:02:46.319825 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:46.317234    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:02:46.319825 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:46.317319    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:02:46.319825 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:46.317367    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:02:47.238319 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:47.238278    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-68b466-openshift-default-57cb5457fc-jgkhq" event={"ID":"19f00f47-605d-4b19-9a1c-664de3661507","Type":"ContainerStarted","Data":"05032f3ae4130fe35addd2d7e6c00f8ff2d86f4825b9a8764db98d5829ad5434"}
Oct 30 19:02:47.239146 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:47.239106    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-68b466-openshift-default-57cb5457fc-jgkhq" event={"ID":"19f00f47-605d-4b19-9a1c-664de3661507","Type":"ContainerStarted","Data":"fb49545060e773d6c39767f3d47f21ef1a4543762041a99e0e17c37c098eec3c"}
Oct 30 19:02:47.241964 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:47.241226    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-clvlk" event={"ID":"e14cca94-de16-4d65-b71a-9ece97424174","Type":"ContainerStarted","Data":"51099112565d75045a8d9cb6d3d3c02280322816fd0cdf6169c2a059fadb714a"}
Oct 30 19:02:47.243583 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:47.243357    4525 generic.go:356] "Generic (PLEG): container finished" podID="116d5ed7-fbe2-4a19-8985-6fc29c87912d" containerID="9e8bddf78828caf065396ba20548508f9b8ad9dd188fe59b4febb72343c4b0de" exitCode=0
Oct 30 19:02:47.243583 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:47.243485    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-a014d1-openshift-default-566cdcf765-4c97j" event={"ID":"116d5ed7-fbe2-4a19-8985-6fc29c87912d","Type":"ContainerDied","Data":"9e8bddf78828caf065396ba20548508f9b8ad9dd188fe59b4febb72343c4b0de"}
Oct 30 19:02:47.263029 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:47.262974    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-68b466-openshift-default-57cb5457fc-jgkhq" podStartSLOduration=2.2629577 podStartE2EDuration="2.2629577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:02:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:02:47.26150875 +0000 UTC m=+1589.887178754" watchObservedRunningTime="2026-10-30 19:02:47.2629577 +0000 UTC m=+1589.888627722"
Oct 30 19:02:47.332222 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:47.332197    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-a014d1-openshift-default-566cdcf765-4c97j"
Oct 30 19:02:47.456898 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:47.456209    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/116d5ed7-fbe2-4a19-8985-6fc29c87912d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/116d5ed7-fbe2-4a19-8985-6fc29c87912d-istiod-ca-cert\") pod \"116d5ed7-fbe2-4a19-8985-6fc29c87912d\" (UID: \"116d5ed7-fbe2-4a19-8985-6fc29c87912d\") "
Oct 30 19:02:47.456898 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:47.456521    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/116d5ed7-fbe2-4a19-8985-6fc29c87912d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/116d5ed7-fbe2-4a19-8985-6fc29c87912d-istio-podinfo\") pod \"116d5ed7-fbe2-4a19-8985-6fc29c87912d\" (UID: \"116d5ed7-fbe2-4a19-8985-6fc29c87912d\") "
Oct 30 19:02:47.456898 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:47.456664    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/116d5ed7-fbe2-4a19-8985-6fc29c87912d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/116d5ed7-fbe2-4a19-8985-6fc29c87912d-istio-envoy\") pod \"116d5ed7-fbe2-4a19-8985-6fc29c87912d\" (UID: \"116d5ed7-fbe2-4a19-8985-6fc29c87912d\") "
Oct 30 19:02:47.457360 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:47.457323    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/116d5ed7-fbe2-4a19-8985-6fc29c87912d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/116d5ed7-fbe2-4a19-8985-6fc29c87912d-workload-socket\") pod \"116d5ed7-fbe2-4a19-8985-6fc29c87912d\" (UID: \"116d5ed7-fbe2-4a19-8985-6fc29c87912d\") "
Oct 30 19:02:47.457515 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:47.457376    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/116d5ed7-fbe2-4a19-8985-6fc29c87912d-istio-token\" (UniqueName: \"kubernetes.io/projected/116d5ed7-fbe2-4a19-8985-6fc29c87912d-istio-token\") pod \"116d5ed7-fbe2-4a19-8985-6fc29c87912d\" (UID: \"116d5ed7-fbe2-4a19-8985-6fc29c87912d\") "
Oct 30 19:02:47.457515 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:47.457412    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/116d5ed7-fbe2-4a19-8985-6fc29c87912d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/116d5ed7-fbe2-4a19-8985-6fc29c87912d-workload-certs\") pod \"116d5ed7-fbe2-4a19-8985-6fc29c87912d\" (UID: \"116d5ed7-fbe2-4a19-8985-6fc29c87912d\") "
Oct 30 19:02:47.457515 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:47.457488    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/116d5ed7-fbe2-4a19-8985-6fc29c87912d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/116d5ed7-fbe2-4a19-8985-6fc29c87912d-istio-data\") pod \"116d5ed7-fbe2-4a19-8985-6fc29c87912d\" (UID: \"116d5ed7-fbe2-4a19-8985-6fc29c87912d\") "
Oct 30 19:02:47.457734 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:47.457526    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/116d5ed7-fbe2-4a19-8985-6fc29c87912d-kube-api-access-nk45p\" (UniqueName: \"kubernetes.io/projected/116d5ed7-fbe2-4a19-8985-6fc29c87912d-kube-api-access-nk45p\") pod \"116d5ed7-fbe2-4a19-8985-6fc29c87912d\" (UID: \"116d5ed7-fbe2-4a19-8985-6fc29c87912d\") "
Oct 30 19:02:47.457734 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:47.457657    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/116d5ed7-fbe2-4a19-8985-6fc29c87912d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/116d5ed7-fbe2-4a19-8985-6fc29c87912d-credential-socket\") pod \"116d5ed7-fbe2-4a19-8985-6fc29c87912d\" (UID: \"116d5ed7-fbe2-4a19-8985-6fc29c87912d\") "
Oct 30 19:02:47.458501 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:47.458470    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/116d5ed7-fbe2-4a19-8985-6fc29c87912d-workload-socket" pod "116d5ed7-fbe2-4a19-8985-6fc29c87912d" (UID: "116d5ed7-fbe2-4a19-8985-6fc29c87912d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:02:47.458618 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:47.458479    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/116d5ed7-fbe2-4a19-8985-6fc29c87912d-workload-certs" pod "116d5ed7-fbe2-4a19-8985-6fc29c87912d" (UID: "116d5ed7-fbe2-4a19-8985-6fc29c87912d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:02:47.458993 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:47.458834    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/116d5ed7-fbe2-4a19-8985-6fc29c87912d-credential-socket" pod "116d5ed7-fbe2-4a19-8985-6fc29c87912d" (UID: "116d5ed7-fbe2-4a19-8985-6fc29c87912d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:02:47.459088 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:47.459052    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/116d5ed7-fbe2-4a19-8985-6fc29c87912d-istio-data" pod "116d5ed7-fbe2-4a19-8985-6fc29c87912d" (UID: "116d5ed7-fbe2-4a19-8985-6fc29c87912d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:02:47.459160 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:47.459085    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/116d5ed7-fbe2-4a19-8985-6fc29c87912d-istiod-ca-cert" pod "116d5ed7-fbe2-4a19-8985-6fc29c87912d" (UID: "116d5ed7-fbe2-4a19-8985-6fc29c87912d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:02:47.475931 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:47.466866    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/116d5ed7-fbe2-4a19-8985-6fc29c87912d-istio-podinfo" pod "116d5ed7-fbe2-4a19-8985-6fc29c87912d" (UID: "116d5ed7-fbe2-4a19-8985-6fc29c87912d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:02:47.475931 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:47.466972    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/116d5ed7-fbe2-4a19-8985-6fc29c87912d-istio-envoy" pod "116d5ed7-fbe2-4a19-8985-6fc29c87912d" (UID: "116d5ed7-fbe2-4a19-8985-6fc29c87912d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:02:47.475931 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:47.468642    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/116d5ed7-fbe2-4a19-8985-6fc29c87912d-kube-api-access-nk45p" pod "116d5ed7-fbe2-4a19-8985-6fc29c87912d" (UID: "116d5ed7-fbe2-4a19-8985-6fc29c87912d"). InnerVolumeSpecName "kube-api-access-nk45p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:02:47.475931 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:47.475834    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/116d5ed7-fbe2-4a19-8985-6fc29c87912d-istio-token" pod "116d5ed7-fbe2-4a19-8985-6fc29c87912d" (UID: "116d5ed7-fbe2-4a19-8985-6fc29c87912d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:02:47.515885 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:02:47.515640    4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d49l" containerName="extractor"
Oct 30 19:02:47.559449 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:47.559412    4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/116d5ed7-fbe2-4a19-8985-6fc29c87912d-istiod-ca-cert\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:02:47.559449 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:47.559447    4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/116d5ed7-fbe2-4a19-8985-6fc29c87912d-istio-podinfo\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:02:47.559853 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:47.559463    4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/116d5ed7-fbe2-4a19-8985-6fc29c87912d-istio-envoy\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:02:47.559853 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:47.559475    4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/116d5ed7-fbe2-4a19-8985-6fc29c87912d-workload-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:02:47.559853 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:47.559486    4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/116d5ed7-fbe2-4a19-8985-6fc29c87912d-istio-token\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:02:47.559853 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:47.559498    4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/116d5ed7-fbe2-4a19-8985-6fc29c87912d-workload-certs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:02:47.559853 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:47.559509    4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/116d5ed7-fbe2-4a19-8985-6fc29c87912d-istio-data\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:02:47.559853 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:47.559523    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nk45p\" (UniqueName: \"kubernetes.io/projected/116d5ed7-fbe2-4a19-8985-6fc29c87912d-kube-api-access-nk45p\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:02:47.559853 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:47.559537    4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/116d5ed7-fbe2-4a19-8985-6fc29c87912d-credential-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:02:48.001080 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:48.001045    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-68b466-openshift-default-57cb5457fc-jgkhq"
Oct 30 19:02:48.139725 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:48.137750    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-clvlk"
Oct 30 19:02:48.139725 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:48.137878    4525 patch_prober.go:29] interesting pod/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-clvlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Oct 30 19:02:48.139725 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:48.137922    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-clvlk" podUID="e14cca94-de16-4d65-b71a-9ece97424174" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Oct 30 19:02:48.172310 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:48.172097    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-68b466-openshift-default-57cb5457fc-jgkhq"
Oct 30 19:02:48.257215 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:48.257117    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-a014d1-openshift-default-566cdcf765-4c97j"
Oct 30 19:02:48.257215 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:48.257128    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-a014d1-openshift-default-566cdcf765-4c97j" event={"ID":"116d5ed7-fbe2-4a19-8985-6fc29c87912d","Type":"ContainerDied","Data":"7dfce15f057502e7afa9da0019783a0ec5674c82ff0696a1f20d4fa91d153e45"}
Oct 30 19:02:48.257215 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:48.257201    4525 scope.go:122] "RemoveContainer" containerID="9e8bddf78828caf065396ba20548508f9b8ad9dd188fe59b4febb72343c4b0de"
Oct 30 19:02:48.257838 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:48.257744    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-68b466-openshift-default-57cb5457fc-jgkhq"
Oct 30 19:02:48.259761 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:48.259360    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-68b466-openshift-default-57cb5457fc-jgkhq"
Oct 30 19:02:48.331279 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:48.331125    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-a014d1-openshift-default-566cdcf765-4c97j"]
Oct 30 19:02:48.337406 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:48.337379    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-a014d1-openshift-default-566cdcf765-4c97j"]
Oct 30 19:02:48.923505 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:48.923475    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-clvlk"
Oct 30 19:02:49.135323 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:49.135285    4525 patch_prober.go:29] interesting pod/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-clvlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Oct 30 19:02:49.135502 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:49.135344    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-clvlk" podUID="e14cca94-de16-4d65-b71a-9ece97424174" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Oct 30 19:02:49.545277 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:49.545246    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="116d5ed7-fbe2-4a19-8985-6fc29c87912d" path="/var/lib/kubelet/pods/116d5ed7-fbe2-4a19-8985-6fc29c87912d/volumes"
Oct 30 19:02:50.135951 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:50.135914    4525 patch_prober.go:29] interesting pod/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-clvlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Oct 30 19:02:50.136136 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:50.135968    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-clvlk" podUID="e14cca94-de16-4d65-b71a-9ece97424174" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Oct 30 19:02:51.144720 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:51.143202    4525 patch_prober.go:29] interesting pod/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-clvlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Oct 30 19:02:51.144720 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:51.143281    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-clvlk" podUID="e14cca94-de16-4d65-b71a-9ece97424174" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Oct 30 19:02:52.135544 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:52.135498    4525 patch_prober.go:29] interesting pod/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-clvlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Oct 30 19:02:52.135782 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:52.135573    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-clvlk" podUID="e14cca94-de16-4d65-b71a-9ece97424174" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Oct 30 19:02:52.911102 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:52.911065    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-ef6b56/auth-test-model-ef6b56-kserve-66c7fd94b9-qgqj4"]
Oct 30 19:02:52.911956 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:52.911933    4525 cpu_manager.go:414] "containerMap: removing container" podUID="116d5ed7-fbe2-4a19-8985-6fc29c87912d" containerName="istio-proxy"
Oct 30 19:02:52.911956 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:52.911955    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="116d5ed7-fbe2-4a19-8985-6fc29c87912d" containerName="istio-proxy"
Oct 30 19:02:52.912147 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:52.912059    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="116d5ed7-fbe2-4a19-8985-6fc29c87912d" containerName="istio-proxy"
Oct 30 19:02:52.917477 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:52.917453    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-ef6b56/auth-test-model-ef6b56-kserve-66c7fd94b9-qgqj4"]
Oct 30 19:02:52.917625 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:52.917544    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-ef6b56/auth-test-model-ef6b56-kserve-66c7fd94b9-qgqj4"
Oct 30 19:02:52.921551 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:52.921521    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-ef6b56\"/\"kube-root-ca.crt\""
Oct 30 19:02:52.922555 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:52.922524    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-ef6b56\"/\"openshift-service-ca.crt\""
Oct 30 19:02:52.922795 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:52.922759    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-ef6b56\"/\"default-dockercfg-k65hn\""
Oct 30 19:02:52.922903 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:52.922886    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-ef6b56\"/\"auth-test-model-ef6b56-kserve-self-signed-certs\""
Oct 30 19:02:53.049765 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:53.049724    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec0f645b-289a-4b05-a984-f0f66e896357-model-cache\") pod \"auth-test-model-ef6b56-kserve-66c7fd94b9-qgqj4\" (UID: \"ec0f645b-289a-4b05-a984-f0f66e896357\") " pod="ai-tenant-e2e-shared-a-w4-ef6b56/auth-test-model-ef6b56-kserve-66c7fd94b9-qgqj4"
Oct 30 19:02:53.049968 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:53.049898    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mn25j\" (UniqueName: \"kubernetes.io/projected/ec0f645b-289a-4b05-a984-f0f66e896357-kube-api-access-mn25j\") pod \"auth-test-model-ef6b56-kserve-66c7fd94b9-qgqj4\" (UID: \"ec0f645b-289a-4b05-a984-f0f66e896357\") " pod="ai-tenant-e2e-shared-a-w4-ef6b56/auth-test-model-ef6b56-kserve-66c7fd94b9-qgqj4"
Oct 30 19:02:53.050103 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:53.050082    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec0f645b-289a-4b05-a984-f0f66e896357-dshm\") pod \"auth-test-model-ef6b56-kserve-66c7fd94b9-qgqj4\" (UID: \"ec0f645b-289a-4b05-a984-f0f66e896357\") " pod="ai-tenant-e2e-shared-a-w4-ef6b56/auth-test-model-ef6b56-kserve-66c7fd94b9-qgqj4"
Oct 30 19:02:53.050175 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:53.050140    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec0f645b-289a-4b05-a984-f0f66e896357-home\") pod \"auth-test-model-ef6b56-kserve-66c7fd94b9-qgqj4\" (UID: \"ec0f645b-289a-4b05-a984-f0f66e896357\") " pod="ai-tenant-e2e-shared-a-w4-ef6b56/auth-test-model-ef6b56-kserve-66c7fd94b9-qgqj4"
Oct 30 19:02:53.050243 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:53.050201    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec0f645b-289a-4b05-a984-f0f66e896357-tls-certs\") pod \"auth-test-model-ef6b56-kserve-66c7fd94b9-qgqj4\" (UID: \"ec0f645b-289a-4b05-a984-f0f66e896357\") " pod="ai-tenant-e2e-shared-a-w4-ef6b56/auth-test-model-ef6b56-kserve-66c7fd94b9-qgqj4"
Oct 30 19:02:53.050300 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:53.050251    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec0f645b-289a-4b05-a984-f0f66e896357-tmp-dir\") pod \"auth-test-model-ef6b56-kserve-66c7fd94b9-qgqj4\" (UID: \"ec0f645b-289a-4b05-a984-f0f66e896357\") " pod="ai-tenant-e2e-shared-a-w4-ef6b56/auth-test-model-ef6b56-kserve-66c7fd94b9-qgqj4"
Oct 30 19:02:53.135608 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:53.135566    4525 patch_prober.go:29] interesting pod/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-clvlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Oct 30 19:02:53.135834 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:53.135632    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-clvlk" podUID="e14cca94-de16-4d65-b71a-9ece97424174" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Oct 30 19:02:53.162757 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:53.155469    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec0f645b-289a-4b05-a984-f0f66e896357-model-cache\") pod \"auth-test-model-ef6b56-kserve-66c7fd94b9-qgqj4\" (UID: \"ec0f645b-289a-4b05-a984-f0f66e896357\") " pod="ai-tenant-e2e-shared-a-w4-ef6b56/auth-test-model-ef6b56-kserve-66c7fd94b9-qgqj4"
Oct 30 19:02:53.162757 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:53.155616    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mn25j\" (UniqueName: \"kubernetes.io/projected/ec0f645b-289a-4b05-a984-f0f66e896357-kube-api-access-mn25j\") pod \"auth-test-model-ef6b56-kserve-66c7fd94b9-qgqj4\" (UID: \"ec0f645b-289a-4b05-a984-f0f66e896357\") " pod="ai-tenant-e2e-shared-a-w4-ef6b56/auth-test-model-ef6b56-kserve-66c7fd94b9-qgqj4"
Oct 30 19:02:53.162757 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:53.155783    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec0f645b-289a-4b05-a984-f0f66e896357-dshm\") pod \"auth-test-model-ef6b56-kserve-66c7fd94b9-qgqj4\" (UID: \"ec0f645b-289a-4b05-a984-f0f66e896357\") " pod="ai-tenant-e2e-shared-a-w4-ef6b56/auth-test-model-ef6b56-kserve-66c7fd94b9-qgqj4"
Oct 30 19:02:53.162757 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:53.155829    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec0f645b-289a-4b05-a984-f0f66e896357-home\") pod \"auth-test-model-ef6b56-kserve-66c7fd94b9-qgqj4\" (UID: \"ec0f645b-289a-4b05-a984-f0f66e896357\") " pod="ai-tenant-e2e-shared-a-w4-ef6b56/auth-test-model-ef6b56-kserve-66c7fd94b9-qgqj4"
Oct 30 19:02:53.162757 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:53.155858    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec0f645b-289a-4b05-a984-f0f66e896357-tls-certs\") pod \"auth-test-model-ef6b56-kserve-66c7fd94b9-qgqj4\" (UID: \"ec0f645b-289a-4b05-a984-f0f66e896357\") " pod="ai-tenant-e2e-shared-a-w4-ef6b56/auth-test-model-ef6b56-kserve-66c7fd94b9-qgqj4"
Oct 30 19:02:53.162757 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:53.155889    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec0f645b-289a-4b05-a984-f0f66e896357-tmp-dir\") pod \"auth-test-model-ef6b56-kserve-66c7fd94b9-qgqj4\" (UID: \"ec0f645b-289a-4b05-a984-f0f66e896357\") " pod="ai-tenant-e2e-shared-a-w4-ef6b56/auth-test-model-ef6b56-kserve-66c7fd94b9-qgqj4"
Oct 30 19:02:53.162757 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:53.156420    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec0f645b-289a-4b05-a984-f0f66e896357-tmp-dir\") pod \"auth-test-model-ef6b56-kserve-66c7fd94b9-qgqj4\" (UID: \"ec0f645b-289a-4b05-a984-f0f66e896357\") " pod="ai-tenant-e2e-shared-a-w4-ef6b56/auth-test-model-ef6b56-kserve-66c7fd94b9-qgqj4"
Oct 30 19:02:53.162757 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:53.156717    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec0f645b-289a-4b05-a984-f0f66e896357-model-cache\") pod \"auth-test-model-ef6b56-kserve-66c7fd94b9-qgqj4\" (UID: \"ec0f645b-289a-4b05-a984-f0f66e896357\") " pod="ai-tenant-e2e-shared-a-w4-ef6b56/auth-test-model-ef6b56-kserve-66c7fd94b9-qgqj4"
Oct 30 19:02:53.162757 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:53.157545    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec0f645b-289a-4b05-a984-f0f66e896357-home\") pod \"auth-test-model-ef6b56-kserve-66c7fd94b9-qgqj4\" (UID: \"ec0f645b-289a-4b05-a984-f0f66e896357\") " pod="ai-tenant-e2e-shared-a-w4-ef6b56/auth-test-model-ef6b56-kserve-66c7fd94b9-qgqj4"
Oct 30 19:02:53.194705 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:53.191383    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mn25j\" (UniqueName: \"kubernetes.io/projected/ec0f645b-289a-4b05-a984-f0f66e896357-kube-api-access-mn25j\") pod \"auth-test-model-ef6b56-kserve-66c7fd94b9-qgqj4\" (UID: \"ec0f645b-289a-4b05-a984-f0f66e896357\") " pod="ai-tenant-e2e-shared-a-w4-ef6b56/auth-test-model-ef6b56-kserve-66c7fd94b9-qgqj4"
Oct 30 19:02:53.194705 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:53.192992    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec0f645b-289a-4b05-a984-f0f66e896357-dshm\") pod \"auth-test-model-ef6b56-kserve-66c7fd94b9-qgqj4\" (UID: \"ec0f645b-289a-4b05-a984-f0f66e896357\") " pod="ai-tenant-e2e-shared-a-w4-ef6b56/auth-test-model-ef6b56-kserve-66c7fd94b9-qgqj4"
Oct 30 19:02:53.218706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:53.218574    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec0f645b-289a-4b05-a984-f0f66e896357-tls-certs\") pod \"auth-test-model-ef6b56-kserve-66c7fd94b9-qgqj4\" (UID: \"ec0f645b-289a-4b05-a984-f0f66e896357\") " pod="ai-tenant-e2e-shared-a-w4-ef6b56/auth-test-model-ef6b56-kserve-66c7fd94b9-qgqj4"
Oct 30 19:02:53.242746 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:53.239986    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-ef6b56/auth-test-model-ef6b56-kserve-66c7fd94b9-qgqj4"
Oct 30 19:02:53.615133 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:53.615103    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-ef6b56/auth-test-model-ef6b56-kserve-66c7fd94b9-qgqj4"]
Oct 30 19:02:53.615447 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:02:53.615410    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podec0f645b_289a_4b05_a984_f0f66e896357.slice/crio-779ce506fd37208bc6176de96803e4b686620d8f63a84b09c6852b84a3c6f17b WatchSource:0}: Error finding container 779ce506fd37208bc6176de96803e4b686620d8f63a84b09c6852b84a3c6f17b: Status 404 returned error can't find the container with id 779ce506fd37208bc6176de96803e4b686620d8f63a84b09c6852b84a3c6f17b
Oct 30 19:02:53.690062 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:53.690029    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9b222d4-cshq9"]
Oct 30 19:02:53.695486 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:53.695455    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9b222d4-cshq9"
Oct 30 19:02:53.701628 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:53.700545    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:02:53.701628 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:53.701169    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94hcn\""
Oct 30 19:02:53.701843 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:53.701729    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9b222d4-cshq9"]
Oct 30 19:02:53.775669 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:53.775626    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaf6f1db-cc65-4e6b-ac33-b4848b527f8a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9b222d4-cshq9\" (UID: \"aaf6f1db-cc65-4e6b-ac33-b4848b527f8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9b222d4-cshq9"
Oct 30 19:02:53.775968 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:53.775675    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pnn4\" (UniqueName: \"kubernetes.io/projected/aaf6f1db-cc65-4e6b-ac33-b4848b527f8a-kube-api-access-9pnn4\") pod \"maas-api-revoke-sub-a9b222d4-cshq9\" (UID: \"aaf6f1db-cc65-4e6b-ac33-b4848b527f8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9b222d4-cshq9"
Oct 30 19:02:53.775968 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:53.775950    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaf6f1db-cc65-4e6b-ac33-b4848b527f8a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9b222d4-cshq9\" (UID: \"aaf6f1db-cc65-4e6b-ac33-b4848b527f8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9b222d4-cshq9"
Oct 30 19:02:53.877349 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:53.877213    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaf6f1db-cc65-4e6b-ac33-b4848b527f8a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9b222d4-cshq9\" (UID: \"aaf6f1db-cc65-4e6b-ac33-b4848b527f8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9b222d4-cshq9"
Oct 30 19:02:53.877349 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:53.877284    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pnn4\" (UniqueName: \"kubernetes.io/projected/aaf6f1db-cc65-4e6b-ac33-b4848b527f8a-kube-api-access-9pnn4\") pod \"maas-api-revoke-sub-a9b222d4-cshq9\" (UID: \"aaf6f1db-cc65-4e6b-ac33-b4848b527f8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9b222d4-cshq9"
Oct 30 19:02:53.877661 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:53.877540    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaf6f1db-cc65-4e6b-ac33-b4848b527f8a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9b222d4-cshq9\" (UID: \"aaf6f1db-cc65-4e6b-ac33-b4848b527f8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9b222d4-cshq9"
Oct 30 19:02:53.878226 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:53.878204    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaf6f1db-cc65-4e6b-ac33-b4848b527f8a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9b222d4-cshq9\" (UID: \"aaf6f1db-cc65-4e6b-ac33-b4848b527f8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9b222d4-cshq9"
Oct 30 19:02:53.880108 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:53.880089    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaf6f1db-cc65-4e6b-ac33-b4848b527f8a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9b222d4-cshq9\" (UID: \"aaf6f1db-cc65-4e6b-ac33-b4848b527f8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9b222d4-cshq9"
Oct 30 19:02:53.900089 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:53.900015    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pnn4\" (UniqueName: \"kubernetes.io/projected/aaf6f1db-cc65-4e6b-ac33-b4848b527f8a-kube-api-access-9pnn4\") pod \"maas-api-revoke-sub-a9b222d4-cshq9\" (UID: \"aaf6f1db-cc65-4e6b-ac33-b4848b527f8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9b222d4-cshq9"
Oct 30 19:02:54.032492 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:54.032459    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9b222d4-cshq9"
Oct 30 19:02:54.139660 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:54.139580    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-clvlk"
Oct 30 19:02:54.140954 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:54.140929    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-clvlk"
Oct 30 19:02:54.317179 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:02:54.317147    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaf6f1db_cc65_4e6b_ac33_b4848b527f8a.slice/crio-54e5522d8f27f70db48dbd3c448b3dd0227266477b633c5b33ab5b2b868cd42f WatchSource:0}: Error finding container 54e5522d8f27f70db48dbd3c448b3dd0227266477b633c5b33ab5b2b868cd42f: Status 404 returned error can't find the container with id 54e5522d8f27f70db48dbd3c448b3dd0227266477b633c5b33ab5b2b868cd42f
Oct 30 19:02:54.318566 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:54.318540    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9b222d4-cshq9"]
Oct 30 19:02:54.355512 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:54.355470    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-ef6b56/auth-test-model-ef6b56-kserve-66c7fd94b9-qgqj4" event={"ID":"ec0f645b-289a-4b05-a984-f0f66e896357","Type":"ContainerStarted","Data":"a10bda61492f43d5ea9ee8f82b401bbf4fa5082b8e78a3ecbd39be5aea40410f"}
Oct 30 19:02:54.355694 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:54.355525    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-ef6b56/auth-test-model-ef6b56-kserve-66c7fd94b9-qgqj4" event={"ID":"ec0f645b-289a-4b05-a984-f0f66e896357","Type":"ContainerStarted","Data":"779ce506fd37208bc6176de96803e4b686620d8f63a84b09c6852b84a3c6f17b"}
Oct 30 19:02:54.357175 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:54.357150    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9b222d4-cshq9" event={"ID":"aaf6f1db-cc65-4e6b-ac33-b4848b527f8a","Type":"ContainerStarted","Data":"54e5522d8f27f70db48dbd3c448b3dd0227266477b633c5b33ab5b2b868cd42f"}
Oct 30 19:02:54.383866 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:54.383198    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-a1d55d/auth-test-model-a1d55d-kserve-7c4f64fd9-njmlg"]
Oct 30 19:02:54.389526 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:54.389499    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-a1d55d/auth-test-model-a1d55d-kserve-7c4f64fd9-njmlg"
Oct 30 19:02:54.390969 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:54.390880    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-ef6b56/auth-test-model-ef6b56-kserve-66c7fd94b9-qgqj4" podStartSLOduration=2.390863071 podStartE2EDuration="2.390863071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:02:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:02:54.37452631 +0000 UTC m=+1597.000196273" watchObservedRunningTime="2026-10-30 19:02:54.390863071 +0000 UTC m=+1597.016533036"
Oct 30 19:02:54.393694 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:54.393648    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-a1d55d\"/\"kube-root-ca.crt\""
Oct 30 19:02:54.393832 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:54.393812    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-a1d55d\"/\"openshift-service-ca.crt\""
Oct 30 19:02:54.393987 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:54.393968    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-a1d55d\"/\"default-dockercfg-kjmfc\""
Oct 30 19:02:54.394558 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:54.394536    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-a1d55d\"/\"auth-test-model-a1d55d-kserve-self-signed-certs\""
Oct 30 19:02:54.406339 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:54.403533    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-a1d55d/auth-test-model-a1d55d-kserve-7c4f64fd9-njmlg"]
Oct 30 19:02:54.486698 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:54.486642    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72b01806-d7cc-4452-896b-32d64e92af97-home\") pod \"auth-test-model-a1d55d-kserve-7c4f64fd9-njmlg\" (UID: \"72b01806-d7cc-4452-896b-32d64e92af97\") " pod="ai-tenant-e2e-shared-a-w2-a1d55d/auth-test-model-a1d55d-kserve-7c4f64fd9-njmlg"
Oct 30 19:02:54.486902 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:54.486756    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72b01806-d7cc-4452-896b-32d64e92af97-tmp-dir\") pod \"auth-test-model-a1d55d-kserve-7c4f64fd9-njmlg\" (UID: \"72b01806-d7cc-4452-896b-32d64e92af97\") " pod="ai-tenant-e2e-shared-a-w2-a1d55d/auth-test-model-a1d55d-kserve-7c4f64fd9-njmlg"
Oct 30 19:02:54.486902 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:54.486792    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72b01806-d7cc-4452-896b-32d64e92af97-model-cache\") pod \"auth-test-model-a1d55d-kserve-7c4f64fd9-njmlg\" (UID: \"72b01806-d7cc-4452-896b-32d64e92af97\") " pod="ai-tenant-e2e-shared-a-w2-a1d55d/auth-test-model-a1d55d-kserve-7c4f64fd9-njmlg"
Oct 30 19:02:54.487036 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:54.486942    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2mz7\" (UniqueName: \"kubernetes.io/projected/72b01806-d7cc-4452-896b-32d64e92af97-kube-api-access-q2mz7\") pod \"auth-test-model-a1d55d-kserve-7c4f64fd9-njmlg\" (UID: \"72b01806-d7cc-4452-896b-32d64e92af97\") " pod="ai-tenant-e2e-shared-a-w2-a1d55d/auth-test-model-a1d55d-kserve-7c4f64fd9-njmlg"
Oct 30 19:02:54.487036 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:54.486995    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72b01806-d7cc-4452-896b-32d64e92af97-tls-certs\") pod \"auth-test-model-a1d55d-kserve-7c4f64fd9-njmlg\" (UID: \"72b01806-d7cc-4452-896b-32d64e92af97\") " pod="ai-tenant-e2e-shared-a-w2-a1d55d/auth-test-model-a1d55d-kserve-7c4f64fd9-njmlg"
Oct 30 19:02:54.487260 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:54.487235    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72b01806-d7cc-4452-896b-32d64e92af97-dshm\") pod \"auth-test-model-a1d55d-kserve-7c4f64fd9-njmlg\" (UID: \"72b01806-d7cc-4452-896b-32d64e92af97\") " pod="ai-tenant-e2e-shared-a-w2-a1d55d/auth-test-model-a1d55d-kserve-7c4f64fd9-njmlg"
Oct 30 19:02:54.591924 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:54.591881    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2mz7\" (UniqueName: \"kubernetes.io/projected/72b01806-d7cc-4452-896b-32d64e92af97-kube-api-access-q2mz7\") pod \"auth-test-model-a1d55d-kserve-7c4f64fd9-njmlg\" (UID: \"72b01806-d7cc-4452-896b-32d64e92af97\") " pod="ai-tenant-e2e-shared-a-w2-a1d55d/auth-test-model-a1d55d-kserve-7c4f64fd9-njmlg"
Oct 30 19:02:54.592598 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:54.592569    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72b01806-d7cc-4452-896b-32d64e92af97-tls-certs\") pod \"auth-test-model-a1d55d-kserve-7c4f64fd9-njmlg\" (UID: \"72b01806-d7cc-4452-896b-32d64e92af97\") " pod="ai-tenant-e2e-shared-a-w2-a1d55d/auth-test-model-a1d55d-kserve-7c4f64fd9-njmlg"
Oct 30 19:02:54.593513 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:54.593491    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72b01806-d7cc-4452-896b-32d64e92af97-dshm\") pod \"auth-test-model-a1d55d-kserve-7c4f64fd9-njmlg\" (UID: \"72b01806-d7cc-4452-896b-32d64e92af97\") " pod="ai-tenant-e2e-shared-a-w2-a1d55d/auth-test-model-a1d55d-kserve-7c4f64fd9-njmlg"
Oct 30 19:02:54.594084 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:54.594066    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72b01806-d7cc-4452-896b-32d64e92af97-home\") pod \"auth-test-model-a1d55d-kserve-7c4f64fd9-njmlg\" (UID: \"72b01806-d7cc-4452-896b-32d64e92af97\") " pod="ai-tenant-e2e-shared-a-w2-a1d55d/auth-test-model-a1d55d-kserve-7c4f64fd9-njmlg"
Oct 30 19:02:54.597136 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:54.597106    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72b01806-d7cc-4452-896b-32d64e92af97-tmp-dir\") pod \"auth-test-model-a1d55d-kserve-7c4f64fd9-njmlg\" (UID: \"72b01806-d7cc-4452-896b-32d64e92af97\") " pod="ai-tenant-e2e-shared-a-w2-a1d55d/auth-test-model-a1d55d-kserve-7c4f64fd9-njmlg"
Oct 30 19:02:54.604463 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:54.604440    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72b01806-d7cc-4452-896b-32d64e92af97-model-cache\") pod \"auth-test-model-a1d55d-kserve-7c4f64fd9-njmlg\" (UID: \"72b01806-d7cc-4452-896b-32d64e92af97\") " pod="ai-tenant-e2e-shared-a-w2-a1d55d/auth-test-model-a1d55d-kserve-7c4f64fd9-njmlg"
Oct 30 19:02:54.604816 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:54.599192    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72b01806-d7cc-4452-896b-32d64e92af97-tmp-dir\") pod \"auth-test-model-a1d55d-kserve-7c4f64fd9-njmlg\" (UID: \"72b01806-d7cc-4452-896b-32d64e92af97\") " pod="ai-tenant-e2e-shared-a-w2-a1d55d/auth-test-model-a1d55d-kserve-7c4f64fd9-njmlg"
Oct 30 19:02:54.604974 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:54.594600    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72b01806-d7cc-4452-896b-32d64e92af97-home\") pod \"auth-test-model-a1d55d-kserve-7c4f64fd9-njmlg\" (UID: \"72b01806-d7cc-4452-896b-32d64e92af97\") " pod="ai-tenant-e2e-shared-a-w2-a1d55d/auth-test-model-a1d55d-kserve-7c4f64fd9-njmlg"
Oct 30 19:02:54.607716 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:54.605342    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72b01806-d7cc-4452-896b-32d64e92af97-model-cache\") pod \"auth-test-model-a1d55d-kserve-7c4f64fd9-njmlg\" (UID: \"72b01806-d7cc-4452-896b-32d64e92af97\") " pod="ai-tenant-e2e-shared-a-w2-a1d55d/auth-test-model-a1d55d-kserve-7c4f64fd9-njmlg"
Oct 30 19:02:54.642909 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:54.632144    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72b01806-d7cc-4452-896b-32d64e92af97-dshm\") pod \"auth-test-model-a1d55d-kserve-7c4f64fd9-njmlg\" (UID: \"72b01806-d7cc-4452-896b-32d64e92af97\") " pod="ai-tenant-e2e-shared-a-w2-a1d55d/auth-test-model-a1d55d-kserve-7c4f64fd9-njmlg"
Oct 30 19:02:54.642909 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:54.633476    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72b01806-d7cc-4452-896b-32d64e92af97-tls-certs\") pod \"auth-test-model-a1d55d-kserve-7c4f64fd9-njmlg\" (UID: \"72b01806-d7cc-4452-896b-32d64e92af97\") " pod="ai-tenant-e2e-shared-a-w2-a1d55d/auth-test-model-a1d55d-kserve-7c4f64fd9-njmlg"
Oct 30 19:02:54.642909 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:54.636462    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2mz7\" (UniqueName: \"kubernetes.io/projected/72b01806-d7cc-4452-896b-32d64e92af97-kube-api-access-q2mz7\") pod \"auth-test-model-a1d55d-kserve-7c4f64fd9-njmlg\" (UID: \"72b01806-d7cc-4452-896b-32d64e92af97\") " pod="ai-tenant-e2e-shared-a-w2-a1d55d/auth-test-model-a1d55d-kserve-7c4f64fd9-njmlg"
Oct 30 19:02:54.748753 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:54.748550    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-a1d55d/auth-test-model-a1d55d-kserve-7c4f64fd9-njmlg"
Oct 30 19:02:55.069040 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:02:55.069007    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod72b01806_d7cc_4452_896b_32d64e92af97.slice/crio-ebcdc7ea7cb04d4540c7cef67aa23acb114b885f57081243c7230ecf485063da WatchSource:0}: Error finding container ebcdc7ea7cb04d4540c7cef67aa23acb114b885f57081243c7230ecf485063da: Status 404 returned error can't find the container with id ebcdc7ea7cb04d4540c7cef67aa23acb114b885f57081243c7230ecf485063da
Oct 30 19:02:55.069835 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:55.069712    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-a1d55d/auth-test-model-a1d55d-kserve-7c4f64fd9-njmlg"]
Oct 30 19:02:55.240659 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:55.240627    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-ef6b56/auth-test-model-ef6b56-kserve-66c7fd94b9-qgqj4"
Oct 30 19:02:55.370389 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:55.370348    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-a1d55d/auth-test-model-a1d55d-kserve-7c4f64fd9-njmlg" event={"ID":"72b01806-d7cc-4452-896b-32d64e92af97","Type":"ContainerStarted","Data":"60fba0878041766f2b49adbac4b18a306253254ca7d10037190ae03cc4d1f0d3"}
Oct 30 19:02:55.370599 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:55.370406    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-a1d55d/auth-test-model-a1d55d-kserve-7c4f64fd9-njmlg" event={"ID":"72b01806-d7cc-4452-896b-32d64e92af97","Type":"ContainerStarted","Data":"ebcdc7ea7cb04d4540c7cef67aa23acb114b885f57081243c7230ecf485063da"}
Oct 30 19:02:55.372513 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:55.372480    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9b222d4-cshq9" event={"ID":"aaf6f1db-cc65-4e6b-ac33-b4848b527f8a","Type":"ContainerStarted","Data":"2aa93c230ceed97b517836d94bc90c1a0c08010b7abbf721f6871f85850bbee9"}
Oct 30 19:02:55.397761 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:55.397709    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-a1d55d/auth-test-model-a1d55d-kserve-7c4f64fd9-njmlg" podStartSLOduration=1.397670215 podStartE2EDuration="1.397670215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:02:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:02:55.392128685 +0000 UTC m=+1598.017798662" watchObservedRunningTime="2026-10-30 19:02:55.397670215 +0000 UTC m=+1598.023340179"
Oct 30 19:02:55.413917 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:55.413850    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9b222d4-cshq9" podStartSLOduration=2.4138277 podStartE2EDuration="2.4138277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:02:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:02:55.409484605 +0000 UTC m=+1598.035154580" watchObservedRunningTime="2026-10-30 19:02:55.4138277 +0000 UTC m=+1598.039497665"
Oct 30 19:02:55.751220 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:55.751127    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-a1d55d/auth-test-model-a1d55d-kserve-7c4f64fd9-njmlg"
Oct 30 19:02:56.474557 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:56.474526    4525 generic.go:356] "Generic (PLEG): container finished" podID="aaf6f1db-cc65-4e6b-ac33-b4848b527f8a" containerID="2aa93c230ceed97b517836d94bc90c1a0c08010b7abbf721f6871f85850bbee9" exitCode=0
Oct 30 19:02:56.475834 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:56.475801    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9b222d4-cshq9" event={"ID":"aaf6f1db-cc65-4e6b-ac33-b4848b527f8a","Type":"ContainerDied","Data":"2aa93c230ceed97b517836d94bc90c1a0c08010b7abbf721f6871f85850bbee9"}
Oct 30 19:02:57.860523 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:57.860498    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9b222d4-cshq9"
Oct 30 19:02:57.959372 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:57.959334    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aaf6f1db-cc65-4e6b-ac33-b4848b527f8a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaf6f1db-cc65-4e6b-ac33-b4848b527f8a-maas-api-openshift-ca\") pod \"aaf6f1db-cc65-4e6b-ac33-b4848b527f8a\" (UID: \"aaf6f1db-cc65-4e6b-ac33-b4848b527f8a\") "
Oct 30 19:02:57.959566 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:57.959437    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aaf6f1db-cc65-4e6b-ac33-b4848b527f8a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaf6f1db-cc65-4e6b-ac33-b4848b527f8a-maas-api-xks-ca\") pod \"aaf6f1db-cc65-4e6b-ac33-b4848b527f8a\" (UID: \"aaf6f1db-cc65-4e6b-ac33-b4848b527f8a\") "
Oct 30 19:02:57.959639 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:57.959620    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaf6f1db-cc65-4e6b-ac33-b4848b527f8a-kube-api-access-9pnn4\" (UniqueName: \"kubernetes.io/projected/aaf6f1db-cc65-4e6b-ac33-b4848b527f8a-kube-api-access-9pnn4\") pod \"aaf6f1db-cc65-4e6b-ac33-b4848b527f8a\" (UID: \"aaf6f1db-cc65-4e6b-ac33-b4848b527f8a\") "
Oct 30 19:02:57.960482 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:57.960447    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaf6f1db-cc65-4e6b-ac33-b4848b527f8a-maas-api-openshift-ca" pod "aaf6f1db-cc65-4e6b-ac33-b4848b527f8a" (UID: "aaf6f1db-cc65-4e6b-ac33-b4848b527f8a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:02:57.961036 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:57.960940    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaf6f1db-cc65-4e6b-ac33-b4848b527f8a-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:02:57.961955 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:57.961932    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aaf6f1db-cc65-4e6b-ac33-b4848b527f8a-maas-api-xks-ca" pod "aaf6f1db-cc65-4e6b-ac33-b4848b527f8a" (UID: "aaf6f1db-cc65-4e6b-ac33-b4848b527f8a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:02:57.962316 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:57.962298    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaf6f1db-cc65-4e6b-ac33-b4848b527f8a-kube-api-access-9pnn4" pod "aaf6f1db-cc65-4e6b-ac33-b4848b527f8a" (UID: "aaf6f1db-cc65-4e6b-ac33-b4848b527f8a"). InnerVolumeSpecName "kube-api-access-9pnn4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:02:58.070443 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:58.068313    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaf6f1db-cc65-4e6b-ac33-b4848b527f8a-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:02:58.070443 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:58.068351    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pnn4\" (UniqueName: \"kubernetes.io/projected/aaf6f1db-cc65-4e6b-ac33-b4848b527f8a-kube-api-access-9pnn4\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:02:58.549699 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:58.549636    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9b222d4-cshq9" event={"ID":"aaf6f1db-cc65-4e6b-ac33-b4848b527f8a","Type":"ContainerDied","Data":"54e5522d8f27f70db48dbd3c448b3dd0227266477b633c5b33ab5b2b868cd42f"}
Oct 30 19:02:58.549923 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:58.549757    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="54e5522d8f27f70db48dbd3c448b3dd0227266477b633c5b33ab5b2b868cd42f"
Oct 30 19:02:58.549993 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:02:58.549928    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9b222d4-cshq9"
Oct 30 19:03:03.240803 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:03.240715    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-ef6b56/auth-test-model-ef6b56-kserve-66c7fd94b9-qgqj4"
Oct 30 19:03:03.255021 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:03.254984    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-ef6b56/auth-test-model-ef6b56-kserve-66c7fd94b9-qgqj4"
Oct 30 19:03:03.740968 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:03.740940    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-ef6b56/auth-test-model-ef6b56-kserve-66c7fd94b9-qgqj4"
Oct 30 19:03:04.309874 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:04.309835    4525 patch_prober.go:29] interesting pod/e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:03:04.309874 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:04.309834    4525 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:03:04.310388 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:04.309925    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn" podUID="e77e530a-8e80-4371-840e-b2345bf68449" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:03:04.310388 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:04.309925    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944" podUID="92a5cb37-4cd6-466c-bf3b-740d3573da07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:03:04.750413 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:04.750373    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-a1d55d/auth-test-model-a1d55d-kserve-7c4f64fd9-njmlg"
Oct 30 19:03:04.763823 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:04.763791    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-a1d55d/auth-test-model-a1d55d-kserve-7c4f64fd9-njmlg"
Oct 30 19:03:05.478481 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:05.478443    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k"]
Oct 30 19:03:05.479417 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:05.479386    4525 cpu_manager.go:414] "containerMap: removing container" podUID="aaf6f1db-cc65-4e6b-ac33-b4848b527f8a" containerName="revoke-keys"
Oct 30 19:03:05.479417 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:05.479418    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaf6f1db-cc65-4e6b-ac33-b4848b527f8a" containerName="revoke-keys"
Oct 30 19:03:05.479629 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:05.479615    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="aaf6f1db-cc65-4e6b-ac33-b4848b527f8a" containerName="revoke-keys"
Oct 30 19:03:05.503079 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:05.503048    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k"]
Oct 30 19:03:05.503271 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:05.503170    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k"
Oct 30 19:03:05.507921 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:05.507880    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-e9ada0-openshift-default-dockercfg-dr8jc\""
Oct 30 19:03:05.617184 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:05.617139    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/755fd8c8-1c8a-4040-b426-405e8e5789aa-workload-socket\") pod \"e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k\" (UID: \"755fd8c8-1c8a-4040-b426-405e8e5789aa\") " pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k"
Oct 30 19:03:05.639756 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:05.631187    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/755fd8c8-1c8a-4040-b426-405e8e5789aa-istio-data\") pod \"e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k\" (UID: \"755fd8c8-1c8a-4040-b426-405e8e5789aa\") " pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k"
Oct 30 19:03:05.639756 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:05.631298    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/755fd8c8-1c8a-4040-b426-405e8e5789aa-credential-socket\") pod \"e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k\" (UID: \"755fd8c8-1c8a-4040-b426-405e8e5789aa\") " pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k"
Oct 30 19:03:05.639756 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:05.631423    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/755fd8c8-1c8a-4040-b426-405e8e5789aa-istio-envoy\") pod \"e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k\" (UID: \"755fd8c8-1c8a-4040-b426-405e8e5789aa\") " pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k"
Oct 30 19:03:05.639756 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:05.631530    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/755fd8c8-1c8a-4040-b426-405e8e5789aa-istiod-ca-cert\") pod \"e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k\" (UID: \"755fd8c8-1c8a-4040-b426-405e8e5789aa\") " pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k"
Oct 30 19:03:05.653705 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:05.652363    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/755fd8c8-1c8a-4040-b426-405e8e5789aa-istio-token\") pod \"e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k\" (UID: \"755fd8c8-1c8a-4040-b426-405e8e5789aa\") " pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k"
Oct 30 19:03:05.653705 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:05.652445    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/755fd8c8-1c8a-4040-b426-405e8e5789aa-istio-podinfo\") pod \"e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k\" (UID: \"755fd8c8-1c8a-4040-b426-405e8e5789aa\") " pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k"
Oct 30 19:03:05.653705 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:05.652491    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dchpr\" (UniqueName: \"kubernetes.io/projected/755fd8c8-1c8a-4040-b426-405e8e5789aa-kube-api-access-dchpr\") pod \"e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k\" (UID: \"755fd8c8-1c8a-4040-b426-405e8e5789aa\") " pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k"
Oct 30 19:03:05.653705 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:05.652589    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/755fd8c8-1c8a-4040-b426-405e8e5789aa-workload-certs\") pod \"e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k\" (UID: \"755fd8c8-1c8a-4040-b426-405e8e5789aa\") " pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k"
Oct 30 19:03:05.822993 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:05.822892    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/755fd8c8-1c8a-4040-b426-405e8e5789aa-workload-certs\") pod \"e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k\" (UID: \"755fd8c8-1c8a-4040-b426-405e8e5789aa\") " pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k"
Oct 30 19:03:05.823777 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:05.823742    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/755fd8c8-1c8a-4040-b426-405e8e5789aa-workload-socket\") pod \"e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k\" (UID: \"755fd8c8-1c8a-4040-b426-405e8e5789aa\") " pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k"
Oct 30 19:03:05.824161 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:05.824128    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/755fd8c8-1c8a-4040-b426-405e8e5789aa-istio-data\") pod \"e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k\" (UID: \"755fd8c8-1c8a-4040-b426-405e8e5789aa\") " pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k"
Oct 30 19:03:05.824436 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:05.824404    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/755fd8c8-1c8a-4040-b426-405e8e5789aa-credential-socket\") pod \"e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k\" (UID: \"755fd8c8-1c8a-4040-b426-405e8e5789aa\") " pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k"
Oct 30 19:03:05.824616 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:05.824585    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/755fd8c8-1c8a-4040-b426-405e8e5789aa-istio-envoy\") pod \"e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k\" (UID: \"755fd8c8-1c8a-4040-b426-405e8e5789aa\") " pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k"
Oct 30 19:03:05.824828 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:05.824805    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/755fd8c8-1c8a-4040-b426-405e8e5789aa-istiod-ca-cert\") pod \"e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k\" (UID: \"755fd8c8-1c8a-4040-b426-405e8e5789aa\") " pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k"
Oct 30 19:03:05.827204 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:05.827181    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/755fd8c8-1c8a-4040-b426-405e8e5789aa-istio-token\") pod \"e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k\" (UID: \"755fd8c8-1c8a-4040-b426-405e8e5789aa\") " pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k"
Oct 30 19:03:05.827408 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:05.827387    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/755fd8c8-1c8a-4040-b426-405e8e5789aa-istio-podinfo\") pod \"e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k\" (UID: \"755fd8c8-1c8a-4040-b426-405e8e5789aa\") " pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k"
Oct 30 19:03:05.827728 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:05.827674    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dchpr\" (UniqueName: \"kubernetes.io/projected/755fd8c8-1c8a-4040-b426-405e8e5789aa-kube-api-access-dchpr\") pod \"e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k\" (UID: \"755fd8c8-1c8a-4040-b426-405e8e5789aa\") " pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k"
Oct 30 19:03:05.828211 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:05.825821    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/755fd8c8-1c8a-4040-b426-405e8e5789aa-istio-data\") pod \"e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k\" (UID: \"755fd8c8-1c8a-4040-b426-405e8e5789aa\") " pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k"
Oct 30 19:03:05.828909 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:05.823623    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/755fd8c8-1c8a-4040-b426-405e8e5789aa-workload-certs\") pod \"e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k\" (UID: \"755fd8c8-1c8a-4040-b426-405e8e5789aa\") " pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k"
Oct 30 19:03:05.829467 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:05.825574    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/755fd8c8-1c8a-4040-b426-405e8e5789aa-credential-socket\") pod \"e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k\" (UID: \"755fd8c8-1c8a-4040-b426-405e8e5789aa\") " pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k"
Oct 30 19:03:05.829630 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:05.825049    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/755fd8c8-1c8a-4040-b426-405e8e5789aa-workload-socket\") pod \"e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k\" (UID: \"755fd8c8-1c8a-4040-b426-405e8e5789aa\") " pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k"
Oct 30 19:03:05.830568 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:05.830545    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/755fd8c8-1c8a-4040-b426-405e8e5789aa-istiod-ca-cert\") pod \"e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k\" (UID: \"755fd8c8-1c8a-4040-b426-405e8e5789aa\") " pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k"
Oct 30 19:03:05.871576 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:05.870620    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-a1d55d/auth-test-model-a1d55d-kserve-7c4f64fd9-njmlg"
Oct 30 19:03:05.879957 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:05.879505    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/755fd8c8-1c8a-4040-b426-405e8e5789aa-istio-envoy\") pod \"e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k\" (UID: \"755fd8c8-1c8a-4040-b426-405e8e5789aa\") " pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k"
Oct 30 19:03:05.879957 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:05.879824    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/755fd8c8-1c8a-4040-b426-405e8e5789aa-istio-token\") pod \"e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k\" (UID: \"755fd8c8-1c8a-4040-b426-405e8e5789aa\") " pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k"
Oct 30 19:03:05.887016 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:05.886895    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/755fd8c8-1c8a-4040-b426-405e8e5789aa-istio-podinfo\") pod \"e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k\" (UID: \"755fd8c8-1c8a-4040-b426-405e8e5789aa\") " pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k"
Oct 30 19:03:05.891319 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:05.891287    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dchpr\" (UniqueName: \"kubernetes.io/projected/755fd8c8-1c8a-4040-b426-405e8e5789aa-kube-api-access-dchpr\") pod \"e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k\" (UID: \"755fd8c8-1c8a-4040-b426-405e8e5789aa\") " pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k"
Oct 30 19:03:06.149861 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:06.149825    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k"
Oct 30 19:03:06.469018 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:03:06.468986    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod755fd8c8_1c8a_4040_b426_405e8e5789aa.slice/crio-72264bf6f4883697d5891848a6dff6ea297e83814766027ced203f83cdac55de WatchSource:0}: Error finding container 72264bf6f4883697d5891848a6dff6ea297e83814766027ced203f83cdac55de: Status 404 returned error can't find the container with id 72264bf6f4883697d5891848a6dff6ea297e83814766027ced203f83cdac55de
Oct 30 19:03:06.471840 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:06.471813    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k"]
Oct 30 19:03:06.472428 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:06.472384    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:03:06.472492 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:06.472473    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:03:06.472563 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:06.472547    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:03:06.885987 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:06.885941    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k" event={"ID":"755fd8c8-1c8a-4040-b426-405e8e5789aa","Type":"ContainerStarted","Data":"a380b91526beebecb86524c7b78d56960e963c0cebf965e7293848ddbb0eb420"}
Oct 30 19:03:06.886509 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:06.886015    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k" event={"ID":"755fd8c8-1c8a-4040-b426-405e8e5789aa","Type":"ContainerStarted","Data":"72264bf6f4883697d5891848a6dff6ea297e83814766027ced203f83cdac55de"}
Oct 30 19:03:06.915391 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:06.915319    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k" podStartSLOduration=1.915303362 podStartE2EDuration="1.915303362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:03:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:03:06.907967267 +0000 UTC m=+1609.533637233" watchObservedRunningTime="2026-10-30 19:03:06.915303362 +0000 UTC m=+1609.540973326"
Oct 30 19:03:07.154187 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:07.154101    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k"
Oct 30 19:03:07.172936 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:07.172894    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k"
Oct 30 19:03:07.821486 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:07.821441    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-c2e3f9/auth-test-model-c2e3f9-kserve-6b9f58d9cc-pmkg2"]
Oct 30 19:03:08.297225 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:07.836947    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c2e3f9/auth-test-model-c2e3f9-kserve-6b9f58d9cc-pmkg2"]
Oct 30 19:03:08.297225 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:07.837063    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-c2e3f9/auth-test-model-c2e3f9-kserve-6b9f58d9cc-pmkg2"
Oct 30 19:03:08.297225 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:07.846889    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-c2e3f9\"/\"default-dockercfg-2tgmd\""
Oct 30 19:03:08.297225 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:07.846914    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-c2e3f9\"/\"openshift-service-ca.crt\""
Oct 30 19:03:08.297225 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:07.846932    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-c2e3f9\"/\"auth-test-model-c2e3f9-kserve-self-signed-certs\""
Oct 30 19:03:08.297225 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:07.846973    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-c2e3f9\"/\"kube-root-ca.crt\""
Oct 30 19:03:08.297225 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:07.878979    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fedf2771-d37f-4d43-83f4-a04290b9fa9c-tls-certs\") pod \"auth-test-model-c2e3f9-kserve-6b9f58d9cc-pmkg2\" (UID: \"fedf2771-d37f-4d43-83f4-a04290b9fa9c\") " pod="ai-tenant-e2e-shared-a-w3-c2e3f9/auth-test-model-c2e3f9-kserve-6b9f58d9cc-pmkg2"
Oct 30 19:03:08.297225 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:07.879048    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fedf2771-d37f-4d43-83f4-a04290b9fa9c-home\") pod \"auth-test-model-c2e3f9-kserve-6b9f58d9cc-pmkg2\" (UID: \"fedf2771-d37f-4d43-83f4-a04290b9fa9c\") " pod="ai-tenant-e2e-shared-a-w3-c2e3f9/auth-test-model-c2e3f9-kserve-6b9f58d9cc-pmkg2"
Oct 30 19:03:08.297225 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:07.879293    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4t9m\" (UniqueName: \"kubernetes.io/projected/fedf2771-d37f-4d43-83f4-a04290b9fa9c-kube-api-access-z4t9m\") pod \"auth-test-model-c2e3f9-kserve-6b9f58d9cc-pmkg2\" (UID: \"fedf2771-d37f-4d43-83f4-a04290b9fa9c\") " pod="ai-tenant-e2e-shared-a-w3-c2e3f9/auth-test-model-c2e3f9-kserve-6b9f58d9cc-pmkg2"
Oct 30 19:03:08.297225 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:07.879328    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fedf2771-d37f-4d43-83f4-a04290b9fa9c-dshm\") pod \"auth-test-model-c2e3f9-kserve-6b9f58d9cc-pmkg2\" (UID: \"fedf2771-d37f-4d43-83f4-a04290b9fa9c\") " pod="ai-tenant-e2e-shared-a-w3-c2e3f9/auth-test-model-c2e3f9-kserve-6b9f58d9cc-pmkg2"
Oct 30 19:03:08.297225 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:07.879352    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fedf2771-d37f-4d43-83f4-a04290b9fa9c-model-cache\") pod \"auth-test-model-c2e3f9-kserve-6b9f58d9cc-pmkg2\" (UID: \"fedf2771-d37f-4d43-83f4-a04290b9fa9c\") " pod="ai-tenant-e2e-shared-a-w3-c2e3f9/auth-test-model-c2e3f9-kserve-6b9f58d9cc-pmkg2"
Oct 30 19:03:08.297225 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:07.879453    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fedf2771-d37f-4d43-83f4-a04290b9fa9c-tmp-dir\") pod \"auth-test-model-c2e3f9-kserve-6b9f58d9cc-pmkg2\" (UID: \"fedf2771-d37f-4d43-83f4-a04290b9fa9c\") " pod="ai-tenant-e2e-shared-a-w3-c2e3f9/auth-test-model-c2e3f9-kserve-6b9f58d9cc-pmkg2"
Oct 30 19:03:08.297225 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:07.894532    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k"
Oct 30 19:03:08.297225 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:07.895563    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k"
Oct 30 19:03:08.297225 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:07.981218    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4t9m\" (UniqueName: \"kubernetes.io/projected/fedf2771-d37f-4d43-83f4-a04290b9fa9c-kube-api-access-z4t9m\") pod \"auth-test-model-c2e3f9-kserve-6b9f58d9cc-pmkg2\" (UID: \"fedf2771-d37f-4d43-83f4-a04290b9fa9c\") " pod="ai-tenant-e2e-shared-a-w3-c2e3f9/auth-test-model-c2e3f9-kserve-6b9f58d9cc-pmkg2"
Oct 30 19:03:08.297225 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:07.981282    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fedf2771-d37f-4d43-83f4-a04290b9fa9c-dshm\") pod \"auth-test-model-c2e3f9-kserve-6b9f58d9cc-pmkg2\" (UID: \"fedf2771-d37f-4d43-83f4-a04290b9fa9c\") " pod="ai-tenant-e2e-shared-a-w3-c2e3f9/auth-test-model-c2e3f9-kserve-6b9f58d9cc-pmkg2"
Oct 30 19:03:08.297225 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:07.981314    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fedf2771-d37f-4d43-83f4-a04290b9fa9c-model-cache\") pod \"auth-test-model-c2e3f9-kserve-6b9f58d9cc-pmkg2\" (UID: \"fedf2771-d37f-4d43-83f4-a04290b9fa9c\") " pod="ai-tenant-e2e-shared-a-w3-c2e3f9/auth-test-model-c2e3f9-kserve-6b9f58d9cc-pmkg2"
Oct 30 19:03:08.297225 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:07.981375    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fedf2771-d37f-4d43-83f4-a04290b9fa9c-tmp-dir\") pod \"auth-test-model-c2e3f9-kserve-6b9f58d9cc-pmkg2\" (UID: \"fedf2771-d37f-4d43-83f4-a04290b9fa9c\") " pod="ai-tenant-e2e-shared-a-w3-c2e3f9/auth-test-model-c2e3f9-kserve-6b9f58d9cc-pmkg2"
Oct 30 19:03:08.297225 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:07.981431    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fedf2771-d37f-4d43-83f4-a04290b9fa9c-tls-certs\") pod \"auth-test-model-c2e3f9-kserve-6b9f58d9cc-pmkg2\" (UID: \"fedf2771-d37f-4d43-83f4-a04290b9fa9c\") " pod="ai-tenant-e2e-shared-a-w3-c2e3f9/auth-test-model-c2e3f9-kserve-6b9f58d9cc-pmkg2"
Oct 30 19:03:08.297225 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:07.981506    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fedf2771-d37f-4d43-83f4-a04290b9fa9c-home\") pod \"auth-test-model-c2e3f9-kserve-6b9f58d9cc-pmkg2\" (UID: \"fedf2771-d37f-4d43-83f4-a04290b9fa9c\") " pod="ai-tenant-e2e-shared-a-w3-c2e3f9/auth-test-model-c2e3f9-kserve-6b9f58d9cc-pmkg2"
Oct 30 19:03:08.297225 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:07.981887    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fedf2771-d37f-4d43-83f4-a04290b9fa9c-model-cache\") pod \"auth-test-model-c2e3f9-kserve-6b9f58d9cc-pmkg2\" (UID: \"fedf2771-d37f-4d43-83f4-a04290b9fa9c\") " pod="ai-tenant-e2e-shared-a-w3-c2e3f9/auth-test-model-c2e3f9-kserve-6b9f58d9cc-pmkg2"
Oct 30 19:03:08.297225 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:07.981895    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fedf2771-d37f-4d43-83f4-a04290b9fa9c-home\") pod \"auth-test-model-c2e3f9-kserve-6b9f58d9cc-pmkg2\" (UID: \"fedf2771-d37f-4d43-83f4-a04290b9fa9c\") " pod="ai-tenant-e2e-shared-a-w3-c2e3f9/auth-test-model-c2e3f9-kserve-6b9f58d9cc-pmkg2"
Oct 30 19:03:08.297225 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:07.981942    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fedf2771-d37f-4d43-83f4-a04290b9fa9c-tmp-dir\") pod \"auth-test-model-c2e3f9-kserve-6b9f58d9cc-pmkg2\" (UID: \"fedf2771-d37f-4d43-83f4-a04290b9fa9c\") " pod="ai-tenant-e2e-shared-a-w3-c2e3f9/auth-test-model-c2e3f9-kserve-6b9f58d9cc-pmkg2"
Oct 30 19:03:08.297225 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:07.984421    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fedf2771-d37f-4d43-83f4-a04290b9fa9c-dshm\") pod \"auth-test-model-c2e3f9-kserve-6b9f58d9cc-pmkg2\" (UID: \"fedf2771-d37f-4d43-83f4-a04290b9fa9c\") " pod="ai-tenant-e2e-shared-a-w3-c2e3f9/auth-test-model-c2e3f9-kserve-6b9f58d9cc-pmkg2"
Oct 30 19:03:08.297225 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:07.984432    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fedf2771-d37f-4d43-83f4-a04290b9fa9c-tls-certs\") pod \"auth-test-model-c2e3f9-kserve-6b9f58d9cc-pmkg2\" (UID: \"fedf2771-d37f-4d43-83f4-a04290b9fa9c\") " pod="ai-tenant-e2e-shared-a-w3-c2e3f9/auth-test-model-c2e3f9-kserve-6b9f58d9cc-pmkg2"
Oct 30 19:03:08.297225 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:07.992278    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4t9m\" (UniqueName: \"kubernetes.io/projected/fedf2771-d37f-4d43-83f4-a04290b9fa9c-kube-api-access-z4t9m\") pod \"auth-test-model-c2e3f9-kserve-6b9f58d9cc-pmkg2\" (UID: \"fedf2771-d37f-4d43-83f4-a04290b9fa9c\") " pod="ai-tenant-e2e-shared-a-w3-c2e3f9/auth-test-model-c2e3f9-kserve-6b9f58d9cc-pmkg2"
Oct 30 19:03:08.297225 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:08.174571    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-c2e3f9/auth-test-model-c2e3f9-kserve-6b9f58d9cc-pmkg2"
Oct 30 19:03:08.488620 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:08.488591    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c2e3f9/auth-test-model-c2e3f9-kserve-6b9f58d9cc-pmkg2"]
Oct 30 19:03:08.905246 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:08.905214    4525 generic.go:356] "Generic (PLEG): container finished" podID="92a5cb37-4cd6-466c-bf3b-740d3573da07" containerID="2ff7347f12d88897fab512adef677fc96a7de0cae3bce0381e7b5ac746082424" exitCode=0
Oct 30 19:03:08.905443 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:08.905284    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944" event={"ID":"92a5cb37-4cd6-466c-bf3b-740d3573da07","Type":"ContainerDied","Data":"2ff7347f12d88897fab512adef677fc96a7de0cae3bce0381e7b5ac746082424"}
Oct 30 19:03:08.905848 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:08.905827    4525 scope.go:122] "RemoveContainer" containerID="2ff7347f12d88897fab512adef677fc96a7de0cae3bce0381e7b5ac746082424"
Oct 30 19:03:08.907370 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:08.907351    4525 generic.go:356] "Generic (PLEG): container finished" podID="e77e530a-8e80-4371-840e-b2345bf68449" containerID="bc1fc55950a91a7538ea3218d9447d112374babdbb61184d2633be1fa66bd903" exitCode=0
Oct 30 19:03:08.907473 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:08.907435    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn" event={"ID":"e77e530a-8e80-4371-840e-b2345bf68449","Type":"ContainerDied","Data":"bc1fc55950a91a7538ea3218d9447d112374babdbb61184d2633be1fa66bd903"}
Oct 30 19:03:08.907589 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:08.907564    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:03:08.907655 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:08.907636    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:03:08.908263 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:08.907710    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:03:08.908263 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:08.907876    4525 scope.go:122] "RemoveContainer" containerID="bc1fc55950a91a7538ea3218d9447d112374babdbb61184d2633be1fa66bd903"
Oct 30 19:03:08.909595 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:08.909563    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:03:08.909771 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:08.909643    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:03:08.909771 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:08.909713    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:03:08.910022 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:08.909998    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-c2e3f9/auth-test-model-c2e3f9-kserve-6b9f58d9cc-pmkg2" event={"ID":"fedf2771-d37f-4d43-83f4-a04290b9fa9c","Type":"ContainerStarted","Data":"db0a0c54e501691d218078b68d968fd9e3da84b12465ce8c84d2b75d0ef38dfb"}
Oct 30 19:03:08.910129 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:08.910048    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-c2e3f9/auth-test-model-c2e3f9-kserve-6b9f58d9cc-pmkg2" event={"ID":"fedf2771-d37f-4d43-83f4-a04290b9fa9c","Type":"ContainerStarted","Data":"b5e9a46ffd085278d35584f89511bbacaeee5b42909d9db2427c8dad5722a1d9"}
Oct 30 19:03:08.950969 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:08.950905    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-c2e3f9/auth-test-model-c2e3f9-kserve-6b9f58d9cc-pmkg2" podStartSLOduration=1.950884705 podStartE2EDuration="1.950884705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:03:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:03:08.945114048 +0000 UTC m=+1611.570784013" watchObservedRunningTime="2026-10-30 19:03:08.950884705 +0000 UTC m=+1611.576554671"
Oct 30 19:03:09.174867 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:09.174781    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-c2e3f9/auth-test-model-c2e3f9-kserve-6b9f58d9cc-pmkg2"
Oct 30 19:03:09.993076 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:09.993024    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944" event={"ID":"92a5cb37-4cd6-466c-bf3b-740d3573da07","Type":"ContainerStarted","Data":"afda9a783e450c5c8cd9b7ccfde31be7702833962128f3da7b465318d4d5c377"}
Oct 30 19:03:10.057985 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:10.056909    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn" event={"ID":"e77e530a-8e80-4371-840e-b2345bf68449","Type":"ContainerStarted","Data":"d740cc14106fe5bc6d83a0eb11e4e9b6d757b4e0a0ff175aea9962d2509198e7"}
Oct 30 19:03:10.072752 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:10.072720    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944"
Oct 30 19:03:10.073053 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:10.073020    4525 patch_prober.go:29] interesting pod/e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Oct 30 19:03:10.073128 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:10.073111    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944" podUID="92a5cb37-4cd6-466c-bf3b-740d3573da07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Oct 30 19:03:10.199242 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:10.199153    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn"
Oct 30 19:03:10.201602 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:10.201571    4525 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Oct 30 19:03:10.201871 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:10.201842    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn" podUID="e77e530a-8e80-4371-840e-b2345bf68449" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Oct 30 19:03:11.072564 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:11.072528    4525 patch_prober.go:29] interesting pod/e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Oct 30 19:03:11.073030 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:11.072591    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944" podUID="92a5cb37-4cd6-466c-bf3b-740d3573da07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Oct 30 19:03:11.204707 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:11.194145    4525 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Oct 30 19:03:11.204707 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:11.194229    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn" podUID="e77e530a-8e80-4371-840e-b2345bf68449" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Oct 30 19:03:12.072811 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:12.072775    4525 patch_prober.go:29] interesting pod/e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Oct 30 19:03:12.073348 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:12.072831    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944" podUID="92a5cb37-4cd6-466c-bf3b-740d3573da07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Oct 30 19:03:12.219714 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:12.210282    4525 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Oct 30 19:03:12.219714 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:12.210346    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn" podUID="e77e530a-8e80-4371-840e-b2345bf68449" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Oct 30 19:03:13.072771 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:13.072732    4525 patch_prober.go:29] interesting pod/e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Oct 30 19:03:13.072984 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:13.072795    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944" podUID="92a5cb37-4cd6-466c-bf3b-740d3573da07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Oct 30 19:03:13.197163 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:13.197121    4525 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Oct 30 19:03:13.197462 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:13.197432    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn" podUID="e77e530a-8e80-4371-840e-b2345bf68449" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Oct 30 19:03:14.074707 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:14.073817    4525 patch_prober.go:29] interesting pod/e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Oct 30 19:03:14.074707 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:14.073881    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944" podUID="92a5cb37-4cd6-466c-bf3b-740d3573da07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Oct 30 19:03:14.191505 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:14.191053    4525 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Oct 30 19:03:14.191505 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:14.191117    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn" podUID="e77e530a-8e80-4371-840e-b2345bf68449" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Oct 30 19:03:14.800061 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:14.792610    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj" podUID="84c99c49-1d60-4176-bda4-cad29c7d801c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 19:03:15.101797 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:15.076786    4525 patch_prober.go:29] interesting pod/e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Oct 30 19:03:15.103014 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:15.102634    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944" podUID="92a5cb37-4cd6-466c-bf3b-740d3573da07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Oct 30 19:03:15.215716 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:15.215406    4525 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Oct 30 19:03:15.215716 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:15.215529    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn" podUID="e77e530a-8e80-4371-840e-b2345bf68449" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Oct 30 19:03:15.760739 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:15.757058    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-ef6b56/auth-test-model-ef6b56-kserve-66c7fd94b9-qgqj4" podUID="ec0f645b-289a-4b05-a984-f0f66e896357" containerName="main" probeResult="failure" output="Get \"https://10.133.0.199:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:03:15.795707 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:15.766272    4525 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-shared-a-w2-a1d55d/auth-test-model-a1d55d-kserve-7c4f64fd9-njmlg" podUID="72b01806-d7cc-4452-896b-32d64e92af97" containerName="main" probeResult="failure" output="Get \"https://10.133.0.201:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:03:15.795707 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:15.767673    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-a1d55d/auth-test-model-a1d55d-kserve-7c4f64fd9-njmlg" podUID="72b01806-d7cc-4452-896b-32d64e92af97" containerName="main" probeResult="failure" output="Get \"https://10.133.0.201:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:03:15.864708 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:15.860989    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj" podUID="84c99c49-1d60-4176-bda4-cad29c7d801c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:03:15.963704 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:15.959784    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts" podUID="be6d4969-285b-4da0-883e-4847f2cbf3af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Oct 30 19:03:16.033772 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:16.014170    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv" podUID="595230e3-55f3-43eb-9a7b-6d89250af9b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 19:03:16.033772 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:16.014534    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t" podUID="424fca14-6d8a-4663-86ec-0c91165f34ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 19:03:16.038704 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:16.038474    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw" podUID="ac6852fd-01be-4d69-8c9b-0c591e50b889" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:03:16.043707 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:16.041347    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb" podUID="12ebc8b9-c794-47b5-b87b-a63c4bd993e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:03:16.120707 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:16.105942    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr" podUID="82bbbf85-ed93-48d2-9f17-49ac1f31c0a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:03:16.156823 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:16.156784    4525 patch_prober.go:29] interesting pod/e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Oct 30 19:03:16.157088 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:16.157058    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944" podUID="92a5cb37-4cd6-466c-bf3b-740d3573da07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Oct 30 19:03:16.215843 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:16.214950    4525 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Oct 30 19:03:16.215843 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:16.215016    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn" podUID="e77e530a-8e80-4371-840e-b2345bf68449" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Oct 30 19:03:16.271573 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:16.238982    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj"
Oct 30 19:03:16.279825 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:16.279788    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2"]
Oct 30 19:03:16.300767 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:16.280267    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2" podUID="72b6833a-13cf-4fe8-9b8d-26f5decea7f6" containerName="istio-proxy" containerID="cri-o://43f2859aefd0804546efa7b62f01106635812344a55c17fea614a262775e3519" gracePeriod=30
Oct 30 19:03:16.847822 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:16.820490    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-ef6b56/auth-test-model-ef6b56-kserve-66c7fd94b9-qgqj4" podUID="ec0f645b-289a-4b05-a984-f0f66e896357" containerName="main" probeResult="failure" output="Get \"https://10.133.0.199:8000/ready\": context deadline exceeded"
Oct 30 19:03:16.877708 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:16.862944    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj" podUID="84c99c49-1d60-4176-bda4-cad29c7d801c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 19:03:16.881876 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:16.879805    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-ef6b56/auth-test-model-ef6b56-kserve-66c7fd94b9-qgqj4"
Oct 30 19:03:16.993718 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:16.985865    4525 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb" podUID="12ebc8b9-c794-47b5-b87b-a63c4bd993e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": context deadline exceeded"
Oct 30 19:03:17.159214 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:17.015058    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t" podUID="424fca14-6d8a-4663-86ec-0c91165f34ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 19:03:17.159214 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:17.015094    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv" podUID="595230e3-55f3-43eb-9a7b-6d89250af9b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 19:03:17.159214 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:17.101158    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t"
Oct 30 19:03:17.159214 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:17.101216    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv"
Oct 30 19:03:17.159214 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:17.068780    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb" podUID="12ebc8b9-c794-47b5-b87b-a63c4bd993e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 19:03:17.159214 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:17.097213    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw" podUID="ac6852fd-01be-4d69-8c9b-0c591e50b889" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:03:17.159214 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:17.123444    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb"
Oct 30 19:03:17.159214 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:17.123525    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw"
Oct 30 19:03:17.159214 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:17.098240    4525 patch_prober.go:29] interesting pod/e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Oct 30 19:03:17.159214 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:17.124357    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944" podUID="92a5cb37-4cd6-466c-bf3b-740d3573da07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Oct 30 19:03:17.159214 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:17.151900    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr" podUID="82bbbf85-ed93-48d2-9f17-49ac1f31c0a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:03:17.159214 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:17.152372    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr"
Oct 30 19:03:17.195531 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:17.195057    4525 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Oct 30 19:03:17.195531 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:17.195116    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn" podUID="e77e530a-8e80-4371-840e-b2345bf68449" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Oct 30 19:03:17.872492 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:17.871778    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-a1d55d/auth-test-model-a1d55d-kserve-7c4f64fd9-njmlg" podUID="72b01806-d7cc-4452-896b-32d64e92af97" containerName="main" probeResult="failure" output="Get \"https://10.133.0.201:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:03:17.872492 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:17.871873    4525 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr" podUID="82bbbf85-ed93-48d2-9f17-49ac1f31c0a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": context deadline exceeded"
Oct 30 19:03:17.872492 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:17.871942    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts" podUID="be6d4969-285b-4da0-883e-4847f2cbf3af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Oct 30 19:03:17.904747 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:17.885201    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-ef6b56/auth-test-model-ef6b56-kserve-66c7fd94b9-qgqj4" podUID="ec0f645b-289a-4b05-a984-f0f66e896357" containerName="main" probeResult="failure" output="Get \"https://10.133.0.199:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:03:17.904747 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:17.885988    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj" podUID="84c99c49-1d60-4176-bda4-cad29c7d801c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:03:17.959824 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:03:17.959779    4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d49l" containerName="extractor"
Oct 30 19:03:18.112875 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:18.112821    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb" podUID="12ebc8b9-c794-47b5-b87b-a63c4bd993e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:03:18.117487 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:18.117272    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv"
Oct 30 19:03:18.123723 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:18.123610    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw" podUID="ac6852fd-01be-4d69-8c9b-0c591e50b889" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:03:18.131190 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:18.131047    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t" podUID="424fca14-6d8a-4663-86ec-0c91165f34ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 19:03:18.158515 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:18.158475    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr" podUID="82bbbf85-ed93-48d2-9f17-49ac1f31c0a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:03:18.213707 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:18.213344    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj"
Oct 30 19:03:18.269325 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:18.269014    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-c2e3f9/auth-test-model-c2e3f9-kserve-6b9f58d9cc-pmkg2"
Oct 30 19:03:18.285124 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:18.285068    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn"
Oct 30 19:03:18.295607 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:18.295576    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944"
Oct 30 19:03:18.352746 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:18.352704    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944"
Oct 30 19:03:18.355222 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:18.355200    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr"
Oct 30 19:03:18.455018 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:18.454932    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw"
Oct 30 19:03:18.455171 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:18.455030    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb"
Oct 30 19:03:18.455171 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:18.455117    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t"
Oct 30 19:03:18.455278 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:18.455180    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-ef6b56/auth-test-model-ef6b56-kserve-66c7fd94b9-qgqj4"
Oct 30 19:03:18.474565 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:18.474536    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn"
Oct 30 19:03:18.772818 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:18.772724    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-c2e3f9/auth-test-model-c2e3f9-kserve-6b9f58d9cc-pmkg2"
Oct 30 19:03:19.147134 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:19.147094    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn"
Oct 30 19:03:19.147929 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:19.147905    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944"
Oct 30 19:03:19.162975 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:19.162952    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-c2e3f9/auth-test-model-c2e3f9-kserve-6b9f58d9cc-pmkg2"
Oct 30 19:03:19.652318 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:19.652183    4525 scope.go:122] "RemoveContainer" containerID="ba37595a8a500eed0b051adbe65314f2291f930670f1fac99ba4c0ee2f2a51ef"
Oct 30 19:03:19.730869 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:19.730105    4525 scope.go:122] "RemoveContainer" containerID="c49dd1bddf33ca71f80ed1baaf158d83718d9edd42cf13dfc708add4f17e0e04"
Oct 30 19:03:19.749175 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:19.749135    4525 scope.go:122] "RemoveContainer" containerID="e5d771e47fd5c5b6ac5f11c1293eb2e4c1596f43efb936fc1f4e62f83c73b687"
Oct 30 19:03:21.336706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:21.332761    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a1d55d-openshift-default-6646f694ff-c6hjq"]
Oct 30 19:03:21.377118 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:21.377091    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a1d55d-openshift-default-6646f694ff-c6hjq"
Oct 30 19:03:21.418709 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:21.394046    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-a1d55d-openshift-default-dockercfg-qstk8\""
Oct 30 19:03:21.443630 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:21.443596    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d05765ac-da9e-4fad-b2e8-e26b8630849e-istio-data\") pod \"e2e-shared-a-w2-a1d55d-openshift-default-6646f694ff-c6hjq\" (UID: \"d05765ac-da9e-4fad-b2e8-e26b8630849e\") " pod="openshift-ingress/e2e-shared-a-w2-a1d55d-openshift-default-6646f694ff-c6hjq"
Oct 30 19:03:21.455844 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:21.454080    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d05765ac-da9e-4fad-b2e8-e26b8630849e-istio-envoy\") pod \"e2e-shared-a-w2-a1d55d-openshift-default-6646f694ff-c6hjq\" (UID: \"d05765ac-da9e-4fad-b2e8-e26b8630849e\") " pod="openshift-ingress/e2e-shared-a-w2-a1d55d-openshift-default-6646f694ff-c6hjq"
Oct 30 19:03:21.455844 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:21.454188    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d05765ac-da9e-4fad-b2e8-e26b8630849e-credential-socket\") pod \"e2e-shared-a-w2-a1d55d-openshift-default-6646f694ff-c6hjq\" (UID: \"d05765ac-da9e-4fad-b2e8-e26b8630849e\") " pod="openshift-ingress/e2e-shared-a-w2-a1d55d-openshift-default-6646f694ff-c6hjq"
Oct 30 19:03:21.455844 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:21.454408    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d05765ac-da9e-4fad-b2e8-e26b8630849e-istio-token\") pod \"e2e-shared-a-w2-a1d55d-openshift-default-6646f694ff-c6hjq\" (UID: \"d05765ac-da9e-4fad-b2e8-e26b8630849e\") " pod="openshift-ingress/e2e-shared-a-w2-a1d55d-openshift-default-6646f694ff-c6hjq"
Oct 30 19:03:21.455844 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:21.454476    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d05765ac-da9e-4fad-b2e8-e26b8630849e-workload-certs\") pod \"e2e-shared-a-w2-a1d55d-openshift-default-6646f694ff-c6hjq\" (UID: \"d05765ac-da9e-4fad-b2e8-e26b8630849e\") " pod="openshift-ingress/e2e-shared-a-w2-a1d55d-openshift-default-6646f694ff-c6hjq"
Oct 30 19:03:21.455844 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:21.454550    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d05765ac-da9e-4fad-b2e8-e26b8630849e-istio-podinfo\") pod \"e2e-shared-a-w2-a1d55d-openshift-default-6646f694ff-c6hjq\" (UID: \"d05765ac-da9e-4fad-b2e8-e26b8630849e\") " pod="openshift-ingress/e2e-shared-a-w2-a1d55d-openshift-default-6646f694ff-c6hjq"
Oct 30 19:03:21.455844 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:21.454586    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d05765ac-da9e-4fad-b2e8-e26b8630849e-workload-socket\") pod \"e2e-shared-a-w2-a1d55d-openshift-default-6646f694ff-c6hjq\" (UID: \"d05765ac-da9e-4fad-b2e8-e26b8630849e\") " pod="openshift-ingress/e2e-shared-a-w2-a1d55d-openshift-default-6646f694ff-c6hjq"
Oct 30 19:03:21.455844 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:21.454723    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bhgr\" (UniqueName: \"kubernetes.io/projected/d05765ac-da9e-4fad-b2e8-e26b8630849e-kube-api-access-4bhgr\") pod \"e2e-shared-a-w2-a1d55d-openshift-default-6646f694ff-c6hjq\" (UID: \"d05765ac-da9e-4fad-b2e8-e26b8630849e\") " pod="openshift-ingress/e2e-shared-a-w2-a1d55d-openshift-default-6646f694ff-c6hjq"
Oct 30 19:03:21.455844 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:21.454834    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d05765ac-da9e-4fad-b2e8-e26b8630849e-istiod-ca-cert\") pod \"e2e-shared-a-w2-a1d55d-openshift-default-6646f694ff-c6hjq\" (UID: \"d05765ac-da9e-4fad-b2e8-e26b8630849e\") " pod="openshift-ingress/e2e-shared-a-w2-a1d55d-openshift-default-6646f694ff-c6hjq"
Oct 30 19:03:21.455844 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:21.454995    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a1d55d-openshift-default-6646f694ff-c6hjq"]
Oct 30 19:03:21.556751 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:21.556712    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d05765ac-da9e-4fad-b2e8-e26b8630849e-credential-socket\") pod \"e2e-shared-a-w2-a1d55d-openshift-default-6646f694ff-c6hjq\" (UID: \"d05765ac-da9e-4fad-b2e8-e26b8630849e\") " pod="openshift-ingress/e2e-shared-a-w2-a1d55d-openshift-default-6646f694ff-c6hjq"
Oct 30 19:03:21.556918 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:21.556817    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d05765ac-da9e-4fad-b2e8-e26b8630849e-istio-token\") pod \"e2e-shared-a-w2-a1d55d-openshift-default-6646f694ff-c6hjq\" (UID: \"d05765ac-da9e-4fad-b2e8-e26b8630849e\") " pod="openshift-ingress/e2e-shared-a-w2-a1d55d-openshift-default-6646f694ff-c6hjq"
Oct 30 19:03:21.556918 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:21.556871    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d05765ac-da9e-4fad-b2e8-e26b8630849e-workload-certs\") pod \"e2e-shared-a-w2-a1d55d-openshift-default-6646f694ff-c6hjq\" (UID: \"d05765ac-da9e-4fad-b2e8-e26b8630849e\") " pod="openshift-ingress/e2e-shared-a-w2-a1d55d-openshift-default-6646f694ff-c6hjq"
Oct 30 19:03:21.557001 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:21.556922    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d05765ac-da9e-4fad-b2e8-e26b8630849e-istio-podinfo\") pod \"e2e-shared-a-w2-a1d55d-openshift-default-6646f694ff-c6hjq\" (UID: \"d05765ac-da9e-4fad-b2e8-e26b8630849e\") " pod="openshift-ingress/e2e-shared-a-w2-a1d55d-openshift-default-6646f694ff-c6hjq"
Oct 30 19:03:21.557001 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:21.556954    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d05765ac-da9e-4fad-b2e8-e26b8630849e-workload-socket\") pod \"e2e-shared-a-w2-a1d55d-openshift-default-6646f694ff-c6hjq\" (UID: \"d05765ac-da9e-4fad-b2e8-e26b8630849e\") " pod="openshift-ingress/e2e-shared-a-w2-a1d55d-openshift-default-6646f694ff-c6hjq"
Oct 30 19:03:21.557067 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:21.557014    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d05765ac-da9e-4fad-b2e8-e26b8630849e-credential-socket\") pod \"e2e-shared-a-w2-a1d55d-openshift-default-6646f694ff-c6hjq\" (UID: \"d05765ac-da9e-4fad-b2e8-e26b8630849e\") " pod="openshift-ingress/e2e-shared-a-w2-a1d55d-openshift-default-6646f694ff-c6hjq"
Oct 30 19:03:21.557067 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:21.557035    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bhgr\" (UniqueName: \"kubernetes.io/projected/d05765ac-da9e-4fad-b2e8-e26b8630849e-kube-api-access-4bhgr\") pod \"e2e-shared-a-w2-a1d55d-openshift-default-6646f694ff-c6hjq\" (UID: \"d05765ac-da9e-4fad-b2e8-e26b8630849e\") " pod="openshift-ingress/e2e-shared-a-w2-a1d55d-openshift-default-6646f694ff-c6hjq"
Oct 30 19:03:21.557153 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:21.557121    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d05765ac-da9e-4fad-b2e8-e26b8630849e-istiod-ca-cert\") pod \"e2e-shared-a-w2-a1d55d-openshift-default-6646f694ff-c6hjq\" (UID: \"d05765ac-da9e-4fad-b2e8-e26b8630849e\") " pod="openshift-ingress/e2e-shared-a-w2-a1d55d-openshift-default-6646f694ff-c6hjq"
Oct 30 19:03:21.557203 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:21.557187    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d05765ac-da9e-4fad-b2e8-e26b8630849e-workload-certs\") pod \"e2e-shared-a-w2-a1d55d-openshift-default-6646f694ff-c6hjq\" (UID: \"d05765ac-da9e-4fad-b2e8-e26b8630849e\") " pod="openshift-ingress/e2e-shared-a-w2-a1d55d-openshift-default-6646f694ff-c6hjq"
Oct 30 19:03:21.557295 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:21.557283    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d05765ac-da9e-4fad-b2e8-e26b8630849e-istio-data\") pod \"e2e-shared-a-w2-a1d55d-openshift-default-6646f694ff-c6hjq\" (UID: \"d05765ac-da9e-4fad-b2e8-e26b8630849e\") " pod="openshift-ingress/e2e-shared-a-w2-a1d55d-openshift-default-6646f694ff-c6hjq"
Oct 30 19:03:21.557364 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:21.557354    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d05765ac-da9e-4fad-b2e8-e26b8630849e-istio-envoy\") pod \"e2e-shared-a-w2-a1d55d-openshift-default-6646f694ff-c6hjq\" (UID: \"d05765ac-da9e-4fad-b2e8-e26b8630849e\") " pod="openshift-ingress/e2e-shared-a-w2-a1d55d-openshift-default-6646f694ff-c6hjq"
Oct 30 19:03:21.558094 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:21.558064    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d05765ac-da9e-4fad-b2e8-e26b8630849e-istio-data\") pod \"e2e-shared-a-w2-a1d55d-openshift-default-6646f694ff-c6hjq\" (UID: \"d05765ac-da9e-4fad-b2e8-e26b8630849e\") " pod="openshift-ingress/e2e-shared-a-w2-a1d55d-openshift-default-6646f694ff-c6hjq"
Oct 30 19:03:21.558302 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:21.558276    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d05765ac-da9e-4fad-b2e8-e26b8630849e-workload-socket\") pod \"e2e-shared-a-w2-a1d55d-openshift-default-6646f694ff-c6hjq\" (UID: \"d05765ac-da9e-4fad-b2e8-e26b8630849e\") " pod="openshift-ingress/e2e-shared-a-w2-a1d55d-openshift-default-6646f694ff-c6hjq"
Oct 30 19:03:21.558628 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:21.558594    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d05765ac-da9e-4fad-b2e8-e26b8630849e-istiod-ca-cert\") pod \"e2e-shared-a-w2-a1d55d-openshift-default-6646f694ff-c6hjq\" (UID: \"d05765ac-da9e-4fad-b2e8-e26b8630849e\") " pod="openshift-ingress/e2e-shared-a-w2-a1d55d-openshift-default-6646f694ff-c6hjq"
Oct 30 19:03:21.560097 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:21.560074    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d05765ac-da9e-4fad-b2e8-e26b8630849e-istio-envoy\") pod \"e2e-shared-a-w2-a1d55d-openshift-default-6646f694ff-c6hjq\" (UID: \"d05765ac-da9e-4fad-b2e8-e26b8630849e\") " pod="openshift-ingress/e2e-shared-a-w2-a1d55d-openshift-default-6646f694ff-c6hjq"
Oct 30 19:03:21.560666 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:21.560650    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d05765ac-da9e-4fad-b2e8-e26b8630849e-istio-podinfo\") pod \"e2e-shared-a-w2-a1d55d-openshift-default-6646f694ff-c6hjq\" (UID: \"d05765ac-da9e-4fad-b2e8-e26b8630849e\") " pod="openshift-ingress/e2e-shared-a-w2-a1d55d-openshift-default-6646f694ff-c6hjq"
Oct 30 19:03:21.566464 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:21.566440    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d05765ac-da9e-4fad-b2e8-e26b8630849e-istio-token\") pod \"e2e-shared-a-w2-a1d55d-openshift-default-6646f694ff-c6hjq\" (UID: \"d05765ac-da9e-4fad-b2e8-e26b8630849e\") " pod="openshift-ingress/e2e-shared-a-w2-a1d55d-openshift-default-6646f694ff-c6hjq"
Oct 30 19:03:21.566588 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:21.566570    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bhgr\" (UniqueName: \"kubernetes.io/projected/d05765ac-da9e-4fad-b2e8-e26b8630849e-kube-api-access-4bhgr\") pod \"e2e-shared-a-w2-a1d55d-openshift-default-6646f694ff-c6hjq\" (UID: \"d05765ac-da9e-4fad-b2e8-e26b8630849e\") " pod="openshift-ingress/e2e-shared-a-w2-a1d55d-openshift-default-6646f694ff-c6hjq"
Oct 30 19:03:21.770016 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:21.769977    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a1d55d-openshift-default-6646f694ff-c6hjq"
Oct 30 19:03:22.294179 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:22.293725    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a1d55d-openshift-default-6646f694ff-c6hjq"]
Oct 30 19:03:22.302575 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:03:22.302528    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd05765ac_da9e_4fad_b2e8_e26b8630849e.slice/crio-d4c854c63e9fb74b6a4670da13938b913997d12619be5ca2e72372667ad4b0e8 WatchSource:0}: Error finding container d4c854c63e9fb74b6a4670da13938b913997d12619be5ca2e72372667ad4b0e8: Status 404 returned error can't find the container with id d4c854c63e9fb74b6a4670da13938b913997d12619be5ca2e72372667ad4b0e8
Oct 30 19:03:22.311924 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:22.311795    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:03:22.312071 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:22.311996    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:03:22.312156 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:22.312118    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:03:22.502883 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:22.492936    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2"
Oct 30 19:03:22.553427 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:22.535643    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72b6833a-13cf-4fe8-9b8d-26f5decea7f6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72b6833a-13cf-4fe8-9b8d-26f5decea7f6-istio-envoy\") pod \"72b6833a-13cf-4fe8-9b8d-26f5decea7f6\" (UID: \"72b6833a-13cf-4fe8-9b8d-26f5decea7f6\") "
Oct 30 19:03:22.553427 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:22.535760    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72b6833a-13cf-4fe8-9b8d-26f5decea7f6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72b6833a-13cf-4fe8-9b8d-26f5decea7f6-workload-socket\") pod \"72b6833a-13cf-4fe8-9b8d-26f5decea7f6\" (UID: \"72b6833a-13cf-4fe8-9b8d-26f5decea7f6\") "
Oct 30 19:03:22.553427 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:22.535813    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72b6833a-13cf-4fe8-9b8d-26f5decea7f6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72b6833a-13cf-4fe8-9b8d-26f5decea7f6-istiod-ca-cert\") pod \"72b6833a-13cf-4fe8-9b8d-26f5decea7f6\" (UID: \"72b6833a-13cf-4fe8-9b8d-26f5decea7f6\") "
Oct 30 19:03:22.553427 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:22.535953    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72b6833a-13cf-4fe8-9b8d-26f5decea7f6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72b6833a-13cf-4fe8-9b8d-26f5decea7f6-workload-certs\") pod \"72b6833a-13cf-4fe8-9b8d-26f5decea7f6\" (UID: \"72b6833a-13cf-4fe8-9b8d-26f5decea7f6\") "
Oct 30 19:03:22.553427 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:22.536017    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72b6833a-13cf-4fe8-9b8d-26f5decea7f6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72b6833a-13cf-4fe8-9b8d-26f5decea7f6-istio-data\") pod \"72b6833a-13cf-4fe8-9b8d-26f5decea7f6\" (UID: \"72b6833a-13cf-4fe8-9b8d-26f5decea7f6\") "
Oct 30 19:03:22.553427 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:22.536112    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72b6833a-13cf-4fe8-9b8d-26f5decea7f6-istio-token\" (UniqueName: \"kubernetes.io/projected/72b6833a-13cf-4fe8-9b8d-26f5decea7f6-istio-token\") pod \"72b6833a-13cf-4fe8-9b8d-26f5decea7f6\" (UID: \"72b6833a-13cf-4fe8-9b8d-26f5decea7f6\") "
Oct 30 19:03:22.553427 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:22.536142    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72b6833a-13cf-4fe8-9b8d-26f5decea7f6-kube-api-access-ldgfp\" (UniqueName: \"kubernetes.io/projected/72b6833a-13cf-4fe8-9b8d-26f5decea7f6-kube-api-access-ldgfp\") pod \"72b6833a-13cf-4fe8-9b8d-26f5decea7f6\" (UID: \"72b6833a-13cf-4fe8-9b8d-26f5decea7f6\") "
Oct 30 19:03:22.553427 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:22.536226    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72b6833a-13cf-4fe8-9b8d-26f5decea7f6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72b6833a-13cf-4fe8-9b8d-26f5decea7f6-credential-socket\") pod \"72b6833a-13cf-4fe8-9b8d-26f5decea7f6\" (UID: \"72b6833a-13cf-4fe8-9b8d-26f5decea7f6\") "
Oct 30 19:03:22.553427 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:22.536287    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/72b6833a-13cf-4fe8-9b8d-26f5decea7f6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72b6833a-13cf-4fe8-9b8d-26f5decea7f6-istio-podinfo\") pod \"72b6833a-13cf-4fe8-9b8d-26f5decea7f6\" (UID: \"72b6833a-13cf-4fe8-9b8d-26f5decea7f6\") "
Oct 30 19:03:22.553427 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:22.548846    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72b6833a-13cf-4fe8-9b8d-26f5decea7f6-credential-socket" pod "72b6833a-13cf-4fe8-9b8d-26f5decea7f6" (UID: "72b6833a-13cf-4fe8-9b8d-26f5decea7f6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:03:22.553427 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:22.550973    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72b6833a-13cf-4fe8-9b8d-26f5decea7f6-istiod-ca-cert" pod "72b6833a-13cf-4fe8-9b8d-26f5decea7f6" (UID: "72b6833a-13cf-4fe8-9b8d-26f5decea7f6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:03:22.553427 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:22.551032    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72b6833a-13cf-4fe8-9b8d-26f5decea7f6-istio-data" pod "72b6833a-13cf-4fe8-9b8d-26f5decea7f6" (UID: "72b6833a-13cf-4fe8-9b8d-26f5decea7f6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:03:22.579468 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:22.555833    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72b6833a-13cf-4fe8-9b8d-26f5decea7f6-workload-socket" pod "72b6833a-13cf-4fe8-9b8d-26f5decea7f6" (UID: "72b6833a-13cf-4fe8-9b8d-26f5decea7f6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:03:22.579468 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:22.555933    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72b6833a-13cf-4fe8-9b8d-26f5decea7f6-workload-certs" pod "72b6833a-13cf-4fe8-9b8d-26f5decea7f6" (UID: "72b6833a-13cf-4fe8-9b8d-26f5decea7f6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:03:22.690733 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:22.668366    4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72b6833a-13cf-4fe8-9b8d-26f5decea7f6-workload-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:03:22.690733 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:22.668403    4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72b6833a-13cf-4fe8-9b8d-26f5decea7f6-istiod-ca-cert\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:03:22.690733 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:22.668419    4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72b6833a-13cf-4fe8-9b8d-26f5decea7f6-workload-certs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:03:22.690733 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:22.668433    4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72b6833a-13cf-4fe8-9b8d-26f5decea7f6-istio-data\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:03:22.690733 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:22.668446    4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72b6833a-13cf-4fe8-9b8d-26f5decea7f6-credential-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:03:22.707946 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:22.707882    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72b6833a-13cf-4fe8-9b8d-26f5decea7f6-istio-envoy" pod "72b6833a-13cf-4fe8-9b8d-26f5decea7f6" (UID: "72b6833a-13cf-4fe8-9b8d-26f5decea7f6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:03:22.734915 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:22.731007    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72b6833a-13cf-4fe8-9b8d-26f5decea7f6-istio-token" pod "72b6833a-13cf-4fe8-9b8d-26f5decea7f6" (UID: "72b6833a-13cf-4fe8-9b8d-26f5decea7f6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:03:22.734915 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:22.731116    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72b6833a-13cf-4fe8-9b8d-26f5decea7f6-kube-api-access-ldgfp" pod "72b6833a-13cf-4fe8-9b8d-26f5decea7f6" (UID: "72b6833a-13cf-4fe8-9b8d-26f5decea7f6"). InnerVolumeSpecName "kube-api-access-ldgfp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:03:22.734915 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:22.732326    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/72b6833a-13cf-4fe8-9b8d-26f5decea7f6-istio-podinfo" pod "72b6833a-13cf-4fe8-9b8d-26f5decea7f6" (UID: "72b6833a-13cf-4fe8-9b8d-26f5decea7f6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:03:22.771504 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:22.770062    4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72b6833a-13cf-4fe8-9b8d-26f5decea7f6-istio-envoy\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:03:22.771504 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:22.770100    4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72b6833a-13cf-4fe8-9b8d-26f5decea7f6-istio-token\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:03:22.771504 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:22.770115    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldgfp\" (UniqueName: \"kubernetes.io/projected/72b6833a-13cf-4fe8-9b8d-26f5decea7f6-kube-api-access-ldgfp\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:03:22.771504 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:22.770131    4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72b6833a-13cf-4fe8-9b8d-26f5decea7f6-istio-podinfo\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:03:23.247908 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:23.247715    4525 generic.go:356] "Generic (PLEG): container finished" podID="72b6833a-13cf-4fe8-9b8d-26f5decea7f6" containerID="43f2859aefd0804546efa7b62f01106635812344a55c17fea614a262775e3519" exitCode=0
Oct 30 19:03:23.247908 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:23.247773    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2"
Oct 30 19:03:23.247908 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:23.247762    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2" event={"ID":"72b6833a-13cf-4fe8-9b8d-26f5decea7f6","Type":"ContainerDied","Data":"43f2859aefd0804546efa7b62f01106635812344a55c17fea614a262775e3519"}
Oct 30 19:03:23.247908 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:23.247858    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2" event={"ID":"72b6833a-13cf-4fe8-9b8d-26f5decea7f6","Type":"ContainerDied","Data":"6f3189413c195fc986a19abfa6e22f949197cfb92653f5473d11cff6bb312221"}
Oct 30 19:03:23.247908 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:23.247896    4525 scope.go:122] "RemoveContainer" containerID="43f2859aefd0804546efa7b62f01106635812344a55c17fea614a262775e3519"
Oct 30 19:03:23.252663 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:23.252050    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a1d55d-openshift-default-6646f694ff-c6hjq" event={"ID":"d05765ac-da9e-4fad-b2e8-e26b8630849e","Type":"ContainerStarted","Data":"d6768cbe6b7d423925def8357d00c26ab069b4c0fc1c95b03abdd16f7875e590"}
Oct 30 19:03:23.252663 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:23.252108    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a1d55d-openshift-default-6646f694ff-c6hjq" event={"ID":"d05765ac-da9e-4fad-b2e8-e26b8630849e","Type":"ContainerStarted","Data":"d4c854c63e9fb74b6a4670da13938b913997d12619be5ca2e72372667ad4b0e8"}
Oct 30 19:03:23.265179 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:23.265158    4525 scope.go:122] "RemoveContainer" containerID="43f2859aefd0804546efa7b62f01106635812344a55c17fea614a262775e3519"
Oct 30 19:03:23.265502 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:03:23.265471    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43f2859aefd0804546efa7b62f01106635812344a55c17fea614a262775e3519\": container with ID starting with 43f2859aefd0804546efa7b62f01106635812344a55c17fea614a262775e3519 not found: ID does not exist" containerID="43f2859aefd0804546efa7b62f01106635812344a55c17fea614a262775e3519"
Oct 30 19:03:23.265604 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:23.265515    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43f2859aefd0804546efa7b62f01106635812344a55c17fea614a262775e3519"} err="failed to get container status \"43f2859aefd0804546efa7b62f01106635812344a55c17fea614a262775e3519\": rpc error: code = NotFound desc = could not find container \"43f2859aefd0804546efa7b62f01106635812344a55c17fea614a262775e3519\": container with ID starting with 43f2859aefd0804546efa7b62f01106635812344a55c17fea614a262775e3519 not found: ID does not exist"
Oct 30 19:03:23.281367 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:23.280964    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-a1d55d-openshift-default-6646f694ff-c6hjq" podStartSLOduration=2.28094775 podStartE2EDuration="2.28094775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:03:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:03:23.276891299 +0000 UTC m=+1625.902561264" watchObservedRunningTime="2026-10-30 19:03:23.28094775 +0000 UTC m=+1625.906617731"
Oct 30 19:03:23.308067 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:23.308030    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2"]
Oct 30 19:03:23.316817 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:23.316777    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-lk4g2"]
Oct 30 19:03:23.548767 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:23.548664    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72b6833a-13cf-4fe8-9b8d-26f5decea7f6" path="/var/lib/kubelet/pods/72b6833a-13cf-4fe8-9b8d-26f5decea7f6/volumes"
Oct 30 19:03:23.807764 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:23.806573    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-a1d55d-openshift-default-6646f694ff-c6hjq"
Oct 30 19:03:23.807764 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:23.806783    4525 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a1d55d-openshift-default-6646f694ff-c6hjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body=
Oct 30 19:03:23.807764 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:23.806847    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a1d55d-openshift-default-6646f694ff-c6hjq" podUID="d05765ac-da9e-4fad-b2e8-e26b8630849e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused"
Oct 30 19:03:23.860435 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:23.860388    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-baa504/auth-test-model-baa504-kserve-64c89767d5-d4pq4"]
Oct 30 19:03:23.861622 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:23.861598    4525 cpu_manager.go:414] "containerMap: removing container" podUID="72b6833a-13cf-4fe8-9b8d-26f5decea7f6" containerName="istio-proxy"
Oct 30 19:03:23.861830 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:23.861816    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72b6833a-13cf-4fe8-9b8d-26f5decea7f6" containerName="istio-proxy"
Oct 30 19:03:23.865892 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:23.862163    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="72b6833a-13cf-4fe8-9b8d-26f5decea7f6" containerName="istio-proxy"
Oct 30 19:03:23.873435 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:23.873400    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-baa504/auth-test-model-baa504-kserve-64c89767d5-d4pq4"]
Oct 30 19:03:23.873617 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:23.873598    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-baa504/auth-test-model-baa504-kserve-64c89767d5-d4pq4"
Oct 30 19:03:23.901362 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:23.901330    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-baa504\"/\"default-dockercfg-22gq7\""
Oct 30 19:03:23.902496 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:23.902470    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-baa504\"/\"kube-root-ca.crt\""
Oct 30 19:03:23.922058 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:23.914009    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-baa504\"/\"auth-test-model-baa504-kserve-self-signed-certs\""
Oct 30 19:03:23.922058 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:23.914106    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-baa504\"/\"openshift-service-ca.crt\""
Oct 30 19:03:24.012860 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:24.012823    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2300b922-2ef9-4c20-8c7d-df89a055374c-tls-certs\") pod \"auth-test-model-baa504-kserve-64c89767d5-d4pq4\" (UID: \"2300b922-2ef9-4c20-8c7d-df89a055374c\") " pod="ai-tenant-e2e-shared-a-w0-baa504/auth-test-model-baa504-kserve-64c89767d5-d4pq4"
Oct 30 19:03:24.013065 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:24.012915    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2300b922-2ef9-4c20-8c7d-df89a055374c-model-cache\") pod \"auth-test-model-baa504-kserve-64c89767d5-d4pq4\" (UID: \"2300b922-2ef9-4c20-8c7d-df89a055374c\") " pod="ai-tenant-e2e-shared-a-w0-baa504/auth-test-model-baa504-kserve-64c89767d5-d4pq4"
Oct 30 19:03:24.013065 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:24.012952    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2300b922-2ef9-4c20-8c7d-df89a055374c-tmp-dir\") pod \"auth-test-model-baa504-kserve-64c89767d5-d4pq4\" (UID: \"2300b922-2ef9-4c20-8c7d-df89a055374c\") " pod="ai-tenant-e2e-shared-a-w0-baa504/auth-test-model-baa504-kserve-64c89767d5-d4pq4"
Oct 30 19:03:24.013065 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:24.013041    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2300b922-2ef9-4c20-8c7d-df89a055374c-home\") pod \"auth-test-model-baa504-kserve-64c89767d5-d4pq4\" (UID: \"2300b922-2ef9-4c20-8c7d-df89a055374c\") " pod="ai-tenant-e2e-shared-a-w0-baa504/auth-test-model-baa504-kserve-64c89767d5-d4pq4"
Oct 30 19:03:24.013223 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:24.013092    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2300b922-2ef9-4c20-8c7d-df89a055374c-dshm\") pod \"auth-test-model-baa504-kserve-64c89767d5-d4pq4\" (UID: \"2300b922-2ef9-4c20-8c7d-df89a055374c\") " pod="ai-tenant-e2e-shared-a-w0-baa504/auth-test-model-baa504-kserve-64c89767d5-d4pq4"
Oct 30 19:03:24.013328 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:24.013306    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zcvf\" (UniqueName: \"kubernetes.io/projected/2300b922-2ef9-4c20-8c7d-df89a055374c-kube-api-access-8zcvf\") pod \"auth-test-model-baa504-kserve-64c89767d5-d4pq4\" (UID: \"2300b922-2ef9-4c20-8c7d-df89a055374c\") " pod="ai-tenant-e2e-shared-a-w0-baa504/auth-test-model-baa504-kserve-64c89767d5-d4pq4"
Oct 30 19:03:24.126093 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:24.125742    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zcvf\" (UniqueName: \"kubernetes.io/projected/2300b922-2ef9-4c20-8c7d-df89a055374c-kube-api-access-8zcvf\") pod \"auth-test-model-baa504-kserve-64c89767d5-d4pq4\" (UID: \"2300b922-2ef9-4c20-8c7d-df89a055374c\") " pod="ai-tenant-e2e-shared-a-w0-baa504/auth-test-model-baa504-kserve-64c89767d5-d4pq4"
Oct 30 19:03:24.126093 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:24.125956    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2300b922-2ef9-4c20-8c7d-df89a055374c-tls-certs\") pod \"auth-test-model-baa504-kserve-64c89767d5-d4pq4\" (UID: \"2300b922-2ef9-4c20-8c7d-df89a055374c\") " pod="ai-tenant-e2e-shared-a-w0-baa504/auth-test-model-baa504-kserve-64c89767d5-d4pq4"
Oct 30 19:03:24.126093 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:24.126035    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2300b922-2ef9-4c20-8c7d-df89a055374c-model-cache\") pod \"auth-test-model-baa504-kserve-64c89767d5-d4pq4\" (UID: \"2300b922-2ef9-4c20-8c7d-df89a055374c\") " pod="ai-tenant-e2e-shared-a-w0-baa504/auth-test-model-baa504-kserve-64c89767d5-d4pq4"
Oct 30 19:03:24.126093 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:24.126078    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2300b922-2ef9-4c20-8c7d-df89a055374c-tmp-dir\") pod \"auth-test-model-baa504-kserve-64c89767d5-d4pq4\" (UID: \"2300b922-2ef9-4c20-8c7d-df89a055374c\") " pod="ai-tenant-e2e-shared-a-w0-baa504/auth-test-model-baa504-kserve-64c89767d5-d4pq4"
Oct 30 19:03:24.126456 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:24.126158    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2300b922-2ef9-4c20-8c7d-df89a055374c-home\") pod \"auth-test-model-baa504-kserve-64c89767d5-d4pq4\" (UID: \"2300b922-2ef9-4c20-8c7d-df89a055374c\") " pod="ai-tenant-e2e-shared-a-w0-baa504/auth-test-model-baa504-kserve-64c89767d5-d4pq4"
Oct 30 19:03:24.126456 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:24.126200    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2300b922-2ef9-4c20-8c7d-df89a055374c-dshm\") pod \"auth-test-model-baa504-kserve-64c89767d5-d4pq4\" (UID: \"2300b922-2ef9-4c20-8c7d-df89a055374c\") " pod="ai-tenant-e2e-shared-a-w0-baa504/auth-test-model-baa504-kserve-64c89767d5-d4pq4"
Oct 30 19:03:24.130317 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:24.127149    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2300b922-2ef9-4c20-8c7d-df89a055374c-model-cache\") pod \"auth-test-model-baa504-kserve-64c89767d5-d4pq4\" (UID: \"2300b922-2ef9-4c20-8c7d-df89a055374c\") " pod="ai-tenant-e2e-shared-a-w0-baa504/auth-test-model-baa504-kserve-64c89767d5-d4pq4"
Oct 30 19:03:24.130317 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:24.128100    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2300b922-2ef9-4c20-8c7d-df89a055374c-tmp-dir\") pod \"auth-test-model-baa504-kserve-64c89767d5-d4pq4\" (UID: \"2300b922-2ef9-4c20-8c7d-df89a055374c\") " pod="ai-tenant-e2e-shared-a-w0-baa504/auth-test-model-baa504-kserve-64c89767d5-d4pq4"
Oct 30 19:03:24.130317 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:24.128336    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2300b922-2ef9-4c20-8c7d-df89a055374c-home\") pod \"auth-test-model-baa504-kserve-64c89767d5-d4pq4\" (UID: \"2300b922-2ef9-4c20-8c7d-df89a055374c\") " pod="ai-tenant-e2e-shared-a-w0-baa504/auth-test-model-baa504-kserve-64c89767d5-d4pq4"
Oct 30 19:03:24.169366 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:24.143356    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2300b922-2ef9-4c20-8c7d-df89a055374c-dshm\") pod \"auth-test-model-baa504-kserve-64c89767d5-d4pq4\" (UID: \"2300b922-2ef9-4c20-8c7d-df89a055374c\") " pod="ai-tenant-e2e-shared-a-w0-baa504/auth-test-model-baa504-kserve-64c89767d5-d4pq4"
Oct 30 19:03:24.179714 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:24.173046    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2300b922-2ef9-4c20-8c7d-df89a055374c-tls-certs\") pod \"auth-test-model-baa504-kserve-64c89767d5-d4pq4\" (UID: \"2300b922-2ef9-4c20-8c7d-df89a055374c\") " pod="ai-tenant-e2e-shared-a-w0-baa504/auth-test-model-baa504-kserve-64c89767d5-d4pq4"
Oct 30 19:03:24.190756 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:24.181238    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zcvf\" (UniqueName: \"kubernetes.io/projected/2300b922-2ef9-4c20-8c7d-df89a055374c-kube-api-access-8zcvf\") pod \"auth-test-model-baa504-kserve-64c89767d5-d4pq4\" (UID: \"2300b922-2ef9-4c20-8c7d-df89a055374c\") " pod="ai-tenant-e2e-shared-a-w0-baa504/auth-test-model-baa504-kserve-64c89767d5-d4pq4"
Oct 30 19:03:24.239321 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:24.238985    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-baa504/auth-test-model-baa504-kserve-64c89767d5-d4pq4"
Oct 30 19:03:24.768346 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:24.768258    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-baa504/auth-test-model-baa504-kserve-64c89767d5-d4pq4"]
Oct 30 19:03:24.809302 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:24.809247    4525 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a1d55d-openshift-default-6646f694ff-c6hjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body=
Oct 30 19:03:24.809467 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:24.809313    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a1d55d-openshift-default-6646f694ff-c6hjq" podUID="d05765ac-da9e-4fad-b2e8-e26b8630849e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused"
Oct 30 19:03:25.131894 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:25.131845    4525 patch_prober.go:29] interesting pod/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:03:25.132064 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:25.131925    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh" podUID="8b37419e-12f0-4e1e-8b2a-48c39c4c9e99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:03:25.338413 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:25.338369    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-baa504/auth-test-model-baa504-kserve-64c89767d5-d4pq4" event={"ID":"2300b922-2ef9-4c20-8c7d-df89a055374c","Type":"ContainerStarted","Data":"a30a587db8cb8888869d9865080e274038daa33ec39a4edfc514961887edf430"}
Oct 30 19:03:25.338617 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:25.338426    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-baa504/auth-test-model-baa504-kserve-64c89767d5-d4pq4" event={"ID":"2300b922-2ef9-4c20-8c7d-df89a055374c","Type":"ContainerStarted","Data":"949d824b2d5271f22ff092e162ecd6c45ba5cc38ba2ef9ddb8034c6e8b9dea9e"}
Oct 30 19:03:25.363019 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:25.362951    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-baa504/auth-test-model-baa504-kserve-64c89767d5-d4pq4" podStartSLOduration=2.362929517 podStartE2EDuration="2.362929517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:03:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:03:25.354495864 +0000 UTC m=+1627.980165827" watchObservedRunningTime="2026-10-30 19:03:25.362929517 +0000 UTC m=+1627.988599482"
Oct 30 19:03:25.803514 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:25.803468    4525 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a1d55d-openshift-default-6646f694ff-c6hjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body=
Oct 30 19:03:25.804075 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:25.803538    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a1d55d-openshift-default-6646f694ff-c6hjq" podUID="d05765ac-da9e-4fad-b2e8-e26b8630849e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused"
Oct 30 19:03:26.240492 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:26.240452    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-baa504/auth-test-model-baa504-kserve-64c89767d5-d4pq4"
Oct 30 19:03:26.805630 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:26.804370    4525 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a1d55d-openshift-default-6646f694ff-c6hjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body=
Oct 30 19:03:26.805630 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:26.804450    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a1d55d-openshift-default-6646f694ff-c6hjq" podUID="d05765ac-da9e-4fad-b2e8-e26b8630849e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused"
Oct 30 19:03:27.805991 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:27.805939    4525 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a1d55d-openshift-default-6646f694ff-c6hjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body=
Oct 30 19:03:27.806514 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:27.806008    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a1d55d-openshift-default-6646f694ff-c6hjq" podUID="d05765ac-da9e-4fad-b2e8-e26b8630849e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused"
Oct 30 19:03:28.669044 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:28.395627    4525 generic.go:356] "Generic (PLEG): container finished" podID="8b37419e-12f0-4e1e-8b2a-48c39c4c9e99" containerID="61529cb0c57fb81b6f1ff254514dbb0a5ae3aa1ce6910eacfc8b06288ce15574" exitCode=0
Oct 30 19:03:28.669044 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:28.400063    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh" event={"ID":"8b37419e-12f0-4e1e-8b2a-48c39c4c9e99","Type":"ContainerDied","Data":"61529cb0c57fb81b6f1ff254514dbb0a5ae3aa1ce6910eacfc8b06288ce15574"}
Oct 30 19:03:28.669044 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:28.400700    4525 scope.go:122] "RemoveContainer" containerID="61529cb0c57fb81b6f1ff254514dbb0a5ae3aa1ce6910eacfc8b06288ce15574"
Oct 30 19:03:28.669044 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:28.407874    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:03:28.669044 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:28.407965    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:03:28.669044 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:28.408012    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:03:28.845548 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:28.844555    4525 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a1d55d-openshift-default-6646f694ff-c6hjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body=
Oct 30 19:03:28.845548 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:28.844616    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a1d55d-openshift-default-6646f694ff-c6hjq" podUID="d05765ac-da9e-4fad-b2e8-e26b8630849e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused"
Oct 30 19:03:29.425866 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:29.425831    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh" event={"ID":"8b37419e-12f0-4e1e-8b2a-48c39c4c9e99","Type":"ContainerStarted","Data":"1eac631cc83233ff18fd8b0a52d9b7d6d2f6c362b9577146208fd2b742196dac"}
Oct 30 19:03:29.427957 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:29.427932    4525 generic.go:356] "Generic (PLEG): container finished" podID="19f00f47-605d-4b19-9a1c-664de3661507" containerID="05032f3ae4130fe35addd2d7e6c00f8ff2d86f4825b9a8764db98d5829ad5434" exitCode=0
Oct 30 19:03:29.428107 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:29.427976    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-68b466-openshift-default-57cb5457fc-jgkhq" event={"ID":"19f00f47-605d-4b19-9a1c-664de3661507","Type":"ContainerDied","Data":"05032f3ae4130fe35addd2d7e6c00f8ff2d86f4825b9a8764db98d5829ad5434"}
Oct 30 19:03:29.428375 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:29.428359    4525 scope.go:122] "RemoveContainer" containerID="05032f3ae4130fe35addd2d7e6c00f8ff2d86f4825b9a8764db98d5829ad5434"
Oct 30 19:03:29.430401 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:29.430368    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:03:29.430499 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:29.430455    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:03:29.430540 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:29.430529    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:03:29.806771 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:29.806157    4525 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a1d55d-openshift-default-6646f694ff-c6hjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body=
Oct 30 19:03:29.806771 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:29.806226    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a1d55d-openshift-default-6646f694ff-c6hjq" podUID="d05765ac-da9e-4fad-b2e8-e26b8630849e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused"
Oct 30 19:03:30.039244 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:30.039200    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh"
Oct 30 19:03:30.039807 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:30.039410    4525 patch_prober.go:29] interesting pod/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Oct 30 19:03:30.039807 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:30.039469    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh" podUID="8b37419e-12f0-4e1e-8b2a-48c39c4c9e99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Oct 30 19:03:30.238583 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:30.236797    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-229cp"]
Oct 30 19:03:30.258711 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:30.256148    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-229cp"]
Oct 30 19:03:30.258711 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:30.256313    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-229cp"
Oct 30 19:03:30.346850 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:30.345632    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d9a2656-6bae-4277-a7f3-b43e8143dbd7-istio-token\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-229cp\" (UID: \"5d9a2656-6bae-4277-a7f3-b43e8143dbd7\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-229cp"
Oct 30 19:03:30.346850 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:30.345752    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d9a2656-6bae-4277-a7f3-b43e8143dbd7-istio-podinfo\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-229cp\" (UID: \"5d9a2656-6bae-4277-a7f3-b43e8143dbd7\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-229cp"
Oct 30 19:03:30.346850 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:30.345833    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d9a2656-6bae-4277-a7f3-b43e8143dbd7-istio-envoy\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-229cp\" (UID: \"5d9a2656-6bae-4277-a7f3-b43e8143dbd7\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-229cp"
Oct 30 19:03:30.346850 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:30.345919    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d9a2656-6bae-4277-a7f3-b43e8143dbd7-workload-certs\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-229cp\" (UID: \"5d9a2656-6bae-4277-a7f3-b43e8143dbd7\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-229cp"
Oct 30 19:03:30.346850 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:30.345952    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d9a2656-6bae-4277-a7f3-b43e8143dbd7-istiod-ca-cert\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-229cp\" (UID: \"5d9a2656-6bae-4277-a7f3-b43e8143dbd7\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-229cp"
Oct 30 19:03:30.346850 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:30.346009    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d9a2656-6bae-4277-a7f3-b43e8143dbd7-workload-socket\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-229cp\" (UID: \"5d9a2656-6bae-4277-a7f3-b43e8143dbd7\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-229cp"
Oct 30 19:03:30.346850 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:30.346064    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d9a2656-6bae-4277-a7f3-b43e8143dbd7-credential-socket\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-229cp\" (UID: \"5d9a2656-6bae-4277-a7f3-b43e8143dbd7\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-229cp"
Oct 30 19:03:30.346850 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:30.346183    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d9a2656-6bae-4277-a7f3-b43e8143dbd7-istio-data\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-229cp\" (UID: \"5d9a2656-6bae-4277-a7f3-b43e8143dbd7\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-229cp"
Oct 30 19:03:30.346850 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:30.346236    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bf66\" (UniqueName: \"kubernetes.io/projected/5d9a2656-6bae-4277-a7f3-b43e8143dbd7-kube-api-access-2bf66\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-229cp\" (UID: \"5d9a2656-6bae-4277-a7f3-b43e8143dbd7\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-229cp"
Oct 30 19:03:30.534729 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:30.471731    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d9a2656-6bae-4277-a7f3-b43e8143dbd7-istio-data\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-229cp\" (UID: \"5d9a2656-6bae-4277-a7f3-b43e8143dbd7\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-229cp"
Oct 30 19:03:30.534729 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:30.471810    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bf66\" (UniqueName: \"kubernetes.io/projected/5d9a2656-6bae-4277-a7f3-b43e8143dbd7-kube-api-access-2bf66\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-229cp\" (UID: \"5d9a2656-6bae-4277-a7f3-b43e8143dbd7\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-229cp"
Oct 30 19:03:30.534729 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:30.471986    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d9a2656-6bae-4277-a7f3-b43e8143dbd7-istio-token\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-229cp\" (UID: \"5d9a2656-6bae-4277-a7f3-b43e8143dbd7\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-229cp"
Oct 30 19:03:30.534729 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:30.472049    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d9a2656-6bae-4277-a7f3-b43e8143dbd7-istio-podinfo\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-229cp\" (UID: \"5d9a2656-6bae-4277-a7f3-b43e8143dbd7\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-229cp"
Oct 30 19:03:30.534729 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:30.472119    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d9a2656-6bae-4277-a7f3-b43e8143dbd7-istio-envoy\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-229cp\" (UID: \"5d9a2656-6bae-4277-a7f3-b43e8143dbd7\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-229cp"
Oct 30 19:03:30.534729 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:30.472210    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d9a2656-6bae-4277-a7f3-b43e8143dbd7-workload-certs\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-229cp\" (UID: \"5d9a2656-6bae-4277-a7f3-b43e8143dbd7\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-229cp"
Oct 30 19:03:30.534729 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:30.472239    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d9a2656-6bae-4277-a7f3-b43e8143dbd7-istiod-ca-cert\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-229cp\" (UID: \"5d9a2656-6bae-4277-a7f3-b43e8143dbd7\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-229cp"
Oct 30 19:03:30.534729 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:30.472275    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d9a2656-6bae-4277-a7f3-b43e8143dbd7-workload-socket\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-229cp\" (UID: \"5d9a2656-6bae-4277-a7f3-b43e8143dbd7\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-229cp"
Oct 30 19:03:30.534729 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:30.472319    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d9a2656-6bae-4277-a7f3-b43e8143dbd7-credential-socket\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-229cp\" (UID: \"5d9a2656-6bae-4277-a7f3-b43e8143dbd7\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-229cp"
Oct 30 19:03:30.534729 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:30.473050    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d9a2656-6bae-4277-a7f3-b43e8143dbd7-credential-socket\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-229cp\" (UID: \"5d9a2656-6bae-4277-a7f3-b43e8143dbd7\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-229cp"
Oct 30 19:03:30.534729 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:30.491163    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d9a2656-6bae-4277-a7f3-b43e8143dbd7-istio-data\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-229cp\" (UID: \"5d9a2656-6bae-4277-a7f3-b43e8143dbd7\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-229cp"
Oct 30 19:03:30.534729 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:30.492266    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d9a2656-6bae-4277-a7f3-b43e8143dbd7-istio-podinfo\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-229cp\" (UID: \"5d9a2656-6bae-4277-a7f3-b43e8143dbd7\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-229cp"
Oct 30 19:03:30.534729 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:30.493287    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d9a2656-6bae-4277-a7f3-b43e8143dbd7-istiod-ca-cert\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-229cp\" (UID: \"5d9a2656-6bae-4277-a7f3-b43e8143dbd7\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-229cp"
Oct 30 19:03:30.534729 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:30.493564    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d9a2656-6bae-4277-a7f3-b43e8143dbd7-workload-certs\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-229cp\" (UID: \"5d9a2656-6bae-4277-a7f3-b43e8143dbd7\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-229cp"
Oct 30 19:03:30.534729 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:30.493849    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d9a2656-6bae-4277-a7f3-b43e8143dbd7-workload-socket\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-229cp\" (UID: \"5d9a2656-6bae-4277-a7f3-b43e8143dbd7\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-229cp"
Oct 30 19:03:30.552719 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:30.549882    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d9a2656-6bae-4277-a7f3-b43e8143dbd7-istio-envoy\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-229cp\" (UID: \"5d9a2656-6bae-4277-a7f3-b43e8143dbd7\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-229cp"
Oct 30 19:03:30.552719 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:30.550498    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d9a2656-6bae-4277-a7f3-b43e8143dbd7-istio-token\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-229cp\" (UID: \"5d9a2656-6bae-4277-a7f3-b43e8143dbd7\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-229cp"
Oct 30 19:03:30.552719 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:30.552113    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bf66\" (UniqueName: \"kubernetes.io/projected/5d9a2656-6bae-4277-a7f3-b43e8143dbd7-kube-api-access-2bf66\") pod \"e2e-worker-w1-85cb41-openshift-default-74bfbc4465-229cp\" (UID: \"5d9a2656-6bae-4277-a7f3-b43e8143dbd7\") " pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-229cp"
Oct 30 19:03:30.560719 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:30.560650    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-68b466-openshift-default-57cb5457fc-jgkhq" event={"ID":"19f00f47-605d-4b19-9a1c-664de3661507","Type":"ContainerStarted","Data":"3f123adeefacee438ae8ecf734e1bd9a72367e41eb105cf70ca97d5e30425608"}
Oct 30 19:03:30.628712 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:30.628248    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-229cp"
Oct 30 19:03:30.844917 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:30.844389    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-a1d55d-openshift-default-6646f694ff-c6hjq"
Oct 30 19:03:30.845407 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:30.845207    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-a1d55d-openshift-default-6646f694ff-c6hjq"
Oct 30 19:03:30.850623 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:30.848181    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-a1d55d-openshift-default-6646f694ff-c6hjq"
Oct 30 19:03:31.049753 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:31.046107    4525 patch_prober.go:29] interesting pod/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Oct 30 19:03:31.049753 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:31.046175    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh" podUID="8b37419e-12f0-4e1e-8b2a-48c39c4c9e99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Oct 30 19:03:31.049753 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:31.047655    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-68b466-openshift-default-57cb5457fc-jgkhq"
Oct 30 19:03:31.049753 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:31.047783    4525 patch_prober.go:29] interesting pod/e2e-shared-b-w0-68b466-openshift-default-57cb5457fc-jgkhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body=
Oct 30 19:03:31.049753 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:31.047832    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-68b466-openshift-default-57cb5457fc-jgkhq" podUID="19f00f47-605d-4b19-9a1c-664de3661507" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused"
Oct 30 19:03:31.052488 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:31.052458    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-229cp"]
Oct 30 19:03:31.056349 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:03:31.056324    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d9a2656_6bae_4277_a7f3_b43e8143dbd7.slice/crio-be3c9abead7112b6b27a4ebe0645812ce49b73d1ac27f6fdbed82fe2754470ba WatchSource:0}: Error finding container be3c9abead7112b6b27a4ebe0645812ce49b73d1ac27f6fdbed82fe2754470ba: Status 404 returned error can't find the container with id be3c9abead7112b6b27a4ebe0645812ce49b73d1ac27f6fdbed82fe2754470ba
Oct 30 19:03:31.059539 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:31.059510    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:03:31.059613 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:31.059577    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:03:31.059613 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:31.059609    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:03:31.570734 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:31.570636    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-229cp" event={"ID":"5d9a2656-6bae-4277-a7f3-b43e8143dbd7","Type":"ContainerStarted","Data":"7396b8db18338af05d47281d1cfb35fe280b6afbe7bd7a122a987ac114fa04f7"}
Oct 30 19:03:31.571040 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:31.570786    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-229cp" event={"ID":"5d9a2656-6bae-4277-a7f3-b43e8143dbd7","Type":"ContainerStarted","Data":"be3c9abead7112b6b27a4ebe0645812ce49b73d1ac27f6fdbed82fe2754470ba"}
Oct 30 19:03:31.638511 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:31.637804    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-229cp"
Oct 30 19:03:31.994709 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:31.994177    4525 patch_prober.go:29] interesting pod/e2e-shared-b-w0-68b466-openshift-default-57cb5457fc-jgkhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body=
Oct 30 19:03:31.994709 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:31.994239    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-68b466-openshift-default-57cb5457fc-jgkhq" podUID="19f00f47-605d-4b19-9a1c-664de3661507" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused"
Oct 30 19:03:32.041708 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:32.040926    4525 patch_prober.go:29] interesting pod/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Oct 30 19:03:32.041708 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:32.041002    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh" podUID="8b37419e-12f0-4e1e-8b2a-48c39c4c9e99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Oct 30 19:03:32.632718 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:32.632446    4525 patch_prober.go:29] interesting pod/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-229cp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body=
Oct 30 19:03:32.632718 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:32.632507    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-229cp" podUID="5d9a2656-6bae-4277-a7f3-b43e8143dbd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused"
Oct 30 19:03:32.999770 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:32.998718    4525 patch_prober.go:29] interesting pod/e2e-shared-b-w0-68b466-openshift-default-57cb5457fc-jgkhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body=
Oct 30 19:03:32.999770 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:32.998792    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-68b466-openshift-default-57cb5457fc-jgkhq" podUID="19f00f47-605d-4b19-9a1c-664de3661507" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused"
Oct 30 19:03:33.043708 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:33.043544    4525 patch_prober.go:29] interesting pod/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Oct 30 19:03:33.043708 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:33.043619    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh" podUID="8b37419e-12f0-4e1e-8b2a-48c39c4c9e99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Oct 30 19:03:33.258747 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:33.257830    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-68b466-openshift-default-57cb5457fc-jgkhq"
Oct 30 19:03:33.664014 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:33.663968    4525 patch_prober.go:29] interesting pod/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-229cp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body=
Oct 30 19:03:33.664553 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:33.664035    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-229cp" podUID="5d9a2656-6bae-4277-a7f3-b43e8143dbd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused"
Oct 30 19:03:33.999773 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:33.996047    4525 patch_prober.go:29] interesting pod/e2e-shared-b-w0-68b466-openshift-default-57cb5457fc-jgkhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body=
Oct 30 19:03:33.999773 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:33.996132    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-68b466-openshift-default-57cb5457fc-jgkhq" podUID="19f00f47-605d-4b19-9a1c-664de3661507" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused"
Oct 30 19:03:34.050710 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:34.041180    4525 patch_prober.go:29] interesting pod/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Oct 30 19:03:34.050710 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:34.041293    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh" podUID="8b37419e-12f0-4e1e-8b2a-48c39c4c9e99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Oct 30 19:03:34.243708 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:34.243273    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-baa504/auth-test-model-baa504-kserve-64c89767d5-d4pq4"
Oct 30 19:03:34.262304 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:34.262211    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-baa504/auth-test-model-baa504-kserve-64c89767d5-d4pq4"
Oct 30 19:03:34.314708 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:34.309955    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-229cp" podStartSLOduration=4.309932421 podStartE2EDuration="4.309932421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:03:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:03:31.61263204 +0000 UTC m=+1634.238302005" watchObservedRunningTime="2026-10-30 19:03:34.309932421 +0000 UTC m=+1636.935602386"
Oct 30 19:03:34.649706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:34.634176    4525 patch_prober.go:29] interesting pod/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-229cp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body=
Oct 30 19:03:34.649706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:34.634254    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-229cp" podUID="5d9a2656-6bae-4277-a7f3-b43e8143dbd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused"
Oct 30 19:03:34.964889 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:34.964793    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-baa504/auth-test-model-baa504-kserve-64c89767d5-d4pq4"
Oct 30 19:03:35.008705 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:35.008604    4525 patch_prober.go:29] interesting pod/e2e-shared-b-w0-68b466-openshift-default-57cb5457fc-jgkhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body=
Oct 30 19:03:35.008914 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:35.008708    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-68b466-openshift-default-57cb5457fc-jgkhq" podUID="19f00f47-605d-4b19-9a1c-664de3661507" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused"
Oct 30 19:03:35.042707 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:35.042346    4525 patch_prober.go:29] interesting pod/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Oct 30 19:03:35.042707 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:35.042415    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh" podUID="8b37419e-12f0-4e1e-8b2a-48c39c4c9e99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Oct 30 19:03:35.651698 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:35.650637    4525 patch_prober.go:29] interesting pod/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-229cp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body=
Oct 30 19:03:35.651698 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:35.650763    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-229cp" podUID="5d9a2656-6bae-4277-a7f3-b43e8143dbd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused"
Oct 30 19:03:35.996770 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:35.993145    4525 patch_prober.go:29] interesting pod/e2e-shared-b-w0-68b466-openshift-default-57cb5457fc-jgkhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body=
Oct 30 19:03:35.996770 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:35.993206    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-68b466-openshift-default-57cb5457fc-jgkhq" podUID="19f00f47-605d-4b19-9a1c-664de3661507" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused"
Oct 30 19:03:36.044139 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:36.044106    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh"
Oct 30 19:03:36.044581 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:36.044559    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh"
Oct 30 19:03:36.045451 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:36.045429    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh"
Oct 30 19:03:36.629579 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:36.629492    4525 patch_prober.go:29] interesting pod/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-229cp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body=
Oct 30 19:03:36.629579 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:36.629562    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-229cp" podUID="5d9a2656-6bae-4277-a7f3-b43e8143dbd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused"
Oct 30 19:03:37.005762 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:37.005637    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-68b466-openshift-default-57cb5457fc-jgkhq"
Oct 30 19:03:37.007246 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:37.007217    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-68b466-openshift-default-57cb5457fc-jgkhq"
Oct 30 19:03:37.629159 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:37.629114    4525 patch_prober.go:29] interesting pod/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-229cp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body=
Oct 30 19:03:37.629350 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:37.629179    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-229cp" podUID="5d9a2656-6bae-4277-a7f3-b43e8143dbd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused"
Oct 30 19:03:38.629011 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:38.628976    4525 patch_prober.go:29] interesting pod/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-229cp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body=
Oct 30 19:03:38.629436 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:38.629050    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-229cp" podUID="5d9a2656-6bae-4277-a7f3-b43e8143dbd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused"
Oct 30 19:03:39.639425 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:39.639394    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-229cp"
Oct 30 19:03:39.640136 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:39.640109    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-229cp"
Oct 30 19:03:39.641210 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:39.641187    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-229cp"
Oct 30 19:03:45.154707 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:45.147475    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-c2e3f9/auth-test-model-c2e3f9-kserve-6b9f58d9cc-pmkg2" podUID="fedf2771-d37f-4d43-83f4-a04290b9fa9c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.203:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:03:47.468936 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:03:47.468901    4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d49l" containerName="extractor"
Oct 30 19:03:50.636928 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:50.636880    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-xsvss"]
Oct 30 19:03:50.646796 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:50.646768    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-xsvss"
Oct 30 19:03:50.653582 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:50.653548    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-xsvss"]
Oct 30 19:03:50.822450 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:50.822404    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a39e87cd-03ff-4777-bf20-8878edb373fa-credential-socket\") pod \"e2e-explicit-e9ada0-openshift-default-685f6fdbfc-xsvss\" (UID: \"a39e87cd-03ff-4777-bf20-8878edb373fa\") " pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-xsvss"
Oct 30 19:03:50.822772 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:50.822738    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a39e87cd-03ff-4777-bf20-8878edb373fa-istio-podinfo\") pod \"e2e-explicit-e9ada0-openshift-default-685f6fdbfc-xsvss\" (UID: \"a39e87cd-03ff-4777-bf20-8878edb373fa\") " pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-xsvss"
Oct 30 19:03:50.822931 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:50.822807    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxdcn\" (UniqueName: \"kubernetes.io/projected/a39e87cd-03ff-4777-bf20-8878edb373fa-kube-api-access-dxdcn\") pod \"e2e-explicit-e9ada0-openshift-default-685f6fdbfc-xsvss\" (UID: \"a39e87cd-03ff-4777-bf20-8878edb373fa\") " pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-xsvss"
Oct 30 19:03:50.822931 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:50.822903    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a39e87cd-03ff-4777-bf20-8878edb373fa-istio-token\") pod \"e2e-explicit-e9ada0-openshift-default-685f6fdbfc-xsvss\" (UID: \"a39e87cd-03ff-4777-bf20-8878edb373fa\") " pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-xsvss"
Oct 30 19:03:50.823075 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:50.822943    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a39e87cd-03ff-4777-bf20-8878edb373fa-workload-socket\") pod \"e2e-explicit-e9ada0-openshift-default-685f6fdbfc-xsvss\" (UID: \"a39e87cd-03ff-4777-bf20-8878edb373fa\") " pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-xsvss"
Oct 30 19:03:50.823075 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:50.822966    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a39e87cd-03ff-4777-bf20-8878edb373fa-istio-envoy\") pod \"e2e-explicit-e9ada0-openshift-default-685f6fdbfc-xsvss\" (UID: \"a39e87cd-03ff-4777-bf20-8878edb373fa\") " pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-xsvss"
Oct 30 19:03:50.823075 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:50.823048    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a39e87cd-03ff-4777-bf20-8878edb373fa-workload-certs\") pod \"e2e-explicit-e9ada0-openshift-default-685f6fdbfc-xsvss\" (UID: \"a39e87cd-03ff-4777-bf20-8878edb373fa\") " pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-xsvss"
Oct 30 19:03:50.823260 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:50.823180    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a39e87cd-03ff-4777-bf20-8878edb373fa-istiod-ca-cert\") pod \"e2e-explicit-e9ada0-openshift-default-685f6fdbfc-xsvss\" (UID: \"a39e87cd-03ff-4777-bf20-8878edb373fa\") " pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-xsvss"
Oct 30 19:03:50.823260 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:50.823221    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a39e87cd-03ff-4777-bf20-8878edb373fa-istio-data\") pod \"e2e-explicit-e9ada0-openshift-default-685f6fdbfc-xsvss\" (UID: \"a39e87cd-03ff-4777-bf20-8878edb373fa\") " pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-xsvss"
Oct 30 19:03:50.924623 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:50.924522    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a39e87cd-03ff-4777-bf20-8878edb373fa-workload-certs\") pod \"e2e-explicit-e9ada0-openshift-default-685f6fdbfc-xsvss\" (UID: \"a39e87cd-03ff-4777-bf20-8878edb373fa\") " pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-xsvss"
Oct 30 19:03:50.924810 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:50.924714    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a39e87cd-03ff-4777-bf20-8878edb373fa-istiod-ca-cert\") pod \"e2e-explicit-e9ada0-openshift-default-685f6fdbfc-xsvss\" (UID: \"a39e87cd-03ff-4777-bf20-8878edb373fa\") " pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-xsvss"
Oct 30 19:03:50.924810 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:50.924781    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a39e87cd-03ff-4777-bf20-8878edb373fa-istio-data\") pod \"e2e-explicit-e9ada0-openshift-default-685f6fdbfc-xsvss\" (UID: \"a39e87cd-03ff-4777-bf20-8878edb373fa\") " pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-xsvss"
Oct 30 19:03:50.924947 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:50.924881    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a39e87cd-03ff-4777-bf20-8878edb373fa-credential-socket\") pod \"e2e-explicit-e9ada0-openshift-default-685f6fdbfc-xsvss\" (UID: \"a39e87cd-03ff-4777-bf20-8878edb373fa\") " pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-xsvss"
Oct 30 19:03:50.924947 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:50.924912    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a39e87cd-03ff-4777-bf20-8878edb373fa-istio-podinfo\") pod \"e2e-explicit-e9ada0-openshift-default-685f6fdbfc-xsvss\" (UID: \"a39e87cd-03ff-4777-bf20-8878edb373fa\") " pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-xsvss"
Oct 30 19:03:50.925061 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:50.924956    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxdcn\" (UniqueName: \"kubernetes.io/projected/a39e87cd-03ff-4777-bf20-8878edb373fa-kube-api-access-dxdcn\") pod \"e2e-explicit-e9ada0-openshift-default-685f6fdbfc-xsvss\" (UID: \"a39e87cd-03ff-4777-bf20-8878edb373fa\") " pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-xsvss"
Oct 30 19:03:50.925061 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:50.925002    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a39e87cd-03ff-4777-bf20-8878edb373fa-workload-certs\") pod \"e2e-explicit-e9ada0-openshift-default-685f6fdbfc-xsvss\" (UID: \"a39e87cd-03ff-4777-bf20-8878edb373fa\") " pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-xsvss"
Oct 30 19:03:50.925061 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:50.925010    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a39e87cd-03ff-4777-bf20-8878edb373fa-istio-token\") pod \"e2e-explicit-e9ada0-openshift-default-685f6fdbfc-xsvss\" (UID: \"a39e87cd-03ff-4777-bf20-8878edb373fa\") " pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-xsvss"
Oct 30 19:03:50.925217 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:50.925077    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a39e87cd-03ff-4777-bf20-8878edb373fa-workload-socket\") pod \"e2e-explicit-e9ada0-openshift-default-685f6fdbfc-xsvss\" (UID: \"a39e87cd-03ff-4777-bf20-8878edb373fa\") " pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-xsvss"
Oct 30 19:03:50.925217 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:50.925113    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a39e87cd-03ff-4777-bf20-8878edb373fa-istio-envoy\") pod \"e2e-explicit-e9ada0-openshift-default-685f6fdbfc-xsvss\" (UID: \"a39e87cd-03ff-4777-bf20-8878edb373fa\") " pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-xsvss"
Oct 30 19:03:50.925329 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:50.925227    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a39e87cd-03ff-4777-bf20-8878edb373fa-istio-data\") pod \"e2e-explicit-e9ada0-openshift-default-685f6fdbfc-xsvss\" (UID: \"a39e87cd-03ff-4777-bf20-8878edb373fa\") " pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-xsvss"
Oct 30 19:03:50.925545 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:50.925516    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a39e87cd-03ff-4777-bf20-8878edb373fa-istiod-ca-cert\") pod \"e2e-explicit-e9ada0-openshift-default-685f6fdbfc-xsvss\" (UID: \"a39e87cd-03ff-4777-bf20-8878edb373fa\") " pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-xsvss"
Oct 30 19:03:50.925545 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:50.925535    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a39e87cd-03ff-4777-bf20-8878edb373fa-workload-socket\") pod \"e2e-explicit-e9ada0-openshift-default-685f6fdbfc-xsvss\" (UID: \"a39e87cd-03ff-4777-bf20-8878edb373fa\") " pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-xsvss"
Oct 30 19:03:50.925716 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:50.925595    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a39e87cd-03ff-4777-bf20-8878edb373fa-credential-socket\") pod \"e2e-explicit-e9ada0-openshift-default-685f6fdbfc-xsvss\" (UID: \"a39e87cd-03ff-4777-bf20-8878edb373fa\") " pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-xsvss"
Oct 30 19:03:50.927701 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:50.927641    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a39e87cd-03ff-4777-bf20-8878edb373fa-istio-envoy\") pod \"e2e-explicit-e9ada0-openshift-default-685f6fdbfc-xsvss\" (UID: \"a39e87cd-03ff-4777-bf20-8878edb373fa\") " pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-xsvss"
Oct 30 19:03:50.928273 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:50.928250    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a39e87cd-03ff-4777-bf20-8878edb373fa-istio-podinfo\") pod \"e2e-explicit-e9ada0-openshift-default-685f6fdbfc-xsvss\" (UID: \"a39e87cd-03ff-4777-bf20-8878edb373fa\") " pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-xsvss"
Oct 30 19:03:50.933604 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:50.933581    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a39e87cd-03ff-4777-bf20-8878edb373fa-istio-token\") pod \"e2e-explicit-e9ada0-openshift-default-685f6fdbfc-xsvss\" (UID: \"a39e87cd-03ff-4777-bf20-8878edb373fa\") " pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-xsvss"
Oct 30 19:03:50.935931 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:50.935912    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxdcn\" (UniqueName: \"kubernetes.io/projected/a39e87cd-03ff-4777-bf20-8878edb373fa-kube-api-access-dxdcn\") pod \"e2e-explicit-e9ada0-openshift-default-685f6fdbfc-xsvss\" (UID: \"a39e87cd-03ff-4777-bf20-8878edb373fa\") " pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-xsvss"
Oct 30 19:03:50.974896 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:50.974862    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-xsvss"
Oct 30 19:03:51.323980 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:51.323607    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-xsvss"]
Oct 30 19:03:51.326206 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:03:51.326168    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda39e87cd_03ff_4777_bf20_8878edb373fa.slice/crio-e610cad3fd0c230b298f2f9daf3db165e1942df6323eb9f34aca312672104868 WatchSource:0}: Error finding container e610cad3fd0c230b298f2f9daf3db165e1942df6323eb9f34aca312672104868: Status 404 returned error can't find the container with id e610cad3fd0c230b298f2f9daf3db165e1942df6323eb9f34aca312672104868
Oct 30 19:03:51.330970 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:51.330930    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:03:51.331064 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:51.331028    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:03:51.331111 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:51.331073    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:03:51.368779 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:51.368740    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-xsvss" event={"ID":"a39e87cd-03ff-4777-bf20-8878edb373fa","Type":"ContainerStarted","Data":"e610cad3fd0c230b298f2f9daf3db165e1942df6323eb9f34aca312672104868"}
Oct 30 19:03:52.386536 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:52.386496    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-xsvss" event={"ID":"a39e87cd-03ff-4777-bf20-8878edb373fa","Type":"ContainerStarted","Data":"05419d18ddf49c03647a8cc31e5cc4bf21312b6429b493f0750c74633d873431"}
Oct 30 19:03:52.453638 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:52.451788    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-xsvss" podStartSLOduration=2.451755558 podStartE2EDuration="2.451755558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:03:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:03:52.441136401 +0000 UTC m=+1655.066806364" watchObservedRunningTime="2026-10-30 19:03:52.451755558 +0000 UTC m=+1655.077425524"
Oct 30 19:03:52.982152 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:52.982114    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-xsvss"
Oct 30 19:03:52.986856 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:52.986814    4525 patch_prober.go:29] interesting pod/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-xsvss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body=
Oct 30 19:03:52.987145 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:52.987107    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-xsvss" podUID="a39e87cd-03ff-4777-bf20-8878edb373fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused"
Oct 30 19:03:53.951878 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:53.951844    4525 patch_prober.go:29] interesting pod/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:03:53.952343 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:53.951915    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k" podUID="755fd8c8-1c8a-4040-b426-405e8e5789aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:03:53.988708 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:53.980118    4525 patch_prober.go:29] interesting pod/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-xsvss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body=
Oct 30 19:03:53.988708 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:53.980186    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-xsvss" podUID="a39e87cd-03ff-4777-bf20-8878edb373fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused"
Oct 30 19:03:54.873305 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:54.873266    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-68b466/auth-test-model-68b466-kserve-6dd5b87d8f-ddr8r"]
Oct 30 19:03:54.932563 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:54.932507    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-68b466/auth-test-model-68b466-kserve-6dd5b87d8f-ddr8r"
Oct 30 19:03:54.940780 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:54.940746    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-68b466\"/\"openshift-service-ca.crt\""
Oct 30 19:03:54.940964 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:54.940906    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-68b466\"/\"default-dockercfg-6bvz4\""
Oct 30 19:03:54.941054 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:54.941018    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-68b466\"/\"auth-test-model-68b466-kserve-self-signed-certs\""
Oct 30 19:03:54.941294 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:54.941263    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-68b466/auth-test-model-68b466-kserve-6dd5b87d8f-ddr8r"]
Oct 30 19:03:54.948148 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:54.948113    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59mv5\" (UniqueName: \"kubernetes.io/projected/1054f7aa-9712-425b-8e66-d78b0f17962a-kube-api-access-59mv5\") pod \"auth-test-model-68b466-kserve-6dd5b87d8f-ddr8r\" (UID: \"1054f7aa-9712-425b-8e66-d78b0f17962a\") " pod="ai-tenant-e2e-shared-b-w0-68b466/auth-test-model-68b466-kserve-6dd5b87d8f-ddr8r"
Oct 30 19:03:54.948307 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:54.948170    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1054f7aa-9712-425b-8e66-d78b0f17962a-tls-certs\") pod \"auth-test-model-68b466-kserve-6dd5b87d8f-ddr8r\" (UID: \"1054f7aa-9712-425b-8e66-d78b0f17962a\") " pod="ai-tenant-e2e-shared-b-w0-68b466/auth-test-model-68b466-kserve-6dd5b87d8f-ddr8r"
Oct 30 19:03:54.948383 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:54.948357    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1054f7aa-9712-425b-8e66-d78b0f17962a-model-cache\") pod \"auth-test-model-68b466-kserve-6dd5b87d8f-ddr8r\" (UID: \"1054f7aa-9712-425b-8e66-d78b0f17962a\") " pod="ai-tenant-e2e-shared-b-w0-68b466/auth-test-model-68b466-kserve-6dd5b87d8f-ddr8r"
Oct 30 19:03:54.948447 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:54.948393    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1054f7aa-9712-425b-8e66-d78b0f17962a-tmp-dir\") pod \"auth-test-model-68b466-kserve-6dd5b87d8f-ddr8r\" (UID: \"1054f7aa-9712-425b-8e66-d78b0f17962a\") " pod="ai-tenant-e2e-shared-b-w0-68b466/auth-test-model-68b466-kserve-6dd5b87d8f-ddr8r"
Oct 30 19:03:54.948447 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:54.948422    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1054f7aa-9712-425b-8e66-d78b0f17962a-dshm\") pod \"auth-test-model-68b466-kserve-6dd5b87d8f-ddr8r\" (UID: \"1054f7aa-9712-425b-8e66-d78b0f17962a\") " pod="ai-tenant-e2e-shared-b-w0-68b466/auth-test-model-68b466-kserve-6dd5b87d8f-ddr8r"
Oct 30 19:03:54.948558 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:54.948532    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1054f7aa-9712-425b-8e66-d78b0f17962a-home\") pod \"auth-test-model-68b466-kserve-6dd5b87d8f-ddr8r\" (UID: \"1054f7aa-9712-425b-8e66-d78b0f17962a\") " pod="ai-tenant-e2e-shared-b-w0-68b466/auth-test-model-68b466-kserve-6dd5b87d8f-ddr8r"
Oct 30 19:03:55.001713 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:55.001199    4525 patch_prober.go:29] interesting pod/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-xsvss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body=
Oct 30 19:03:55.001713 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:55.001276    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-xsvss" podUID="a39e87cd-03ff-4777-bf20-8878edb373fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused"
Oct 30 19:03:55.042923 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:55.042894    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-68b466\"/\"kube-root-ca.crt\""
Oct 30 19:03:55.065520 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:55.065479    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59mv5\" (UniqueName: \"kubernetes.io/projected/1054f7aa-9712-425b-8e66-d78b0f17962a-kube-api-access-59mv5\") pod \"auth-test-model-68b466-kserve-6dd5b87d8f-ddr8r\" (UID: \"1054f7aa-9712-425b-8e66-d78b0f17962a\") " pod="ai-tenant-e2e-shared-b-w0-68b466/auth-test-model-68b466-kserve-6dd5b87d8f-ddr8r"
Oct 30 19:03:55.065755 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:55.065546    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1054f7aa-9712-425b-8e66-d78b0f17962a-tls-certs\") pod \"auth-test-model-68b466-kserve-6dd5b87d8f-ddr8r\" (UID: \"1054f7aa-9712-425b-8e66-d78b0f17962a\") " pod="ai-tenant-e2e-shared-b-w0-68b466/auth-test-model-68b466-kserve-6dd5b87d8f-ddr8r"
Oct 30 19:03:55.065839 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:55.065790    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1054f7aa-9712-425b-8e66-d78b0f17962a-model-cache\") pod \"auth-test-model-68b466-kserve-6dd5b87d8f-ddr8r\" (UID: \"1054f7aa-9712-425b-8e66-d78b0f17962a\") " pod="ai-tenant-e2e-shared-b-w0-68b466/auth-test-model-68b466-kserve-6dd5b87d8f-ddr8r"
Oct 30 19:03:55.065839 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:55.065832    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1054f7aa-9712-425b-8e66-d78b0f17962a-tmp-dir\") pod \"auth-test-model-68b466-kserve-6dd5b87d8f-ddr8r\" (UID: \"1054f7aa-9712-425b-8e66-d78b0f17962a\") " pod="ai-tenant-e2e-shared-b-w0-68b466/auth-test-model-68b466-kserve-6dd5b87d8f-ddr8r"
Oct 30 19:03:55.066064 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:55.065861    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1054f7aa-9712-425b-8e66-d78b0f17962a-dshm\") pod \"auth-test-model-68b466-kserve-6dd5b87d8f-ddr8r\" (UID: \"1054f7aa-9712-425b-8e66-d78b0f17962a\") " pod="ai-tenant-e2e-shared-b-w0-68b466/auth-test-model-68b466-kserve-6dd5b87d8f-ddr8r"
Oct 30 19:03:55.069510 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:55.069484    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1054f7aa-9712-425b-8e66-d78b0f17962a-model-cache\") pod \"auth-test-model-68b466-kserve-6dd5b87d8f-ddr8r\" (UID: \"1054f7aa-9712-425b-8e66-d78b0f17962a\") " pod="ai-tenant-e2e-shared-b-w0-68b466/auth-test-model-68b466-kserve-6dd5b87d8f-ddr8r"
Oct 30 19:03:55.097706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:55.091593    4525 patch_prober.go:29] interesting pod/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:03:55.097706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:55.091658    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm" podUID="6ccf27c7-f6d3-457b-9ff8-69b54f9b9287" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:03:55.097706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:55.092029    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1054f7aa-9712-425b-8e66-d78b0f17962a-tls-certs\") pod \"auth-test-model-68b466-kserve-6dd5b87d8f-ddr8r\" (UID: \"1054f7aa-9712-425b-8e66-d78b0f17962a\") " pod="ai-tenant-e2e-shared-b-w0-68b466/auth-test-model-68b466-kserve-6dd5b87d8f-ddr8r"
Oct 30 19:03:55.101704 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:55.098374    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1054f7aa-9712-425b-8e66-d78b0f17962a-tmp-dir\") pod \"auth-test-model-68b466-kserve-6dd5b87d8f-ddr8r\" (UID: \"1054f7aa-9712-425b-8e66-d78b0f17962a\") " pod="ai-tenant-e2e-shared-b-w0-68b466/auth-test-model-68b466-kserve-6dd5b87d8f-ddr8r"
Oct 30 19:03:55.107815 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:55.098674    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1054f7aa-9712-425b-8e66-d78b0f17962a-home\") pod \"auth-test-model-68b466-kserve-6dd5b87d8f-ddr8r\" (UID: \"1054f7aa-9712-425b-8e66-d78b0f17962a\") " pod="ai-tenant-e2e-shared-b-w0-68b466/auth-test-model-68b466-kserve-6dd5b87d8f-ddr8r"
Oct 30 19:03:55.125782 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:55.098963    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1054f7aa-9712-425b-8e66-d78b0f17962a-home\") pod \"auth-test-model-68b466-kserve-6dd5b87d8f-ddr8r\" (UID: \"1054f7aa-9712-425b-8e66-d78b0f17962a\") " pod="ai-tenant-e2e-shared-b-w0-68b466/auth-test-model-68b466-kserve-6dd5b87d8f-ddr8r"
Oct 30 19:03:55.148984 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:55.110019    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1054f7aa-9712-425b-8e66-d78b0f17962a-dshm\") pod \"auth-test-model-68b466-kserve-6dd5b87d8f-ddr8r\" (UID: \"1054f7aa-9712-425b-8e66-d78b0f17962a\") " pod="ai-tenant-e2e-shared-b-w0-68b466/auth-test-model-68b466-kserve-6dd5b87d8f-ddr8r"
Oct 30 19:03:55.155708 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:55.152322    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59mv5\" (UniqueName: \"kubernetes.io/projected/1054f7aa-9712-425b-8e66-d78b0f17962a-kube-api-access-59mv5\") pod \"auth-test-model-68b466-kserve-6dd5b87d8f-ddr8r\" (UID: \"1054f7aa-9712-425b-8e66-d78b0f17962a\") " pod="ai-tenant-e2e-shared-b-w0-68b466/auth-test-model-68b466-kserve-6dd5b87d8f-ddr8r"
Oct 30 19:03:55.312520 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:55.311260    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-68b466/auth-test-model-68b466-kserve-6dd5b87d8f-ddr8r"
Oct 30 19:03:55.687754 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:03:55.679393    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1054f7aa_9712_425b_8e66_d78b0f17962a.slice/crio-2b552003c8632db1dcc7532c442d6cdce9079133091a56e81cd7e89ae2c0a6a4 WatchSource:0}: Error finding container 2b552003c8632db1dcc7532c442d6cdce9079133091a56e81cd7e89ae2c0a6a4: Status 404 returned error can't find the container with id 2b552003c8632db1dcc7532c442d6cdce9079133091a56e81cd7e89ae2c0a6a4
Oct 30 19:03:55.708202 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:55.708161    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-68b466/auth-test-model-68b466-kserve-6dd5b87d8f-ddr8r"]
Oct 30 19:03:56.006799 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:56.006440    4525 patch_prober.go:29] interesting pod/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-xsvss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body=
Oct 30 19:03:56.006799 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:56.006503    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-xsvss" podUID="a39e87cd-03ff-4777-bf20-8878edb373fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused"
Oct 30 19:03:56.454830 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:56.454788    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-68b466/auth-test-model-68b466-kserve-6dd5b87d8f-ddr8r" event={"ID":"1054f7aa-9712-425b-8e66-d78b0f17962a","Type":"ContainerStarted","Data":"27c64be629dbbf8354b606dc7787fdb11e3633e329c6ee1fbdd848ab55d62f84"}
Oct 30 19:03:56.455146 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:56.455122    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-68b466/auth-test-model-68b466-kserve-6dd5b87d8f-ddr8r" event={"ID":"1054f7aa-9712-425b-8e66-d78b0f17962a","Type":"ContainerStarted","Data":"2b552003c8632db1dcc7532c442d6cdce9079133091a56e81cd7e89ae2c0a6a4"}
Oct 30 19:03:56.498799 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:56.498723    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-68b466/auth-test-model-68b466-kserve-6dd5b87d8f-ddr8r" podStartSLOduration=2.498670838 podStartE2EDuration="2.498670838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:03:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:03:56.475119345 +0000 UTC m=+1659.100789308" watchObservedRunningTime="2026-10-30 19:03:56.498670838 +0000 UTC m=+1659.124340803"
Oct 30 19:03:56.975579 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:56.975534    4525 patch_prober.go:29] interesting pod/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-xsvss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body=
Oct 30 19:03:56.975885 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:56.975613    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-xsvss" podUID="a39e87cd-03ff-4777-bf20-8878edb373fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused"
Oct 30 19:03:57.312744 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:57.312653    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-68b466/auth-test-model-68b466-kserve-6dd5b87d8f-ddr8r"
Oct 30 19:03:58.039463 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:57.975983    4525 patch_prober.go:29] interesting pod/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-xsvss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body=
Oct 30 19:03:58.039463 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:57.976047    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-xsvss" podUID="a39e87cd-03ff-4777-bf20-8878edb373fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused"
Oct 30 19:03:58.475826 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:58.475794    4525 generic.go:356] "Generic (PLEG): container finished" podID="755fd8c8-1c8a-4040-b426-405e8e5789aa" containerID="a380b91526beebecb86524c7b78d56960e963c0cebf965e7293848ddbb0eb420" exitCode=0
Oct 30 19:03:58.476301 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:58.475874    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k" event={"ID":"755fd8c8-1c8a-4040-b426-405e8e5789aa","Type":"ContainerDied","Data":"a380b91526beebecb86524c7b78d56960e963c0cebf965e7293848ddbb0eb420"}
Oct 30 19:03:58.476390 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:58.476375    4525 scope.go:122] "RemoveContainer" containerID="a380b91526beebecb86524c7b78d56960e963c0cebf965e7293848ddbb0eb420"
Oct 30 19:03:58.478443 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:58.478402    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:03:58.478587 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:58.478496    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:03:58.478587 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:58.478543    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:03:58.981070 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:58.981032    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-xsvss"
Oct 30 19:03:58.981559 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:58.981533    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-xsvss"
Oct 30 19:03:58.982571 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:58.982554    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-xsvss"
Oct 30 19:03:59.273566 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:59.273485    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-e9ada0-98x29"]
Oct 30 19:03:59.277515 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:59.277493    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-e9ada0-98x29"
Oct 30 19:03:59.283322 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:59.282853    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:03:59.283322 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:59.282960    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94hcn\""
Oct 30 19:03:59.288980 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:59.288952    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-e9ada0-98x29"]
Oct 30 19:03:59.365026 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:59.364992    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9kmd\" (UniqueName: \"kubernetes.io/projected/bcf99dfe-605a-4962-aaa4-ec1a6f7f31b1-kube-api-access-m9kmd\") pod \"maas-api-revoke-keys-e2e-explicit-e9ada0-98x29\" (UID: \"bcf99dfe-605a-4962-aaa4-ec1a6f7f31b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-e9ada0-98x29"
Oct 30 19:03:59.365197 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:59.365145    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcf99dfe-605a-4962-aaa4-ec1a6f7f31b1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-e9ada0-98x29\" (UID: \"bcf99dfe-605a-4962-aaa4-ec1a6f7f31b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-e9ada0-98x29"
Oct 30 19:03:59.365321 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:59.365304    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcf99dfe-605a-4962-aaa4-ec1a6f7f31b1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-e9ada0-98x29\" (UID: \"bcf99dfe-605a-4962-aaa4-ec1a6f7f31b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-e9ada0-98x29"
Oct 30 19:03:59.467095 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:59.467057    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9kmd\" (UniqueName: \"kubernetes.io/projected/bcf99dfe-605a-4962-aaa4-ec1a6f7f31b1-kube-api-access-m9kmd\") pod \"maas-api-revoke-keys-e2e-explicit-e9ada0-98x29\" (UID: \"bcf99dfe-605a-4962-aaa4-ec1a6f7f31b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-e9ada0-98x29"
Oct 30 19:03:59.467327 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:59.467190    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcf99dfe-605a-4962-aaa4-ec1a6f7f31b1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-e9ada0-98x29\" (UID: \"bcf99dfe-605a-4962-aaa4-ec1a6f7f31b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-e9ada0-98x29"
Oct 30 19:03:59.467391 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:59.467375    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcf99dfe-605a-4962-aaa4-ec1a6f7f31b1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-e9ada0-98x29\" (UID: \"bcf99dfe-605a-4962-aaa4-ec1a6f7f31b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-e9ada0-98x29"
Oct 30 19:03:59.468084 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:59.468057    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcf99dfe-605a-4962-aaa4-ec1a6f7f31b1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-e9ada0-98x29\" (UID: \"bcf99dfe-605a-4962-aaa4-ec1a6f7f31b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-e9ada0-98x29"
Oct 30 19:03:59.469707 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:59.469671    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcf99dfe-605a-4962-aaa4-ec1a6f7f31b1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-e9ada0-98x29\" (UID: \"bcf99dfe-605a-4962-aaa4-ec1a6f7f31b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-e9ada0-98x29"
Oct 30 19:03:59.479202 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:59.479172    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9kmd\" (UniqueName: \"kubernetes.io/projected/bcf99dfe-605a-4962-aaa4-ec1a6f7f31b1-kube-api-access-m9kmd\") pod \"maas-api-revoke-keys-e2e-explicit-e9ada0-98x29\" (UID: \"bcf99dfe-605a-4962-aaa4-ec1a6f7f31b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-e9ada0-98x29"
Oct 30 19:03:59.504374 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:59.504337    4525 generic.go:356] "Generic (PLEG): container finished" podID="6ccf27c7-f6d3-457b-9ff8-69b54f9b9287" containerID="325128f3e14fb21cd06d956ada021baba82277fcd141493de35a31bccde645a0" exitCode=0
Oct 30 19:03:59.504554 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:59.504414    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm" event={"ID":"6ccf27c7-f6d3-457b-9ff8-69b54f9b9287","Type":"ContainerDied","Data":"325128f3e14fb21cd06d956ada021baba82277fcd141493de35a31bccde645a0"}
Oct 30 19:03:59.504973 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:59.504950    4525 scope.go:122] "RemoveContainer" containerID="325128f3e14fb21cd06d956ada021baba82277fcd141493de35a31bccde645a0"
Oct 30 19:03:59.507034 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:59.507006    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:03:59.507144 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:59.507069    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:03:59.507144 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:59.507083    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k" event={"ID":"755fd8c8-1c8a-4040-b426-405e8e5789aa","Type":"ContainerStarted","Data":"2dd428a6d3426837292b028d088263871789dfeef3c999fb2ea8c8c5c5ea3aa0"}
Oct 30 19:03:59.507144 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:59.507109    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:03:59.600779 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:03:59.597788    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-e9ada0-98x29"
Oct 30 19:04:00.027816 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:04:00.027779    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcf99dfe_605a_4962_aaa4_ec1a6f7f31b1.slice/crio-571c9e696de07757985e34204e79a57efdc0b4010ea1a0c660818f0c8d88e2e8 WatchSource:0}: Error finding container 571c9e696de07757985e34204e79a57efdc0b4010ea1a0c660818f0c8d88e2e8: Status 404 returned error can't find the container with id 571c9e696de07757985e34204e79a57efdc0b4010ea1a0c660818f0c8d88e2e8
Oct 30 19:04:00.028778 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:00.028754    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-e9ada0-98x29"]
Oct 30 19:04:00.150243 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:00.150210    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k"
Oct 30 19:04:00.150467 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:00.150372    4525 patch_prober.go:29] interesting pod/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body=
Oct 30 19:04:00.150467 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:00.150434    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k" podUID="755fd8c8-1c8a-4040-b426-405e8e5789aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused"
Oct 30 19:04:00.519332 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:00.519298    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-e9ada0-98x29" event={"ID":"bcf99dfe-605a-4962-aaa4-ec1a6f7f31b1","Type":"ContainerStarted","Data":"571c9e696de07757985e34204e79a57efdc0b4010ea1a0c660818f0c8d88e2e8"}
Oct 30 19:04:00.522607 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:00.522558    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm" event={"ID":"6ccf27c7-f6d3-457b-9ff8-69b54f9b9287","Type":"ContainerStarted","Data":"ebc7f76d96aafbd104dfa29a040cffffaf1790bdc6806aa0827284f5fe573d09"}
Oct 30 19:04:01.153158 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:01.153111    4525 patch_prober.go:29] interesting pod/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body=
Oct 30 19:04:01.153395 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:01.153188    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k" podUID="755fd8c8-1c8a-4040-b426-405e8e5789aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused"
Oct 30 19:04:01.438522 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:01.438437    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm"
Oct 30 19:04:01.438706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:01.438530    4525 patch_prober.go:29] interesting pod/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Oct 30 19:04:01.438706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:01.438563    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm" podUID="6ccf27c7-f6d3-457b-9ff8-69b54f9b9287" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Oct 30 19:04:01.539862 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:01.539817    4525 generic.go:356] "Generic (PLEG): container finished" podID="bcf99dfe-605a-4962-aaa4-ec1a6f7f31b1" containerID="b7901d56252fc83784ba3e6cb5095a20432b97256969e1f369d0af5a0015849d" exitCode=0
Oct 30 19:04:01.561044 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:01.560971    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-e9ada0-98x29" event={"ID":"bcf99dfe-605a-4962-aaa4-ec1a6f7f31b1","Type":"ContainerDied","Data":"b7901d56252fc83784ba3e6cb5095a20432b97256969e1f369d0af5a0015849d"}
Oct 30 19:04:02.150563 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:02.150529    4525 patch_prober.go:29] interesting pod/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body=
Oct 30 19:04:02.150816 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:02.150589    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k" podUID="755fd8c8-1c8a-4040-b426-405e8e5789aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused"
Oct 30 19:04:02.438765 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:02.438317    4525 patch_prober.go:29] interesting pod/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Oct 30 19:04:02.438765 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:02.438383    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm" podUID="6ccf27c7-f6d3-457b-9ff8-69b54f9b9287" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Oct 30 19:04:02.848935 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:02.848905    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-e9ada0-98x29"
Oct 30 19:04:02.946168 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:02.941855    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcf99dfe-605a-4962-aaa4-ec1a6f7f31b1-kube-api-access-m9kmd\" (UniqueName: \"kubernetes.io/projected/bcf99dfe-605a-4962-aaa4-ec1a6f7f31b1-kube-api-access-m9kmd\") pod \"bcf99dfe-605a-4962-aaa4-ec1a6f7f31b1\" (UID: \"bcf99dfe-605a-4962-aaa4-ec1a6f7f31b1\") "
Oct 30 19:04:02.946168 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:02.942093    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcf99dfe-605a-4962-aaa4-ec1a6f7f31b1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcf99dfe-605a-4962-aaa4-ec1a6f7f31b1-maas-api-openshift-ca\") pod \"bcf99dfe-605a-4962-aaa4-ec1a6f7f31b1\" (UID: \"bcf99dfe-605a-4962-aaa4-ec1a6f7f31b1\") "
Oct 30 19:04:02.946168 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:02.942238    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bcf99dfe-605a-4962-aaa4-ec1a6f7f31b1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcf99dfe-605a-4962-aaa4-ec1a6f7f31b1-maas-api-xks-ca\") pod \"bcf99dfe-605a-4962-aaa4-ec1a6f7f31b1\" (UID: \"bcf99dfe-605a-4962-aaa4-ec1a6f7f31b1\") "
Oct 30 19:04:02.946168 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:02.943566    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcf99dfe-605a-4962-aaa4-ec1a6f7f31b1-maas-api-openshift-ca" pod "bcf99dfe-605a-4962-aaa4-ec1a6f7f31b1" (UID: "bcf99dfe-605a-4962-aaa4-ec1a6f7f31b1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:04:02.946168 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:02.946117    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcf99dfe-605a-4962-aaa4-ec1a6f7f31b1-kube-api-access-m9kmd" pod "bcf99dfe-605a-4962-aaa4-ec1a6f7f31b1" (UID: "bcf99dfe-605a-4962-aaa4-ec1a6f7f31b1"). InnerVolumeSpecName "kube-api-access-m9kmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:04:02.951239 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:02.951208    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcf99dfe-605a-4962-aaa4-ec1a6f7f31b1-maas-api-xks-ca" pod "bcf99dfe-605a-4962-aaa4-ec1a6f7f31b1" (UID: "bcf99dfe-605a-4962-aaa4-ec1a6f7f31b1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:04:03.068927 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:03.053842    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcf99dfe-605a-4962-aaa4-ec1a6f7f31b1-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:03.068927 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:03.053886    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9kmd\" (UniqueName: \"kubernetes.io/projected/bcf99dfe-605a-4962-aaa4-ec1a6f7f31b1-kube-api-access-m9kmd\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:03.068927 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:03.053902    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcf99dfe-605a-4962-aaa4-ec1a6f7f31b1-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:03.185716 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:03.185556    4525 patch_prober.go:29] interesting pod/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body=
Oct 30 19:04:03.185716 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:03.185664    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k" podUID="755fd8c8-1c8a-4040-b426-405e8e5789aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused"
Oct 30 19:04:03.438889 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:03.438855    4525 patch_prober.go:29] interesting pod/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Oct 30 19:04:03.439116 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:03.438917    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm" podUID="6ccf27c7-f6d3-457b-9ff8-69b54f9b9287" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Oct 30 19:04:03.560238 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:03.560209    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-e9ada0-98x29"
Oct 30 19:04:03.560496 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:03.560241    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-e9ada0-98x29" event={"ID":"bcf99dfe-605a-4962-aaa4-ec1a6f7f31b1","Type":"ContainerDied","Data":"571c9e696de07757985e34204e79a57efdc0b4010ea1a0c660818f0c8d88e2e8"}
Oct 30 19:04:03.560579 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:03.560523    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="571c9e696de07757985e34204e79a57efdc0b4010ea1a0c660818f0c8d88e2e8"
Oct 30 19:04:04.150126 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:04.150095    4525 patch_prober.go:29] interesting pod/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body=
Oct 30 19:04:04.150624 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:04.150592    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k" podUID="755fd8c8-1c8a-4040-b426-405e8e5789aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused"
Oct 30 19:04:04.439973 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:04.439754    4525 patch_prober.go:29] interesting pod/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Oct 30 19:04:04.439973 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:04.439822    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm" podUID="6ccf27c7-f6d3-457b-9ff8-69b54f9b9287" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Oct 30 19:04:05.159706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:05.159255    4525 patch_prober.go:29] interesting pod/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body=
Oct 30 19:04:05.159706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:05.159323    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k" podUID="755fd8c8-1c8a-4040-b426-405e8e5789aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused"
Oct 30 19:04:05.329814 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:05.320145    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-68b466/auth-test-model-68b466-kserve-6dd5b87d8f-ddr8r"
Oct 30 19:04:05.488770 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:05.476364    4525 patch_prober.go:29] interesting pod/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Oct 30 19:04:05.488770 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:05.476439    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm" podUID="6ccf27c7-f6d3-457b-9ff8-69b54f9b9287" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Oct 30 19:04:05.824760 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:05.824455    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-68b466/auth-test-model-68b466-kserve-6dd5b87d8f-ddr8r"
Oct 30 19:04:05.958613 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:05.958581    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-68b466/auth-test-model-68b466-kserve-6dd5b87d8f-ddr8r"
Oct 30 19:04:06.439028 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:06.438989    4525 patch_prober.go:29] interesting pod/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Oct 30 19:04:06.439452 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:06.439050    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm" podUID="6ccf27c7-f6d3-457b-9ff8-69b54f9b9287" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Oct 30 19:04:06.991867 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:06.991832    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k"
Oct 30 19:04:06.992772 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:06.992746    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k"
Oct 30 19:04:06.995342 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:06.995306    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k"
Oct 30 19:04:07.448966 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:07.448928    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm"
Oct 30 19:04:07.449937 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:07.449915    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm"
Oct 30 19:04:07.451060 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:07.451040    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm"
Oct 30 19:04:11.563305 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:11.563271    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-e9ada0-98x29"]
Oct 30 19:04:11.586920 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:11.584608    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-e9ada0-98x29"]
Oct 30 19:04:13.545581 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:13.545546    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcf99dfe-605a-4962-aaa4-ec1a6f7f31b1" path="/var/lib/kubelet/pods/bcf99dfe-605a-4962-aaa4-ec1a6f7f31b1/volumes"
Oct 30 19:04:17.399878 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:04:17.399837    4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d49l" containerName="extractor"
Oct 30 19:04:19.183701 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:19.178038    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k"]
Oct 30 19:04:19.183701 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:19.178322    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k" podUID="755fd8c8-1c8a-4040-b426-405e8e5789aa" containerName="istio-proxy" containerID="cri-o://2dd428a6d3426837292b028d088263871789dfeef3c999fb2ea8c8c5c5ea3aa0" gracePeriod=30
Oct 30 19:04:19.185530 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:19.185503    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-xsvss"]
Oct 30 19:04:19.185799 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:19.185757    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-xsvss" podUID="a39e87cd-03ff-4777-bf20-8878edb373fa" containerName="istio-proxy" containerID="cri-o://05419d18ddf49c03647a8cc31e5cc4bf21312b6429b493f0750c74633d873431" gracePeriod=30
Oct 30 19:04:19.997573 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:19.997537    4525 scope.go:122] "RemoveContainer" containerID="f7f580731b8bb57c3fd705f500cc319552464ed4b6f44cfc5beb57ec1e41e0c9"
Oct 30 19:04:20.011105 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:20.011080    4525 scope.go:122] "RemoveContainer" containerID="6f6e7c38cf81f9888734a208501f27d045f7cfa123a722c9067feba60023bf88"
Oct 30 19:04:20.021458 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:20.021440    4525 scope.go:122] "RemoveContainer" containerID="f757d57200c0cf51301ccc9edf446344169219ba5f3790ada7f50a04b1cf87ac"
Oct 30 19:04:21.995080 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:21.995040    4525 patch_prober.go:29] interesting pod/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:04:21.995691 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:21.995649    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k" podUID="755fd8c8-1c8a-4040-b426-405e8e5789aa" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:04:24.487256 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:24.487228    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-xsvss"
Oct 30 19:04:24.554552 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:24.554519    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a39e87cd-03ff-4777-bf20-8878edb373fa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a39e87cd-03ff-4777-bf20-8878edb373fa-workload-socket\") pod \"a39e87cd-03ff-4777-bf20-8878edb373fa\" (UID: \"a39e87cd-03ff-4777-bf20-8878edb373fa\") "
Oct 30 19:04:24.554765 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:24.554617    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a39e87cd-03ff-4777-bf20-8878edb373fa-istio-token\" (UniqueName: \"kubernetes.io/projected/a39e87cd-03ff-4777-bf20-8878edb373fa-istio-token\") pod \"a39e87cd-03ff-4777-bf20-8878edb373fa\" (UID: \"a39e87cd-03ff-4777-bf20-8878edb373fa\") "
Oct 30 19:04:24.554765 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:24.554655    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a39e87cd-03ff-4777-bf20-8878edb373fa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a39e87cd-03ff-4777-bf20-8878edb373fa-istio-envoy\") pod \"a39e87cd-03ff-4777-bf20-8878edb373fa\" (UID: \"a39e87cd-03ff-4777-bf20-8878edb373fa\") "
Oct 30 19:04:24.554765 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:24.554725    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a39e87cd-03ff-4777-bf20-8878edb373fa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a39e87cd-03ff-4777-bf20-8878edb373fa-istio-data\") pod \"a39e87cd-03ff-4777-bf20-8878edb373fa\" (UID: \"a39e87cd-03ff-4777-bf20-8878edb373fa\") "
Oct 30 19:04:24.554917 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:24.554768    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a39e87cd-03ff-4777-bf20-8878edb373fa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a39e87cd-03ff-4777-bf20-8878edb373fa-workload-certs\") pod \"a39e87cd-03ff-4777-bf20-8878edb373fa\" (UID: \"a39e87cd-03ff-4777-bf20-8878edb373fa\") "
Oct 30 19:04:24.554917 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:24.554849    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a39e87cd-03ff-4777-bf20-8878edb373fa-workload-socket" pod "a39e87cd-03ff-4777-bf20-8878edb373fa" (UID: "a39e87cd-03ff-4777-bf20-8878edb373fa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:04:24.554917 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:24.554859    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a39e87cd-03ff-4777-bf20-8878edb373fa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a39e87cd-03ff-4777-bf20-8878edb373fa-istiod-ca-cert\") pod \"a39e87cd-03ff-4777-bf20-8878edb373fa\" (UID: \"a39e87cd-03ff-4777-bf20-8878edb373fa\") "
Oct 30 19:04:24.555015 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:24.554915    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a39e87cd-03ff-4777-bf20-8878edb373fa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a39e87cd-03ff-4777-bf20-8878edb373fa-credential-socket\") pod \"a39e87cd-03ff-4777-bf20-8878edb373fa\" (UID: \"a39e87cd-03ff-4777-bf20-8878edb373fa\") "
Oct 30 19:04:24.555049 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:24.555016    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a39e87cd-03ff-4777-bf20-8878edb373fa-kube-api-access-dxdcn\" (UniqueName: \"kubernetes.io/projected/a39e87cd-03ff-4777-bf20-8878edb373fa-kube-api-access-dxdcn\") pod \"a39e87cd-03ff-4777-bf20-8878edb373fa\" (UID: \"a39e87cd-03ff-4777-bf20-8878edb373fa\") "
Oct 30 19:04:24.555123 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:24.555105    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a39e87cd-03ff-4777-bf20-8878edb373fa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a39e87cd-03ff-4777-bf20-8878edb373fa-istio-podinfo\") pod \"a39e87cd-03ff-4777-bf20-8878edb373fa\" (UID: \"a39e87cd-03ff-4777-bf20-8878edb373fa\") "
Oct 30 19:04:24.555412 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:24.555385    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a39e87cd-03ff-4777-bf20-8878edb373fa-istio-data" pod "a39e87cd-03ff-4777-bf20-8878edb373fa" (UID: "a39e87cd-03ff-4777-bf20-8878edb373fa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:04:24.555824 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:24.555801    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a39e87cd-03ff-4777-bf20-8878edb373fa-credential-socket" pod "a39e87cd-03ff-4777-bf20-8878edb373fa" (UID: "a39e87cd-03ff-4777-bf20-8878edb373fa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:04:24.556194 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:24.556163    4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a39e87cd-03ff-4777-bf20-8878edb373fa-workload-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:24.556301 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:24.556196    4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a39e87cd-03ff-4777-bf20-8878edb373fa-istio-data\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:24.556301 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:24.556211    4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a39e87cd-03ff-4777-bf20-8878edb373fa-credential-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:24.557458 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:24.557419    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a39e87cd-03ff-4777-bf20-8878edb373fa-istio-envoy" pod "a39e87cd-03ff-4777-bf20-8878edb373fa" (UID: "a39e87cd-03ff-4777-bf20-8878edb373fa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:04:24.558032 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:24.558001    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a39e87cd-03ff-4777-bf20-8878edb373fa-istiod-ca-cert" pod "a39e87cd-03ff-4777-bf20-8878edb373fa" (UID: "a39e87cd-03ff-4777-bf20-8878edb373fa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:04:24.558251 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:24.558196    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a39e87cd-03ff-4777-bf20-8878edb373fa-workload-certs" pod "a39e87cd-03ff-4777-bf20-8878edb373fa" (UID: "a39e87cd-03ff-4777-bf20-8878edb373fa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:04:24.560756 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:24.560655    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a39e87cd-03ff-4777-bf20-8878edb373fa-kube-api-access-dxdcn" pod "a39e87cd-03ff-4777-bf20-8878edb373fa" (UID: "a39e87cd-03ff-4777-bf20-8878edb373fa"). InnerVolumeSpecName "kube-api-access-dxdcn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:04:24.560839 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:24.560777    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a39e87cd-03ff-4777-bf20-8878edb373fa-istio-token" pod "a39e87cd-03ff-4777-bf20-8878edb373fa" (UID: "a39e87cd-03ff-4777-bf20-8878edb373fa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:04:24.562851 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:24.562817    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a39e87cd-03ff-4777-bf20-8878edb373fa-istio-podinfo" pod "a39e87cd-03ff-4777-bf20-8878edb373fa" (UID: "a39e87cd-03ff-4777-bf20-8878edb373fa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:04:24.658042 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:24.658006    4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a39e87cd-03ff-4777-bf20-8878edb373fa-istio-podinfo\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:24.658042 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:24.658041    4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a39e87cd-03ff-4777-bf20-8878edb373fa-istio-token\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:24.658335 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:24.658055    4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a39e87cd-03ff-4777-bf20-8878edb373fa-istio-envoy\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:24.658335 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:24.658067    4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a39e87cd-03ff-4777-bf20-8878edb373fa-workload-certs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:24.658335 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:24.658080    4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a39e87cd-03ff-4777-bf20-8878edb373fa-istiod-ca-cert\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:24.658335 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:24.658093    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxdcn\" (UniqueName: \"kubernetes.io/projected/a39e87cd-03ff-4777-bf20-8878edb373fa-kube-api-access-dxdcn\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:24.913737 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:24.913113    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k"
Oct 30 19:04:24.964338 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:24.964301    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/755fd8c8-1c8a-4040-b426-405e8e5789aa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/755fd8c8-1c8a-4040-b426-405e8e5789aa-credential-socket\") pod \"755fd8c8-1c8a-4040-b426-405e8e5789aa\" (UID: \"755fd8c8-1c8a-4040-b426-405e8e5789aa\") "
Oct 30 19:04:24.964509 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:24.964356    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/755fd8c8-1c8a-4040-b426-405e8e5789aa-kube-api-access-dchpr\" (UniqueName: \"kubernetes.io/projected/755fd8c8-1c8a-4040-b426-405e8e5789aa-kube-api-access-dchpr\") pod \"755fd8c8-1c8a-4040-b426-405e8e5789aa\" (UID: \"755fd8c8-1c8a-4040-b426-405e8e5789aa\") "
Oct 30 19:04:24.964509 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:24.964400    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/755fd8c8-1c8a-4040-b426-405e8e5789aa-istio-token\" (UniqueName: \"kubernetes.io/projected/755fd8c8-1c8a-4040-b426-405e8e5789aa-istio-token\") pod \"755fd8c8-1c8a-4040-b426-405e8e5789aa\" (UID: \"755fd8c8-1c8a-4040-b426-405e8e5789aa\") "
Oct 30 19:04:24.964509 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:24.964462    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/755fd8c8-1c8a-4040-b426-405e8e5789aa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/755fd8c8-1c8a-4040-b426-405e8e5789aa-istio-envoy\") pod \"755fd8c8-1c8a-4040-b426-405e8e5789aa\" (UID: \"755fd8c8-1c8a-4040-b426-405e8e5789aa\") "
Oct 30 19:04:24.964766 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:24.964553    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/755fd8c8-1c8a-4040-b426-405e8e5789aa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/755fd8c8-1c8a-4040-b426-405e8e5789aa-istio-podinfo\") pod \"755fd8c8-1c8a-4040-b426-405e8e5789aa\" (UID: \"755fd8c8-1c8a-4040-b426-405e8e5789aa\") "
Oct 30 19:04:24.964766 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:24.964634    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/755fd8c8-1c8a-4040-b426-405e8e5789aa-credential-socket" pod "755fd8c8-1c8a-4040-b426-405e8e5789aa" (UID: "755fd8c8-1c8a-4040-b426-405e8e5789aa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:04:24.964879 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:24.964810    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/755fd8c8-1c8a-4040-b426-405e8e5789aa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/755fd8c8-1c8a-4040-b426-405e8e5789aa-istio-data\") pod \"755fd8c8-1c8a-4040-b426-405e8e5789aa\" (UID: \"755fd8c8-1c8a-4040-b426-405e8e5789aa\") "
Oct 30 19:04:24.964940 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:24.964879    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/755fd8c8-1c8a-4040-b426-405e8e5789aa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/755fd8c8-1c8a-4040-b426-405e8e5789aa-workload-certs\") pod \"755fd8c8-1c8a-4040-b426-405e8e5789aa\" (UID: \"755fd8c8-1c8a-4040-b426-405e8e5789aa\") "
Oct 30 19:04:24.965002 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:24.964955    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/755fd8c8-1c8a-4040-b426-405e8e5789aa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/755fd8c8-1c8a-4040-b426-405e8e5789aa-istiod-ca-cert\") pod \"755fd8c8-1c8a-4040-b426-405e8e5789aa\" (UID: \"755fd8c8-1c8a-4040-b426-405e8e5789aa\") "
Oct 30 19:04:24.965060 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:24.965002    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/755fd8c8-1c8a-4040-b426-405e8e5789aa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/755fd8c8-1c8a-4040-b426-405e8e5789aa-workload-socket\") pod \"755fd8c8-1c8a-4040-b426-405e8e5789aa\" (UID: \"755fd8c8-1c8a-4040-b426-405e8e5789aa\") "
Oct 30 19:04:24.965138 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:24.965107    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/755fd8c8-1c8a-4040-b426-405e8e5789aa-istio-data" pod "755fd8c8-1c8a-4040-b426-405e8e5789aa" (UID: "755fd8c8-1c8a-4040-b426-405e8e5789aa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:04:24.965428 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:24.965350    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/755fd8c8-1c8a-4040-b426-405e8e5789aa-istiod-ca-cert" pod "755fd8c8-1c8a-4040-b426-405e8e5789aa" (UID: "755fd8c8-1c8a-4040-b426-405e8e5789aa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:04:24.965428 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:24.965361    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/755fd8c8-1c8a-4040-b426-405e8e5789aa-workload-socket" pod "755fd8c8-1c8a-4040-b426-405e8e5789aa" (UID: "755fd8c8-1c8a-4040-b426-405e8e5789aa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:04:24.965846 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:24.965819    4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/755fd8c8-1c8a-4040-b426-405e8e5789aa-istio-data\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:24.965971 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:24.965849    4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/755fd8c8-1c8a-4040-b426-405e8e5789aa-istiod-ca-cert\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:24.965971 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:24.965863    4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/755fd8c8-1c8a-4040-b426-405e8e5789aa-workload-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:24.965971 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:24.965877    4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/755fd8c8-1c8a-4040-b426-405e8e5789aa-credential-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:24.966182 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:24.966160    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/755fd8c8-1c8a-4040-b426-405e8e5789aa-workload-certs" pod "755fd8c8-1c8a-4040-b426-405e8e5789aa" (UID: "755fd8c8-1c8a-4040-b426-405e8e5789aa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:04:24.967656 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:24.967634    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/755fd8c8-1c8a-4040-b426-405e8e5789aa-kube-api-access-dchpr" pod "755fd8c8-1c8a-4040-b426-405e8e5789aa" (UID: "755fd8c8-1c8a-4040-b426-405e8e5789aa"). InnerVolumeSpecName "kube-api-access-dchpr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:04:24.967656 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:24.967633    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/755fd8c8-1c8a-4040-b426-405e8e5789aa-istio-envoy" pod "755fd8c8-1c8a-4040-b426-405e8e5789aa" (UID: "755fd8c8-1c8a-4040-b426-405e8e5789aa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:04:24.967869 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:24.967711    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/755fd8c8-1c8a-4040-b426-405e8e5789aa-istio-token" pod "755fd8c8-1c8a-4040-b426-405e8e5789aa" (UID: "755fd8c8-1c8a-4040-b426-405e8e5789aa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:04:24.967981 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:24.967960    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/755fd8c8-1c8a-4040-b426-405e8e5789aa-istio-podinfo" pod "755fd8c8-1c8a-4040-b426-405e8e5789aa" (UID: "755fd8c8-1c8a-4040-b426-405e8e5789aa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:04:25.067563 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:25.067458    4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/755fd8c8-1c8a-4040-b426-405e8e5789aa-workload-certs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:25.067563 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:25.067498    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dchpr\" (UniqueName: \"kubernetes.io/projected/755fd8c8-1c8a-4040-b426-405e8e5789aa-kube-api-access-dchpr\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:25.067563 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:25.067515    4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/755fd8c8-1c8a-4040-b426-405e8e5789aa-istio-token\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:25.067563 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:25.067527    4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/755fd8c8-1c8a-4040-b426-405e8e5789aa-istio-envoy\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:25.067563 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:25.067538    4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/755fd8c8-1c8a-4040-b426-405e8e5789aa-istio-podinfo\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:25.150593 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:25.150557    4525 generic.go:356] "Generic (PLEG): container finished" podID="755fd8c8-1c8a-4040-b426-405e8e5789aa" containerID="2dd428a6d3426837292b028d088263871789dfeef3c999fb2ea8c8c5c5ea3aa0" exitCode=0
Oct 30 19:04:25.150810 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:25.150617    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k"
Oct 30 19:04:25.150810 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:25.150636    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k" event={"ID":"755fd8c8-1c8a-4040-b426-405e8e5789aa","Type":"ContainerDied","Data":"2dd428a6d3426837292b028d088263871789dfeef3c999fb2ea8c8c5c5ea3aa0"}
Oct 30 19:04:25.150810 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:25.150730    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k" event={"ID":"755fd8c8-1c8a-4040-b426-405e8e5789aa","Type":"ContainerDied","Data":"72264bf6f4883697d5891848a6dff6ea297e83814766027ced203f83cdac55de"}
Oct 30 19:04:25.150810 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:25.150755    4525 scope.go:122] "RemoveContainer" containerID="2dd428a6d3426837292b028d088263871789dfeef3c999fb2ea8c8c5c5ea3aa0"
Oct 30 19:04:25.152783 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:25.152761    4525 generic.go:356] "Generic (PLEG): container finished" podID="a39e87cd-03ff-4777-bf20-8878edb373fa" containerID="05419d18ddf49c03647a8cc31e5cc4bf21312b6429b493f0750c74633d873431" exitCode=0
Oct 30 19:04:25.152928 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:25.152805    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-xsvss"
Oct 30 19:04:25.152928 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:25.152827    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-xsvss" event={"ID":"a39e87cd-03ff-4777-bf20-8878edb373fa","Type":"ContainerDied","Data":"05419d18ddf49c03647a8cc31e5cc4bf21312b6429b493f0750c74633d873431"}
Oct 30 19:04:25.152928 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:25.152872    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-xsvss" event={"ID":"a39e87cd-03ff-4777-bf20-8878edb373fa","Type":"ContainerDied","Data":"e610cad3fd0c230b298f2f9daf3db165e1942df6323eb9f34aca312672104868"}
Oct 30 19:04:25.168172 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:25.168123    4525 scope.go:122] "RemoveContainer" containerID="a380b91526beebecb86524c7b78d56960e963c0cebf965e7293848ddbb0eb420"
Oct 30 19:04:25.181783 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:25.181755    4525 scope.go:122] "RemoveContainer" containerID="2dd428a6d3426837292b028d088263871789dfeef3c999fb2ea8c8c5c5ea3aa0"
Oct 30 19:04:25.182079 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:04:25.182060    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dd428a6d3426837292b028d088263871789dfeef3c999fb2ea8c8c5c5ea3aa0\": container with ID starting with 2dd428a6d3426837292b028d088263871789dfeef3c999fb2ea8c8c5c5ea3aa0 not found: ID does not exist" containerID="2dd428a6d3426837292b028d088263871789dfeef3c999fb2ea8c8c5c5ea3aa0"
Oct 30 19:04:25.182176 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:25.182092    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dd428a6d3426837292b028d088263871789dfeef3c999fb2ea8c8c5c5ea3aa0"} err="failed to get container status \"2dd428a6d3426837292b028d088263871789dfeef3c999fb2ea8c8c5c5ea3aa0\": rpc error: code = NotFound desc = could not find container \"2dd428a6d3426837292b028d088263871789dfeef3c999fb2ea8c8c5c5ea3aa0\": container with ID starting with 2dd428a6d3426837292b028d088263871789dfeef3c999fb2ea8c8c5c5ea3aa0 not found: ID does not exist"
Oct 30 19:04:25.182176 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:25.182115    4525 scope.go:122] "RemoveContainer" containerID="a380b91526beebecb86524c7b78d56960e963c0cebf965e7293848ddbb0eb420"
Oct 30 19:04:25.182420 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:04:25.182402    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a380b91526beebecb86524c7b78d56960e963c0cebf965e7293848ddbb0eb420\": container with ID starting with a380b91526beebecb86524c7b78d56960e963c0cebf965e7293848ddbb0eb420 not found: ID does not exist" containerID="a380b91526beebecb86524c7b78d56960e963c0cebf965e7293848ddbb0eb420"
Oct 30 19:04:25.182469 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:25.182430    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a380b91526beebecb86524c7b78d56960e963c0cebf965e7293848ddbb0eb420"} err="failed to get container status \"a380b91526beebecb86524c7b78d56960e963c0cebf965e7293848ddbb0eb420\": rpc error: code = NotFound desc = could not find container \"a380b91526beebecb86524c7b78d56960e963c0cebf965e7293848ddbb0eb420\": container with ID starting with a380b91526beebecb86524c7b78d56960e963c0cebf965e7293848ddbb0eb420 not found: ID does not exist"
Oct 30 19:04:25.182469 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:25.182445    4525 scope.go:122] "RemoveContainer" containerID="05419d18ddf49c03647a8cc31e5cc4bf21312b6429b493f0750c74633d873431"
Oct 30 19:04:25.184996 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:25.184974    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k"]
Oct 30 19:04:25.192531 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:25.192510    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-kjx6k"]
Oct 30 19:04:25.194323 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:25.194296    4525 scope.go:122] "RemoveContainer" containerID="05419d18ddf49c03647a8cc31e5cc4bf21312b6429b493f0750c74633d873431"
Oct 30 19:04:25.194657 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:04:25.194634    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05419d18ddf49c03647a8cc31e5cc4bf21312b6429b493f0750c74633d873431\": container with ID starting with 05419d18ddf49c03647a8cc31e5cc4bf21312b6429b493f0750c74633d873431 not found: ID does not exist" containerID="05419d18ddf49c03647a8cc31e5cc4bf21312b6429b493f0750c74633d873431"
Oct 30 19:04:25.194773 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:25.194668    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05419d18ddf49c03647a8cc31e5cc4bf21312b6429b493f0750c74633d873431"} err="failed to get container status \"05419d18ddf49c03647a8cc31e5cc4bf21312b6429b493f0750c74633d873431\": rpc error: code = NotFound desc = could not find container \"05419d18ddf49c03647a8cc31e5cc4bf21312b6429b493f0750c74633d873431\": container with ID starting with 05419d18ddf49c03647a8cc31e5cc4bf21312b6429b493f0750c74633d873431 not found: ID does not exist"
Oct 30 19:04:25.198481 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:25.198457    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-xsvss"]
Oct 30 19:04:25.203798 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:25.203775    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-e9ada0-openshift-default-685f6fdbfc-xsvss"]
Oct 30 19:04:25.545942 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:25.545910    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="755fd8c8-1c8a-4040-b426-405e8e5789aa" path="/var/lib/kubelet/pods/755fd8c8-1c8a-4040-b426-405e8e5789aa/volumes"
Oct 30 19:04:25.546504 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:25.546488    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a39e87cd-03ff-4777-bf20-8878edb373fa" path="/var/lib/kubelet/pods/a39e87cd-03ff-4777-bf20-8878edb373fa/volumes"
Oct 30 19:04:38.059451 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:38.059407    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29e4bbf3-rptrl"]
Oct 30 19:04:38.066937 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:38.066905    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29e4bbf3-rptrl"]
Oct 30 19:04:39.561920 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:04:39.561876    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b712abe-9c69-4d7f-b4d3-c172d6c0298a" path="/var/lib/kubelet/pods/3b712abe-9c69-4d7f-b4d3-c172d6c0298a/volumes"
Oct 30 19:04:47.405568 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:04:47.405525    4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d49l" containerName="extractor"
Oct 30 19:05:01.037767 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:01.037723    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfa695b1-kmg2l"]
Oct 30 19:05:01.046057 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:01.046007    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfa695b1-kmg2l"]
Oct 30 19:05:01.546664 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:01.546629    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="340f330d-01ae-46bd-bc7f-98d8984ef925" path="/var/lib/kubelet/pods/340f330d-01ae-46bd-bc7f-98d8984ef925/volumes"
Oct 30 19:05:13.038224 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:13.038184    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4290dae6-9spnf"]
Oct 30 19:05:13.043289 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:13.043265    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4290dae6-9spnf"]
Oct 30 19:05:13.546213 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:13.546178    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7334df45-994f-4964-acf0-65e1405b42df" path="/var/lib/kubelet/pods/7334df45-994f-4964-acf0-65e1405b42df/volumes"
Oct 30 19:05:14.033340 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:14.033296    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-740adeaf-jpbwv"]
Oct 30 19:05:14.039206 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:14.039175    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-740adeaf-jpbwv"]
Oct 30 19:05:15.546844 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:15.546811    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbcfc0ae-880d-4c9e-a36a-db1659ce4f94" path="/var/lib/kubelet/pods/fbcfc0ae-880d-4c9e-a36a-db1659ce4f94/volumes"
Oct 30 19:05:17.398107 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:05:17.398069    4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d49l" containerName="extractor"
Oct 30 19:05:20.069297 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:20.069262    4525 scope.go:122] "RemoveContainer" containerID="db9d1e2f9b14d90a3167a321cea61b8c5f0b7426b85d0cc2a1676977230054f3"
Oct 30 19:05:24.774980 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:24.774942    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw"]
Oct 30 19:05:24.775910 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:24.775882    4525 cpu_manager.go:414] "containerMap: removing container" podUID="a39e87cd-03ff-4777-bf20-8878edb373fa" containerName="istio-proxy"
Oct 30 19:05:24.775910 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:24.775909    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a39e87cd-03ff-4777-bf20-8878edb373fa" containerName="istio-proxy"
Oct 30 19:05:24.776130 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:24.775927    4525 cpu_manager.go:414] "containerMap: removing container" podUID="755fd8c8-1c8a-4040-b426-405e8e5789aa" containerName="istio-proxy"
Oct 30 19:05:24.776130 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:24.775935    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="755fd8c8-1c8a-4040-b426-405e8e5789aa" containerName="istio-proxy"
Oct 30 19:05:24.776130 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:24.775956    4525 cpu_manager.go:414] "containerMap: removing container" podUID="755fd8c8-1c8a-4040-b426-405e8e5789aa" containerName="istio-proxy"
Oct 30 19:05:24.776130 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:24.775963    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="755fd8c8-1c8a-4040-b426-405e8e5789aa" containerName="istio-proxy"
Oct 30 19:05:24.776130 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:24.775994    4525 cpu_manager.go:414] "containerMap: removing container" podUID="bcf99dfe-605a-4962-aaa4-ec1a6f7f31b1" containerName="revoke-keys"
Oct 30 19:05:24.776130 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:24.776002    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcf99dfe-605a-4962-aaa4-ec1a6f7f31b1" containerName="revoke-keys"
Oct 30 19:05:24.776535 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:24.776160    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="a39e87cd-03ff-4777-bf20-8878edb373fa" containerName="istio-proxy"
Oct 30 19:05:24.776535 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:24.776175    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="755fd8c8-1c8a-4040-b426-405e8e5789aa" containerName="istio-proxy"
Oct 30 19:05:24.776535 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:24.776187    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="755fd8c8-1c8a-4040-b426-405e8e5789aa" containerName="istio-proxy"
Oct 30 19:05:24.776535 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:24.776200    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcf99dfe-605a-4962-aaa4-ec1a6f7f31b1" containerName="revoke-keys"
Oct 30 19:05:24.789476 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:24.789453    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw"]
Oct 30 19:05:24.789722 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:24.789566    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw"
Oct 30 19:05:24.796747 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:24.796492    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-dcd326-openshift-default-dockercfg-hmw94\""
Oct 30 19:05:24.891889 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:24.891855    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3330720a-f130-4cd2-8156-643a79a24235-istio-data\") pod \"e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw\" (UID: \"3330720a-f130-4cd2-8156-643a79a24235\") " pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw"
Oct 30 19:05:24.892061 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:24.891902    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3330720a-f130-4cd2-8156-643a79a24235-credential-socket\") pod \"e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw\" (UID: \"3330720a-f130-4cd2-8156-643a79a24235\") " pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw"
Oct 30 19:05:24.892061 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:24.891969    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3330720a-f130-4cd2-8156-643a79a24235-workload-certs\") pod \"e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw\" (UID: \"3330720a-f130-4cd2-8156-643a79a24235\") " pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw"
Oct 30 19:05:24.892061 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:24.892021    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3330720a-f130-4cd2-8156-643a79a24235-istio-token\") pod \"e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw\" (UID: \"3330720a-f130-4cd2-8156-643a79a24235\") " pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw"
Oct 30 19:05:24.892217 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:24.892102    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k27m6\" (UniqueName: \"kubernetes.io/projected/3330720a-f130-4cd2-8156-643a79a24235-kube-api-access-k27m6\") pod \"e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw\" (UID: \"3330720a-f130-4cd2-8156-643a79a24235\") " pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw"
Oct 30 19:05:24.892323 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:24.892305    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3330720a-f130-4cd2-8156-643a79a24235-istio-envoy\") pod \"e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw\" (UID: \"3330720a-f130-4cd2-8156-643a79a24235\") " pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw"
Oct 30 19:05:24.892388 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:24.892360    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3330720a-f130-4cd2-8156-643a79a24235-workload-socket\") pod \"e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw\" (UID: \"3330720a-f130-4cd2-8156-643a79a24235\") " pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw"
Oct 30 19:05:24.892484 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:24.892470    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3330720a-f130-4cd2-8156-643a79a24235-istio-podinfo\") pod \"e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw\" (UID: \"3330720a-f130-4cd2-8156-643a79a24235\") " pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw"
Oct 30 19:05:24.892584 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:24.892570    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3330720a-f130-4cd2-8156-643a79a24235-istiod-ca-cert\") pod \"e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw\" (UID: \"3330720a-f130-4cd2-8156-643a79a24235\") " pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw"
Oct 30 19:05:24.996108 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:24.994012    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3330720a-f130-4cd2-8156-643a79a24235-istio-data\") pod \"e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw\" (UID: \"3330720a-f130-4cd2-8156-643a79a24235\") " pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw"
Oct 30 19:05:24.996108 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:24.994100    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3330720a-f130-4cd2-8156-643a79a24235-credential-socket\") pod \"e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw\" (UID: \"3330720a-f130-4cd2-8156-643a79a24235\") " pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw"
Oct 30 19:05:24.996108 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:24.994139    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3330720a-f130-4cd2-8156-643a79a24235-workload-certs\") pod \"e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw\" (UID: \"3330720a-f130-4cd2-8156-643a79a24235\") " pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw"
Oct 30 19:05:24.996108 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:24.994186    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3330720a-f130-4cd2-8156-643a79a24235-istio-token\") pod \"e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw\" (UID: \"3330720a-f130-4cd2-8156-643a79a24235\") " pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw"
Oct 30 19:05:24.996108 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:24.994219    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k27m6\" (UniqueName: \"kubernetes.io/projected/3330720a-f130-4cd2-8156-643a79a24235-kube-api-access-k27m6\") pod \"e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw\" (UID: \"3330720a-f130-4cd2-8156-643a79a24235\") " pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw"
Oct 30 19:05:24.996108 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:24.994442    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3330720a-f130-4cd2-8156-643a79a24235-istio-envoy\") pod \"e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw\" (UID: \"3330720a-f130-4cd2-8156-643a79a24235\") " pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw"
Oct 30 19:05:24.996108 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:24.994507    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3330720a-f130-4cd2-8156-643a79a24235-workload-socket\") pod \"e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw\" (UID: \"3330720a-f130-4cd2-8156-643a79a24235\") " pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw"
Oct 30 19:05:24.996108 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:24.994610    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3330720a-f130-4cd2-8156-643a79a24235-istio-podinfo\") pod \"e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw\" (UID: \"3330720a-f130-4cd2-8156-643a79a24235\") " pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw"
Oct 30 19:05:24.996108 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:24.994825    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3330720a-f130-4cd2-8156-643a79a24235-istiod-ca-cert\") pod \"e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw\" (UID: \"3330720a-f130-4cd2-8156-643a79a24235\") " pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw"
Oct 30 19:05:24.996108 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:24.995811    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3330720a-f130-4cd2-8156-643a79a24235-istiod-ca-cert\") pod \"e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw\" (UID: \"3330720a-f130-4cd2-8156-643a79a24235\") " pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw"
Oct 30 19:05:24.996781 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:24.996164    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3330720a-f130-4cd2-8156-643a79a24235-istio-data\") pod \"e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw\" (UID: \"3330720a-f130-4cd2-8156-643a79a24235\") " pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw"
Oct 30 19:05:25.013704 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:25.007972    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3330720a-f130-4cd2-8156-643a79a24235-credential-socket\") pod \"e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw\" (UID: \"3330720a-f130-4cd2-8156-643a79a24235\") " pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw"
Oct 30 19:05:25.013704 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:25.008517    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3330720a-f130-4cd2-8156-643a79a24235-workload-certs\") pod \"e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw\" (UID: \"3330720a-f130-4cd2-8156-643a79a24235\") " pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw"
Oct 30 19:05:25.013704 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:25.009052    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3330720a-f130-4cd2-8156-643a79a24235-workload-socket\") pod \"e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw\" (UID: \"3330720a-f130-4cd2-8156-643a79a24235\") " pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw"
Oct 30 19:05:25.036748 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:25.029929    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3330720a-f130-4cd2-8156-643a79a24235-istio-envoy\") pod \"e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw\" (UID: \"3330720a-f130-4cd2-8156-643a79a24235\") " pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw"
Oct 30 19:05:25.048255 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:25.048193    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3330720a-f130-4cd2-8156-643a79a24235-istio-podinfo\") pod \"e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw\" (UID: \"3330720a-f130-4cd2-8156-643a79a24235\") " pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw"
Oct 30 19:05:25.068767 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:25.058742    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k27m6\" (UniqueName: \"kubernetes.io/projected/3330720a-f130-4cd2-8156-643a79a24235-kube-api-access-k27m6\") pod \"e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw\" (UID: \"3330720a-f130-4cd2-8156-643a79a24235\") " pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw"
Oct 30 19:05:25.079563 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:25.074264    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3330720a-f130-4cd2-8156-643a79a24235-istio-token\") pod \"e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw\" (UID: \"3330720a-f130-4cd2-8156-643a79a24235\") " pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw"
Oct 30 19:05:25.117701 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:25.117043    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw"
Oct 30 19:05:25.502854 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:05:25.502823    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3330720a_f130_4cd2_8156_643a79a24235.slice/crio-c83d9ca472d9d0b63c49603a86297e718e5701ec0664f0efbf8ccbdc50fce136 WatchSource:0}: Error finding container c83d9ca472d9d0b63c49603a86297e718e5701ec0664f0efbf8ccbdc50fce136: Status 404 returned error can't find the container with id c83d9ca472d9d0b63c49603a86297e718e5701ec0664f0efbf8ccbdc50fce136
Oct 30 19:05:25.503298 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:25.503278    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw"]
Oct 30 19:05:25.505597 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:25.505564    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:05:25.505716 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:25.505627    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:05:25.505716 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:25.505657    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:05:26.030945 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:26.030911    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5ec3983-sb9z7"]
Oct 30 19:05:26.036039 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:26.036012    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5ec3983-sb9z7"]
Oct 30 19:05:26.046036 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:26.046001    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw" event={"ID":"3330720a-f130-4cd2-8156-643a79a24235","Type":"ContainerStarted","Data":"8f922ecb3e1d7b4a7c194ed789ba3fbba1a01f20884f74726110792f014f4b2e"}
Oct 30 19:05:26.046186 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:26.046083    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw" event={"ID":"3330720a-f130-4cd2-8156-643a79a24235","Type":"ContainerStarted","Data":"c83d9ca472d9d0b63c49603a86297e718e5701ec0664f0efbf8ccbdc50fce136"}
Oct 30 19:05:26.066493 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:26.066435    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw" podStartSLOduration=2.066422764 podStartE2EDuration="2.066422764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:05:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:05:26.064768479 +0000 UTC m=+1748.690438443" watchObservedRunningTime="2026-10-30 19:05:26.066422764 +0000 UTC m=+1748.692092808"
Oct 30 19:05:26.118101 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:26.118069    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw"
Oct 30 19:05:26.124366 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:26.124344    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw"
Oct 30 19:05:27.025077 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:27.025042    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fcad69e-99x4r"]
Oct 30 19:05:27.029786 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:27.029764    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fcad69e-99x4r"]
Oct 30 19:05:27.054984 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:27.054957    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw"
Oct 30 19:05:27.056074 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:27.056054    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw"
Oct 30 19:05:27.547879 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:27.547846    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="128e7ce4-6afa-400b-9585-d5f112f2e102" path="/var/lib/kubelet/pods/128e7ce4-6afa-400b-9585-d5f112f2e102/volumes"
Oct 30 19:05:27.548408 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:27.548393    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f98b875e-ea67-402e-a050-67e67f925367" path="/var/lib/kubelet/pods/f98b875e-ea67-402e-a050-67e67f925367/volumes"
Oct 30 19:05:31.056512 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:31.056432    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f9b7116-4ntcq"]
Oct 30 19:05:31.066772 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:31.066735    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f9b7116-4ntcq"]
Oct 30 19:05:31.546048 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:31.546007    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03ce656d-ce81-4db9-b834-3d3ea7be50b2" path="/var/lib/kubelet/pods/03ce656d-ce81-4db9-b834-3d3ea7be50b2/volumes"
Oct 30 19:05:40.081021 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:40.080986    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ae07349-gg5jk"]
Oct 30 19:05:40.091859 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:40.091823    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd5b9f09-7j65v"]
Oct 30 19:05:40.097356 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:40.097324    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ae07349-gg5jk"]
Oct 30 19:05:40.102050 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:40.102021    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd5b9f09-7j65v"]
Oct 30 19:05:41.545884 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:41.545853    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e658646-ee3c-4e32-8ec1-f4572b96b1b2" path="/var/lib/kubelet/pods/7e658646-ee3c-4e32-8ec1-f4572b96b1b2/volumes"
Oct 30 19:05:41.546377 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:41.546362    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9a64a60-8374-4a31-a87f-a35e80c0e6af" path="/var/lib/kubelet/pods/a9a64a60-8374-4a31-a87f-a35e80c0e6af/volumes"
Oct 30 19:05:47.453849 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:05:47.452878    4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d49l" containerName="extractor"
Oct 30 19:05:48.025267 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:48.025225    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0a29621-rmzp2"]
Oct 30 19:05:48.030045 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:48.030010    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0a29621-rmzp2"]
Oct 30 19:05:49.030955 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:49.030906    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db33a84d-5x889"]
Oct 30 19:05:49.035752 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:49.035723    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db33a84d-5x889"]
Oct 30 19:05:49.565937 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:49.565895    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="107a0724-9d25-4b33-9866-e689ceb95234" path="/var/lib/kubelet/pods/107a0724-9d25-4b33-9866-e689ceb95234/volumes"
Oct 30 19:05:49.567077 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:49.567051    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b248c08-1780-454f-9ef9-596fb0cfb250" path="/var/lib/kubelet/pods/1b248c08-1780-454f-9ef9-596fb0cfb250/volumes"
Oct 30 19:05:50.031998 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:50.031967    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9073694d-9dlpm"]
Oct 30 19:05:50.052245 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:50.052210    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9073694d-9dlpm"]
Oct 30 19:05:51.546619 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:51.546582    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e47d550d-c724-482c-9352-077d0d9e17c9" path="/var/lib/kubelet/pods/e47d550d-c724-482c-9352-077d0d9e17c9/volumes"
Oct 30 19:05:54.432941 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:54.432901    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-46385c-openshift-default-6ccdb978d5-4gmh4"]
Oct 30 19:05:54.439558 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:54.439533    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-46385c-openshift-default-6ccdb978d5-4gmh4"
Oct 30 19:05:54.446829 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:54.446800    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-46385c-openshift-default-6ccdb978d5-4gmh4"]
Oct 30 19:05:54.449354 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:54.449332    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-46385c-openshift-default-dockercfg-qfc56\""
Oct 30 19:05:54.529328 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:54.522999    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ec64e5e-2c2b-47d8-8070-296ce41ada09-credential-socket\") pod \"e2e-shared-b-w1-46385c-openshift-default-6ccdb978d5-4gmh4\" (UID: \"7ec64e5e-2c2b-47d8-8070-296ce41ada09\") " pod="openshift-ingress/e2e-shared-b-w1-46385c-openshift-default-6ccdb978d5-4gmh4"
Oct 30 19:05:54.529328 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:54.523183    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ec64e5e-2c2b-47d8-8070-296ce41ada09-workload-certs\") pod \"e2e-shared-b-w1-46385c-openshift-default-6ccdb978d5-4gmh4\" (UID: \"7ec64e5e-2c2b-47d8-8070-296ce41ada09\") " pod="openshift-ingress/e2e-shared-b-w1-46385c-openshift-default-6ccdb978d5-4gmh4"
Oct 30 19:05:54.529328 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:54.523282    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ec64e5e-2c2b-47d8-8070-296ce41ada09-workload-socket\") pod \"e2e-shared-b-w1-46385c-openshift-default-6ccdb978d5-4gmh4\" (UID: \"7ec64e5e-2c2b-47d8-8070-296ce41ada09\") " pod="openshift-ingress/e2e-shared-b-w1-46385c-openshift-default-6ccdb978d5-4gmh4"
Oct 30 19:05:54.529328 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:54.523312    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sncml\" (UniqueName: \"kubernetes.io/projected/7ec64e5e-2c2b-47d8-8070-296ce41ada09-kube-api-access-sncml\") pod \"e2e-shared-b-w1-46385c-openshift-default-6ccdb978d5-4gmh4\" (UID: \"7ec64e5e-2c2b-47d8-8070-296ce41ada09\") " pod="openshift-ingress/e2e-shared-b-w1-46385c-openshift-default-6ccdb978d5-4gmh4"
Oct 30 19:05:54.529328 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:54.523463    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ec64e5e-2c2b-47d8-8070-296ce41ada09-istiod-ca-cert\") pod \"e2e-shared-b-w1-46385c-openshift-default-6ccdb978d5-4gmh4\" (UID: \"7ec64e5e-2c2b-47d8-8070-296ce41ada09\") " pod="openshift-ingress/e2e-shared-b-w1-46385c-openshift-default-6ccdb978d5-4gmh4"
Oct 30 19:05:54.529328 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:54.523532    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ec64e5e-2c2b-47d8-8070-296ce41ada09-istio-podinfo\") pod \"e2e-shared-b-w1-46385c-openshift-default-6ccdb978d5-4gmh4\" (UID: \"7ec64e5e-2c2b-47d8-8070-296ce41ada09\") " pod="openshift-ingress/e2e-shared-b-w1-46385c-openshift-default-6ccdb978d5-4gmh4"
Oct 30 19:05:54.529328 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:54.523642    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ec64e5e-2c2b-47d8-8070-296ce41ada09-istio-token\") pod \"e2e-shared-b-w1-46385c-openshift-default-6ccdb978d5-4gmh4\" (UID: \"7ec64e5e-2c2b-47d8-8070-296ce41ada09\") " pod="openshift-ingress/e2e-shared-b-w1-46385c-openshift-default-6ccdb978d5-4gmh4"
Oct 30 19:05:54.529328 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:54.523740    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ec64e5e-2c2b-47d8-8070-296ce41ada09-istio-envoy\") pod \"e2e-shared-b-w1-46385c-openshift-default-6ccdb978d5-4gmh4\" (UID: \"7ec64e5e-2c2b-47d8-8070-296ce41ada09\") " pod="openshift-ingress/e2e-shared-b-w1-46385c-openshift-default-6ccdb978d5-4gmh4"
Oct 30 19:05:54.529328 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:54.523842    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ec64e5e-2c2b-47d8-8070-296ce41ada09-istio-data\") pod \"e2e-shared-b-w1-46385c-openshift-default-6ccdb978d5-4gmh4\" (UID: \"7ec64e5e-2c2b-47d8-8070-296ce41ada09\") " pod="openshift-ingress/e2e-shared-b-w1-46385c-openshift-default-6ccdb978d5-4gmh4"
Oct 30 19:05:54.632707 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:54.626736    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ec64e5e-2c2b-47d8-8070-296ce41ada09-istio-token\") pod \"e2e-shared-b-w1-46385c-openshift-default-6ccdb978d5-4gmh4\" (UID: \"7ec64e5e-2c2b-47d8-8070-296ce41ada09\") " pod="openshift-ingress/e2e-shared-b-w1-46385c-openshift-default-6ccdb978d5-4gmh4"
Oct 30 19:05:54.632707 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:54.626820    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ec64e5e-2c2b-47d8-8070-296ce41ada09-istio-envoy\") pod \"e2e-shared-b-w1-46385c-openshift-default-6ccdb978d5-4gmh4\" (UID: \"7ec64e5e-2c2b-47d8-8070-296ce41ada09\") " pod="openshift-ingress/e2e-shared-b-w1-46385c-openshift-default-6ccdb978d5-4gmh4"
Oct 30 19:05:54.632707 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:54.626925    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ec64e5e-2c2b-47d8-8070-296ce41ada09-istio-data\") pod \"e2e-shared-b-w1-46385c-openshift-default-6ccdb978d5-4gmh4\" (UID: \"7ec64e5e-2c2b-47d8-8070-296ce41ada09\") " pod="openshift-ingress/e2e-shared-b-w1-46385c-openshift-default-6ccdb978d5-4gmh4"
Oct 30 19:05:54.632707 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:54.626988    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ec64e5e-2c2b-47d8-8070-296ce41ada09-credential-socket\") pod \"e2e-shared-b-w1-46385c-openshift-default-6ccdb978d5-4gmh4\" (UID: \"7ec64e5e-2c2b-47d8-8070-296ce41ada09\") " pod="openshift-ingress/e2e-shared-b-w1-46385c-openshift-default-6ccdb978d5-4gmh4"
Oct 30 19:05:54.632707 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:54.627057    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ec64e5e-2c2b-47d8-8070-296ce41ada09-workload-certs\") pod \"e2e-shared-b-w1-46385c-openshift-default-6ccdb978d5-4gmh4\" (UID: \"7ec64e5e-2c2b-47d8-8070-296ce41ada09\") " pod="openshift-ingress/e2e-shared-b-w1-46385c-openshift-default-6ccdb978d5-4gmh4"
Oct 30 19:05:54.632707 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:54.627130    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ec64e5e-2c2b-47d8-8070-296ce41ada09-workload-socket\") pod \"e2e-shared-b-w1-46385c-openshift-default-6ccdb978d5-4gmh4\" (UID: \"7ec64e5e-2c2b-47d8-8070-296ce41ada09\") " pod="openshift-ingress/e2e-shared-b-w1-46385c-openshift-default-6ccdb978d5-4gmh4"
Oct 30 19:05:54.632707 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:54.627159    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sncml\" (UniqueName: \"kubernetes.io/projected/7ec64e5e-2c2b-47d8-8070-296ce41ada09-kube-api-access-sncml\") pod \"e2e-shared-b-w1-46385c-openshift-default-6ccdb978d5-4gmh4\" (UID: \"7ec64e5e-2c2b-47d8-8070-296ce41ada09\") " pod="openshift-ingress/e2e-shared-b-w1-46385c-openshift-default-6ccdb978d5-4gmh4"
Oct 30 19:05:54.632707 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:54.627284    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ec64e5e-2c2b-47d8-8070-296ce41ada09-istiod-ca-cert\") pod \"e2e-shared-b-w1-46385c-openshift-default-6ccdb978d5-4gmh4\" (UID: \"7ec64e5e-2c2b-47d8-8070-296ce41ada09\") " pod="openshift-ingress/e2e-shared-b-w1-46385c-openshift-default-6ccdb978d5-4gmh4"
Oct 30 19:05:54.632707 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:54.627347    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ec64e5e-2c2b-47d8-8070-296ce41ada09-istio-podinfo\") pod \"e2e-shared-b-w1-46385c-openshift-default-6ccdb978d5-4gmh4\" (UID: \"7ec64e5e-2c2b-47d8-8070-296ce41ada09\") " pod="openshift-ingress/e2e-shared-b-w1-46385c-openshift-default-6ccdb978d5-4gmh4"
Oct 30 19:05:54.632707 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:54.628292    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ec64e5e-2c2b-47d8-8070-296ce41ada09-credential-socket\") pod \"e2e-shared-b-w1-46385c-openshift-default-6ccdb978d5-4gmh4\" (UID: \"7ec64e5e-2c2b-47d8-8070-296ce41ada09\") " pod="openshift-ingress/e2e-shared-b-w1-46385c-openshift-default-6ccdb978d5-4gmh4"
Oct 30 19:05:54.632707 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:54.629212    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ec64e5e-2c2b-47d8-8070-296ce41ada09-workload-socket\") pod \"e2e-shared-b-w1-46385c-openshift-default-6ccdb978d5-4gmh4\" (UID: \"7ec64e5e-2c2b-47d8-8070-296ce41ada09\") " pod="openshift-ingress/e2e-shared-b-w1-46385c-openshift-default-6ccdb978d5-4gmh4"
Oct 30 19:05:54.632707 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:54.629471    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ec64e5e-2c2b-47d8-8070-296ce41ada09-workload-certs\") pod \"e2e-shared-b-w1-46385c-openshift-default-6ccdb978d5-4gmh4\" (UID: \"7ec64e5e-2c2b-47d8-8070-296ce41ada09\") " pod="openshift-ingress/e2e-shared-b-w1-46385c-openshift-default-6ccdb978d5-4gmh4"
Oct 30 19:05:54.632707 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:54.630302    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ec64e5e-2c2b-47d8-8070-296ce41ada09-istio-data\") pod \"e2e-shared-b-w1-46385c-openshift-default-6ccdb978d5-4gmh4\" (UID: \"7ec64e5e-2c2b-47d8-8070-296ce41ada09\") " pod="openshift-ingress/e2e-shared-b-w1-46385c-openshift-default-6ccdb978d5-4gmh4"
Oct 30 19:05:54.632707 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:54.630962    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ec64e5e-2c2b-47d8-8070-296ce41ada09-istiod-ca-cert\") pod \"e2e-shared-b-w1-46385c-openshift-default-6ccdb978d5-4gmh4\" (UID: \"7ec64e5e-2c2b-47d8-8070-296ce41ada09\") " pod="openshift-ingress/e2e-shared-b-w1-46385c-openshift-default-6ccdb978d5-4gmh4"
Oct 30 19:05:54.688752 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:54.683465    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ec64e5e-2c2b-47d8-8070-296ce41ada09-istio-podinfo\") pod \"e2e-shared-b-w1-46385c-openshift-default-6ccdb978d5-4gmh4\" (UID: \"7ec64e5e-2c2b-47d8-8070-296ce41ada09\") " pod="openshift-ingress/e2e-shared-b-w1-46385c-openshift-default-6ccdb978d5-4gmh4"
Oct 30 19:05:54.688752 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:54.686180    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sncml\" (UniqueName: \"kubernetes.io/projected/7ec64e5e-2c2b-47d8-8070-296ce41ada09-kube-api-access-sncml\") pod \"e2e-shared-b-w1-46385c-openshift-default-6ccdb978d5-4gmh4\" (UID: \"7ec64e5e-2c2b-47d8-8070-296ce41ada09\") " pod="openshift-ingress/e2e-shared-b-w1-46385c-openshift-default-6ccdb978d5-4gmh4"
Oct 30 19:05:54.688752 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:54.687607    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ec64e5e-2c2b-47d8-8070-296ce41ada09-istio-envoy\") pod \"e2e-shared-b-w1-46385c-openshift-default-6ccdb978d5-4gmh4\" (UID: \"7ec64e5e-2c2b-47d8-8070-296ce41ada09\") " pod="openshift-ingress/e2e-shared-b-w1-46385c-openshift-default-6ccdb978d5-4gmh4"
Oct 30 19:05:54.694262 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:54.694233    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ec64e5e-2c2b-47d8-8070-296ce41ada09-istio-token\") pod \"e2e-shared-b-w1-46385c-openshift-default-6ccdb978d5-4gmh4\" (UID: \"7ec64e5e-2c2b-47d8-8070-296ce41ada09\") " pod="openshift-ingress/e2e-shared-b-w1-46385c-openshift-default-6ccdb978d5-4gmh4"
Oct 30 19:05:54.763715 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:54.763337    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-46385c-openshift-default-6ccdb978d5-4gmh4"
Oct 30 19:05:55.131222 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:05:55.131178    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ec64e5e_2c2b_47d8_8070_296ce41ada09.slice/crio-b4ee24bd41e3c25e7d8fec1ba1ffe7abf2e47ef0393452146a58fdbfcc4f4d28 WatchSource:0}: Error finding container b4ee24bd41e3c25e7d8fec1ba1ffe7abf2e47ef0393452146a58fdbfcc4f4d28: Status 404 returned error can't find the container with id b4ee24bd41e3c25e7d8fec1ba1ffe7abf2e47ef0393452146a58fdbfcc4f4d28
Oct 30 19:05:55.131973 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:55.131946    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-46385c-openshift-default-6ccdb978d5-4gmh4"]
Oct 30 19:05:55.134153 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:55.134115    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:05:55.134242 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:55.134187    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:05:55.134242 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:55.134223    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:05:55.341578 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:55.341540    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-46385c-openshift-default-6ccdb978d5-4gmh4" event={"ID":"7ec64e5e-2c2b-47d8-8070-296ce41ada09","Type":"ContainerStarted","Data":"6370674b13dd336f02fd228a6ea078f74a5fbbdef9f26eedd9354ada3ed2fdbd"}
Oct 30 19:05:55.341828 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:55.341599    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-46385c-openshift-default-6ccdb978d5-4gmh4" event={"ID":"7ec64e5e-2c2b-47d8-8070-296ce41ada09","Type":"ContainerStarted","Data":"b4ee24bd41e3c25e7d8fec1ba1ffe7abf2e47ef0393452146a58fdbfcc4f4d28"}
Oct 30 19:05:55.368372 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:55.368316    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-46385c-openshift-default-6ccdb978d5-4gmh4" podStartSLOduration=1.368298714 podStartE2EDuration="1.368298714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:05:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:05:55.363416975 +0000 UTC m=+1777.989086941" watchObservedRunningTime="2026-10-30 19:05:55.368298714 +0000 UTC m=+1777.993968678"
Oct 30 19:05:55.764187 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:55.764154    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-46385c-openshift-default-6ccdb978d5-4gmh4"
Oct 30 19:05:56.770383 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:56.770354    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-46385c-openshift-default-6ccdb978d5-4gmh4"
Oct 30 19:05:57.382771 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:57.382735    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-46385c-openshift-default-6ccdb978d5-4gmh4"
Oct 30 19:05:57.383927 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:05:57.383894    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-46385c-openshift-default-6ccdb978d5-4gmh4"
Oct 30 19:06:08.036057 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:08.036015    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bc6e719-7xvb9"]
Oct 30 19:06:08.048027 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:08.047978    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bc6e719-7xvb9"]
Oct 30 19:06:09.028028 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:09.027996    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca415337-wwhn8"]
Oct 30 19:06:09.032511 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:09.032481    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca415337-wwhn8"]
Oct 30 19:06:09.546674 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:09.546642    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d59f7bbb-4eda-4e43-bb21-e2359fc2688c" path="/var/lib/kubelet/pods/d59f7bbb-4eda-4e43-bb21-e2359fc2688c/volumes"
Oct 30 19:06:09.547259 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:09.547241    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9aba5b9-263f-40b1-9543-ef813ebdc852" path="/var/lib/kubelet/pods/d9aba5b9-263f-40b1-9543-ef813ebdc852/volumes"
Oct 30 19:06:13.027964 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:13.027928    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f4588e8-qs7k2"]
Oct 30 19:06:13.032710 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:13.032664    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f4588e8-qs7k2"]
Oct 30 19:06:13.096877 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:13.096831    4525 patch_prober.go:29] interesting pod/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:06:13.097081 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:13.096902    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw" podUID="3330720a-f130-4cd2-8156-643a79a24235" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:06:13.549279 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:13.549244    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c4c61cc-e88a-405f-908d-ae577af9b93a" path="/var/lib/kubelet/pods/8c4c61cc-e88a-405f-908d-ae577af9b93a/volumes"
Oct 30 19:06:14.678944 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:14.678905    4525 generic.go:356] "Generic (PLEG): container finished" podID="3330720a-f130-4cd2-8156-643a79a24235" containerID="8f922ecb3e1d7b4a7c194ed789ba3fbba1a01f20884f74726110792f014f4b2e" exitCode=0
Oct 30 19:06:14.679409 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:14.678972    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw" event={"ID":"3330720a-f130-4cd2-8156-643a79a24235","Type":"ContainerDied","Data":"8f922ecb3e1d7b4a7c194ed789ba3fbba1a01f20884f74726110792f014f4b2e"}
Oct 30 19:06:14.679532 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:14.679509    4525 scope.go:122] "RemoveContainer" containerID="8f922ecb3e1d7b4a7c194ed789ba3fbba1a01f20884f74726110792f014f4b2e"
Oct 30 19:06:14.681640 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:14.681609    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:06:14.681761 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:14.681708    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:06:14.681761 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:14.681749    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:06:15.711109 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:15.711072    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw" event={"ID":"3330720a-f130-4cd2-8156-643a79a24235","Type":"ContainerStarted","Data":"f2e5fb680cdbedf57e345dcaca5ac439e1d270920ec383be3498906e46da14c6"}
Oct 30 19:06:16.119012 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:16.118979    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw"
Oct 30 19:06:16.119240 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:16.119214    4525 patch_prober.go:29] interesting pod/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Oct 30 19:06:16.119301 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:16.119275    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw" podUID="3330720a-f130-4cd2-8156-643a79a24235" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Oct 30 19:06:17.118048 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:17.117998    4525 patch_prober.go:29] interesting pod/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Oct 30 19:06:17.118487 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:17.118076    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw" podUID="3330720a-f130-4cd2-8156-643a79a24235" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Oct 30 19:06:17.436335 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:06:17.436241    4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d49l" containerName="extractor"
Oct 30 19:06:18.138704 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:18.131272    4525 patch_prober.go:29] interesting pod/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Oct 30 19:06:18.138704 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:18.131353    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw" podUID="3330720a-f130-4cd2-8156-643a79a24235" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Oct 30 19:06:19.121707 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:19.120044    4525 patch_prober.go:29] interesting pod/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Oct 30 19:06:19.121707 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:19.120121    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw" podUID="3330720a-f130-4cd2-8156-643a79a24235" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Oct 30 19:06:20.035618 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:20.035578    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c5f4bd6-9vh7q"]
Oct 30 19:06:20.040089 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:20.040062    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c5f4bd6-9vh7q"]
Oct 30 19:06:20.117746 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:20.116618    4525 scope.go:122] "RemoveContainer" containerID="8f96b96ded4db1e45ad05caaa6c466c3db7a8706d67eb394790ec79c30e1fab9"
Oct 30 19:06:20.120554 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:20.120530    4525 patch_prober.go:29] interesting pod/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Oct 30 19:06:20.120796 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:20.120769    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw" podUID="3330720a-f130-4cd2-8156-643a79a24235" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Oct 30 19:06:20.136018 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:20.135998    4525 scope.go:122] "RemoveContainer" containerID="92d0af373283536101e96497ed13431ce734b7d159aa203d01b397382259a142"
Oct 30 19:06:20.148127 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:20.148104    4525 scope.go:122] "RemoveContainer" containerID="787ec368354820bad674908a98df2bed483589dba61c803f3e6858a66afc4635"
Oct 30 19:06:20.160534 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:20.160512    4525 scope.go:122] "RemoveContainer" containerID="06aca8bce78f98b54b68c884c2164a558e7773c0463ce67f4619ec9cff50a933"
Oct 30 19:06:20.171359 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:20.171338    4525 scope.go:122] "RemoveContainer" containerID="a0e5783c4a80866da15737955cb0ab2f55553649def9edeababbf22b46766061"
Oct 30 19:06:20.208611 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:20.208587    4525 scope.go:122] "RemoveContainer" containerID="c4106c7b4c2dde0616ce890ec4819ceb3a4da192cc5f73e947839648e05ff770"
Oct 30 19:06:20.224951 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:20.223309    4525 scope.go:122] "RemoveContainer" containerID="99f0dfb1bddd00e89c7fa39da1789ef5fe9219f18c2acb5646587066e3c0c674"
Oct 30 19:06:20.239163 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:20.239137    4525 scope.go:122] "RemoveContainer" containerID="7e1d41d6537275ae19e87c1526a5e553481a55d49fd89ea5a5448b5c86a180f9"
Oct 30 19:06:20.251294 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:20.251270    4525 scope.go:122] "RemoveContainer" containerID="44b3dab77ff842ffa8aefe7a4f8dc1e80159ce8353109298f80738cd5c24a4c2"
Oct 30 19:06:20.300950 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:20.300910    4525 scope.go:122] "RemoveContainer" containerID="2620ccbfb4d896807bc5ced5bafea6d411ac5b4fd080007a7cd8cafdbea9cbdf"
Oct 30 19:06:21.027702 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:21.027367    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50605e61-5dwzf"]
Oct 30 19:06:21.036857 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:21.036507    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50605e61-5dwzf"
Oct 30 19:06:21.040398 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:21.040370    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:06:21.040917 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:21.040875    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94hcn\""
Oct 30 19:06:21.041535 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:21.041506    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50605e61-5dwzf"]
Oct 30 19:06:21.118485 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:21.118440    4525 patch_prober.go:29] interesting pod/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Oct 30 19:06:21.118670 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:21.118533    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw" podUID="3330720a-f130-4cd2-8156-643a79a24235" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Oct 30 19:06:21.152648 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:21.151789    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vkml\" (UniqueName: \"kubernetes.io/projected/e87cddb0-df80-4259-a332-3f3ea69f431a-kube-api-access-8vkml\") pod \"maas-api-revoke-sub-50605e61-5dwzf\" (UID: \"e87cddb0-df80-4259-a332-3f3ea69f431a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50605e61-5dwzf"
Oct 30 19:06:21.152648 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:21.152131    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e87cddb0-df80-4259-a332-3f3ea69f431a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50605e61-5dwzf\" (UID: \"e87cddb0-df80-4259-a332-3f3ea69f431a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50605e61-5dwzf"
Oct 30 19:06:21.152648 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:21.152179    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e87cddb0-df80-4259-a332-3f3ea69f431a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50605e61-5dwzf\" (UID: \"e87cddb0-df80-4259-a332-3f3ea69f431a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50605e61-5dwzf"
Oct 30 19:06:21.253423 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:21.253380    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e87cddb0-df80-4259-a332-3f3ea69f431a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50605e61-5dwzf\" (UID: \"e87cddb0-df80-4259-a332-3f3ea69f431a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50605e61-5dwzf"
Oct 30 19:06:21.253605 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:21.253452    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e87cddb0-df80-4259-a332-3f3ea69f431a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50605e61-5dwzf\" (UID: \"e87cddb0-df80-4259-a332-3f3ea69f431a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50605e61-5dwzf"
Oct 30 19:06:21.253605 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:21.253580    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vkml\" (UniqueName: \"kubernetes.io/projected/e87cddb0-df80-4259-a332-3f3ea69f431a-kube-api-access-8vkml\") pod \"maas-api-revoke-sub-50605e61-5dwzf\" (UID: \"e87cddb0-df80-4259-a332-3f3ea69f431a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50605e61-5dwzf"
Oct 30 19:06:21.254196 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:21.254167    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e87cddb0-df80-4259-a332-3f3ea69f431a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50605e61-5dwzf\" (UID: \"e87cddb0-df80-4259-a332-3f3ea69f431a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50605e61-5dwzf"
Oct 30 19:06:21.256380 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:21.256355    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e87cddb0-df80-4259-a332-3f3ea69f431a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50605e61-5dwzf\" (UID: \"e87cddb0-df80-4259-a332-3f3ea69f431a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50605e61-5dwzf"
Oct 30 19:06:21.265634 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:21.265605    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vkml\" (UniqueName: \"kubernetes.io/projected/e87cddb0-df80-4259-a332-3f3ea69f431a-kube-api-access-8vkml\") pod \"maas-api-revoke-sub-50605e61-5dwzf\" (UID: \"e87cddb0-df80-4259-a332-3f3ea69f431a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50605e61-5dwzf"
Oct 30 19:06:21.356921 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:21.356891    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50605e61-5dwzf"
Oct 30 19:06:21.460499 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:21.459730    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e92d4a69-pqvbh"]
Oct 30 19:06:21.471560 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:21.470814    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e92d4a69-pqvbh"]
Oct 30 19:06:21.471560 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:21.470938    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e92d4a69-pqvbh"
Oct 30 19:06:21.553586 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:21.553553    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2865be82-0924-44fa-b215-28622b0957d3" path="/var/lib/kubelet/pods/2865be82-0924-44fa-b215-28622b0957d3/volumes"
Oct 30 19:06:21.562886 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:21.562852    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4qmx\" (UniqueName: \"kubernetes.io/projected/6d5589b4-0028-4529-91d8-5ef4b4684476-kube-api-access-n4qmx\") pod \"maas-api-revoke-sub-e92d4a69-pqvbh\" (UID: \"6d5589b4-0028-4529-91d8-5ef4b4684476\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e92d4a69-pqvbh"
Oct 30 19:06:21.563064 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:21.563045    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d5589b4-0028-4529-91d8-5ef4b4684476-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e92d4a69-pqvbh\" (UID: \"6d5589b4-0028-4529-91d8-5ef4b4684476\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e92d4a69-pqvbh"
Oct 30 19:06:21.563124 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:21.563111    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d5589b4-0028-4529-91d8-5ef4b4684476-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e92d4a69-pqvbh\" (UID: \"6d5589b4-0028-4529-91d8-5ef4b4684476\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e92d4a69-pqvbh"
Oct 30 19:06:21.655221 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:21.655185    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50605e61-5dwzf"]
Oct 30 19:06:21.655630 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:06:21.655588    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode87cddb0_df80_4259_a332_3f3ea69f431a.slice/crio-eb69d14fa43751530945221110977ca6b7b0cea5f5c1c068a8c5ba72e09bdd78 WatchSource:0}: Error finding container eb69d14fa43751530945221110977ca6b7b0cea5f5c1c068a8c5ba72e09bdd78: Status 404 returned error can't find the container with id eb69d14fa43751530945221110977ca6b7b0cea5f5c1c068a8c5ba72e09bdd78
Oct 30 19:06:21.663976 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:21.663953    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d5589b4-0028-4529-91d8-5ef4b4684476-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e92d4a69-pqvbh\" (UID: \"6d5589b4-0028-4529-91d8-5ef4b4684476\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e92d4a69-pqvbh"
Oct 30 19:06:21.664099 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:21.664012    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d5589b4-0028-4529-91d8-5ef4b4684476-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e92d4a69-pqvbh\" (UID: \"6d5589b4-0028-4529-91d8-5ef4b4684476\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e92d4a69-pqvbh"
Oct 30 19:06:21.664169 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:21.664125    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4qmx\" (UniqueName: \"kubernetes.io/projected/6d5589b4-0028-4529-91d8-5ef4b4684476-kube-api-access-n4qmx\") pod \"maas-api-revoke-sub-e92d4a69-pqvbh\" (UID: \"6d5589b4-0028-4529-91d8-5ef4b4684476\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e92d4a69-pqvbh"
Oct 30 19:06:21.664630 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:21.664606    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d5589b4-0028-4529-91d8-5ef4b4684476-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e92d4a69-pqvbh\" (UID: \"6d5589b4-0028-4529-91d8-5ef4b4684476\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e92d4a69-pqvbh"
Oct 30 19:06:21.666743 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:21.666672    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d5589b4-0028-4529-91d8-5ef4b4684476-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e92d4a69-pqvbh\" (UID: \"6d5589b4-0028-4529-91d8-5ef4b4684476\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e92d4a69-pqvbh"
Oct 30 19:06:21.674986 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:21.674956    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4qmx\" (UniqueName: \"kubernetes.io/projected/6d5589b4-0028-4529-91d8-5ef4b4684476-kube-api-access-n4qmx\") pod \"maas-api-revoke-sub-e92d4a69-pqvbh\" (UID: \"6d5589b4-0028-4529-91d8-5ef4b4684476\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e92d4a69-pqvbh"
Oct 30 19:06:21.798474 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:21.798445    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e92d4a69-pqvbh"
Oct 30 19:06:21.828403 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:21.828355    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50605e61-5dwzf" event={"ID":"e87cddb0-df80-4259-a332-3f3ea69f431a","Type":"ContainerStarted","Data":"eb69d14fa43751530945221110977ca6b7b0cea5f5c1c068a8c5ba72e09bdd78"}
Oct 30 19:06:22.093828 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:06:22.093795    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d5589b4_0028_4529_91d8_5ef4b4684476.slice/crio-890f1d81829d374beedeb5c022547330ccbf4c621a2ecaf12385c23810e6b7a9 WatchSource:0}: Error finding container 890f1d81829d374beedeb5c022547330ccbf4c621a2ecaf12385c23810e6b7a9: Status 404 returned error can't find the container with id 890f1d81829d374beedeb5c022547330ccbf4c621a2ecaf12385c23810e6b7a9
Oct 30 19:06:22.094935 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:22.094908    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e92d4a69-pqvbh"]
Oct 30 19:06:22.122509 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:22.122475    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw"
Oct 30 19:06:22.122912 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:22.122887    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw"
Oct 30 19:06:22.123729 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:22.123713    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw"
Oct 30 19:06:22.912253 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:22.912217    4525 generic.go:356] "Generic (PLEG): container finished" podID="e87cddb0-df80-4259-a332-3f3ea69f431a" containerID="97ac4b5b7c2cfaeb460e82e1b6b0d8d1f984219b2a61082166c7e661b3b665fb" exitCode=0
Oct 30 19:06:22.912465 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:22.912306    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50605e61-5dwzf" event={"ID":"e87cddb0-df80-4259-a332-3f3ea69f431a","Type":"ContainerDied","Data":"97ac4b5b7c2cfaeb460e82e1b6b0d8d1f984219b2a61082166c7e661b3b665fb"}
Oct 30 19:06:22.915757 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:22.915729    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e92d4a69-pqvbh" event={"ID":"6d5589b4-0028-4529-91d8-5ef4b4684476","Type":"ContainerStarted","Data":"e65e6c6222d7bcc31fa1cea35167935ec59fbe28951d1bb9303a3ccd2a4660db"}
Oct 30 19:06:22.915942 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:22.915929    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e92d4a69-pqvbh" event={"ID":"6d5589b4-0028-4529-91d8-5ef4b4684476","Type":"ContainerStarted","Data":"890f1d81829d374beedeb5c022547330ccbf4c621a2ecaf12385c23810e6b7a9"}
Oct 30 19:06:22.953349 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:22.953297    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e92d4a69-pqvbh" podStartSLOduration=1.953282958 podStartE2EDuration="1.953282958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:06:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:06:22.9482909 +0000 UTC m=+1805.573960865" watchObservedRunningTime="2026-10-30 19:06:22.953282958 +0000 UTC m=+1805.578952922"
Oct 30 19:06:23.036737 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:23.036704    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdc7e4a4-cnzvg"]
Oct 30 19:06:23.041341 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:23.041312    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdc7e4a4-cnzvg"]
Oct 30 19:06:23.561766 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:23.561650    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac9d22c9-c98b-4857-bd09-6c55f8d79667" path="/var/lib/kubelet/pods/ac9d22c9-c98b-4857-bd09-6c55f8d79667/volumes"
Oct 30 19:06:23.928134 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:23.928098    4525 generic.go:356] "Generic (PLEG): container finished" podID="6d5589b4-0028-4529-91d8-5ef4b4684476" containerID="e65e6c6222d7bcc31fa1cea35167935ec59fbe28951d1bb9303a3ccd2a4660db" exitCode=0
Oct 30 19:06:23.928318 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:23.928178    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e92d4a69-pqvbh" event={"ID":"6d5589b4-0028-4529-91d8-5ef4b4684476","Type":"ContainerDied","Data":"e65e6c6222d7bcc31fa1cea35167935ec59fbe28951d1bb9303a3ccd2a4660db"}
Oct 30 19:06:24.092811 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:24.092780    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50605e61-5dwzf"
Oct 30 19:06:24.245995 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:24.245909    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e87cddb0-df80-4259-a332-3f3ea69f431a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e87cddb0-df80-4259-a332-3f3ea69f431a-maas-api-openshift-ca\") pod \"e87cddb0-df80-4259-a332-3f3ea69f431a\" (UID: \"e87cddb0-df80-4259-a332-3f3ea69f431a\") "
Oct 30 19:06:24.245995 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:24.245972    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e87cddb0-df80-4259-a332-3f3ea69f431a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e87cddb0-df80-4259-a332-3f3ea69f431a-maas-api-xks-ca\") pod \"e87cddb0-df80-4259-a332-3f3ea69f431a\" (UID: \"e87cddb0-df80-4259-a332-3f3ea69f431a\") "
Oct 30 19:06:24.246231 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:24.246116    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e87cddb0-df80-4259-a332-3f3ea69f431a-kube-api-access-8vkml\" (UniqueName: \"kubernetes.io/projected/e87cddb0-df80-4259-a332-3f3ea69f431a-kube-api-access-8vkml\") pod \"e87cddb0-df80-4259-a332-3f3ea69f431a\" (UID: \"e87cddb0-df80-4259-a332-3f3ea69f431a\") "
Oct 30 19:06:24.246363 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:24.246336    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e87cddb0-df80-4259-a332-3f3ea69f431a-maas-api-openshift-ca" pod "e87cddb0-df80-4259-a332-3f3ea69f431a" (UID: "e87cddb0-df80-4259-a332-3f3ea69f431a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:06:24.246868 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:24.246764    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e87cddb0-df80-4259-a332-3f3ea69f431a-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:24.248306 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:24.248282    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e87cddb0-df80-4259-a332-3f3ea69f431a-maas-api-xks-ca" pod "e87cddb0-df80-4259-a332-3f3ea69f431a" (UID: "e87cddb0-df80-4259-a332-3f3ea69f431a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:06:24.248400 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:24.248335    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e87cddb0-df80-4259-a332-3f3ea69f431a-kube-api-access-8vkml" pod "e87cddb0-df80-4259-a332-3f3ea69f431a" (UID: "e87cddb0-df80-4259-a332-3f3ea69f431a"). InnerVolumeSpecName "kube-api-access-8vkml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:06:24.348025 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:24.347989    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vkml\" (UniqueName: \"kubernetes.io/projected/e87cddb0-df80-4259-a332-3f3ea69f431a-kube-api-access-8vkml\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:24.348025 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:24.348019    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e87cddb0-df80-4259-a332-3f3ea69f431a-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:24.939779 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:24.939742    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50605e61-5dwzf"
Oct 30 19:06:24.940225 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:24.939779    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50605e61-5dwzf" event={"ID":"e87cddb0-df80-4259-a332-3f3ea69f431a","Type":"ContainerDied","Data":"eb69d14fa43751530945221110977ca6b7b0cea5f5c1c068a8c5ba72e09bdd78"}
Oct 30 19:06:24.940225 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:24.939835    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb69d14fa43751530945221110977ca6b7b0cea5f5c1c068a8c5ba72e09bdd78"
Oct 30 19:06:25.095927 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:25.095897    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e92d4a69-pqvbh"
Oct 30 19:06:25.276767 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:25.266321    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d5589b4-0028-4529-91d8-5ef4b4684476-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d5589b4-0028-4529-91d8-5ef4b4684476-maas-api-xks-ca\") pod \"6d5589b4-0028-4529-91d8-5ef4b4684476\" (UID: \"6d5589b4-0028-4529-91d8-5ef4b4684476\") "
Oct 30 19:06:25.276767 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:25.268990    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d5589b4-0028-4529-91d8-5ef4b4684476-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d5589b4-0028-4529-91d8-5ef4b4684476-maas-api-openshift-ca\") pod \"6d5589b4-0028-4529-91d8-5ef4b4684476\" (UID: \"6d5589b4-0028-4529-91d8-5ef4b4684476\") "
Oct 30 19:06:25.276767 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:25.270200    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d5589b4-0028-4529-91d8-5ef4b4684476-maas-api-openshift-ca" pod "6d5589b4-0028-4529-91d8-5ef4b4684476" (UID: "6d5589b4-0028-4529-91d8-5ef4b4684476"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:06:25.276767 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:25.270259    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d5589b4-0028-4529-91d8-5ef4b4684476-kube-api-access-n4qmx\" (UniqueName: \"kubernetes.io/projected/6d5589b4-0028-4529-91d8-5ef4b4684476-kube-api-access-n4qmx\") pod \"6d5589b4-0028-4529-91d8-5ef4b4684476\" (UID: \"6d5589b4-0028-4529-91d8-5ef4b4684476\") "
Oct 30 19:06:25.276767 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:25.272070    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d5589b4-0028-4529-91d8-5ef4b4684476-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:25.276767 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:25.272729    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d5589b4-0028-4529-91d8-5ef4b4684476-maas-api-xks-ca" pod "6d5589b4-0028-4529-91d8-5ef4b4684476" (UID: "6d5589b4-0028-4529-91d8-5ef4b4684476"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:06:25.276767 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:25.276423    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d5589b4-0028-4529-91d8-5ef4b4684476-kube-api-access-n4qmx" pod "6d5589b4-0028-4529-91d8-5ef4b4684476" (UID: "6d5589b4-0028-4529-91d8-5ef4b4684476"). InnerVolumeSpecName "kube-api-access-n4qmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:06:25.373367 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:25.373337    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4qmx\" (UniqueName: \"kubernetes.io/projected/6d5589b4-0028-4529-91d8-5ef4b4684476-kube-api-access-n4qmx\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:25.373367 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:25.373364    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d5589b4-0028-4529-91d8-5ef4b4684476-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:25.955559 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:25.955528    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e92d4a69-pqvbh"
Oct 30 19:06:25.955559 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:25.955556    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e92d4a69-pqvbh" event={"ID":"6d5589b4-0028-4529-91d8-5ef4b4684476","Type":"ContainerDied","Data":"890f1d81829d374beedeb5c022547330ccbf4c621a2ecaf12385c23810e6b7a9"}
Oct 30 19:06:25.956059 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:25.955608    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="890f1d81829d374beedeb5c022547330ccbf4c621a2ecaf12385c23810e6b7a9"
Oct 30 19:06:27.032704 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:27.032647    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b701f1a6-xglvp"]
Oct 30 19:06:27.037142 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:27.037119    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b701f1a6-xglvp"]
Oct 30 19:06:27.547550 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:27.547518    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa9ee8b1-cd03-459f-9ba7-e86b8dd0c5dd" path="/var/lib/kubelet/pods/aa9ee8b1-cd03-459f-9ba7-e86b8dd0c5dd/volumes"
Oct 30 19:06:30.806455 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:30.806422    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-ca2cd5-openshift-default-59664bf6df-xqn5s"]
Oct 30 19:06:30.807433 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:30.807404    4525 cpu_manager.go:414] "containerMap: removing container" podUID="6d5589b4-0028-4529-91d8-5ef4b4684476" containerName="revoke-keys"
Oct 30 19:06:30.807433 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:30.807432    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d5589b4-0028-4529-91d8-5ef4b4684476" containerName="revoke-keys"
Oct 30 19:06:30.807647 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:30.807452    4525 cpu_manager.go:414] "containerMap: removing container" podUID="e87cddb0-df80-4259-a332-3f3ea69f431a" containerName="revoke-keys"
Oct 30 19:06:30.807647 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:30.807460    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e87cddb0-df80-4259-a332-3f3ea69f431a" containerName="revoke-keys"
Oct 30 19:06:30.807910 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:30.807889    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d5589b4-0028-4529-91d8-5ef4b4684476" containerName="revoke-keys"
Oct 30 19:06:30.808008 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:30.807916    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="e87cddb0-df80-4259-a332-3f3ea69f431a" containerName="revoke-keys"
Oct 30 19:06:30.812549 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:30.812528    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ca2cd5-openshift-default-59664bf6df-xqn5s"
Oct 30 19:06:30.818901 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:30.818878    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-ca2cd5-openshift-default-dockercfg-2c8gl\""
Oct 30 19:06:30.827051 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:30.827012    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-ca2cd5-openshift-default-59664bf6df-xqn5s"]
Oct 30 19:06:30.881398 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:30.881344    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84c44eb7-6483-4f83-ab59-447f04ba5d1d-istio-podinfo\") pod \"e2e-extmod-a-ca2cd5-openshift-default-59664bf6df-xqn5s\" (UID: \"84c44eb7-6483-4f83-ab59-447f04ba5d1d\") " pod="openshift-ingress/e2e-extmod-a-ca2cd5-openshift-default-59664bf6df-xqn5s"
Oct 30 19:06:30.881605 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:30.881536    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84c44eb7-6483-4f83-ab59-447f04ba5d1d-istiod-ca-cert\") pod \"e2e-extmod-a-ca2cd5-openshift-default-59664bf6df-xqn5s\" (UID: \"84c44eb7-6483-4f83-ab59-447f04ba5d1d\") " pod="openshift-ingress/e2e-extmod-a-ca2cd5-openshift-default-59664bf6df-xqn5s"
Oct 30 19:06:30.881605 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:30.881587    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52499\" (UniqueName: \"kubernetes.io/projected/84c44eb7-6483-4f83-ab59-447f04ba5d1d-kube-api-access-52499\") pod \"e2e-extmod-a-ca2cd5-openshift-default-59664bf6df-xqn5s\" (UID: \"84c44eb7-6483-4f83-ab59-447f04ba5d1d\") " pod="openshift-ingress/e2e-extmod-a-ca2cd5-openshift-default-59664bf6df-xqn5s"
Oct 30 19:06:30.881748 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:30.881624    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84c44eb7-6483-4f83-ab59-447f04ba5d1d-istio-token\") pod \"e2e-extmod-a-ca2cd5-openshift-default-59664bf6df-xqn5s\" (UID: \"84c44eb7-6483-4f83-ab59-447f04ba5d1d\") " pod="openshift-ingress/e2e-extmod-a-ca2cd5-openshift-default-59664bf6df-xqn5s"
Oct 30 19:06:30.881964 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:30.881942    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84c44eb7-6483-4f83-ab59-447f04ba5d1d-workload-certs\") pod \"e2e-extmod-a-ca2cd5-openshift-default-59664bf6df-xqn5s\" (UID: \"84c44eb7-6483-4f83-ab59-447f04ba5d1d\") " pod="openshift-ingress/e2e-extmod-a-ca2cd5-openshift-default-59664bf6df-xqn5s"
Oct 30 19:06:30.911868 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:30.882042    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84c44eb7-6483-4f83-ab59-447f04ba5d1d-credential-socket\") pod \"e2e-extmod-a-ca2cd5-openshift-default-59664bf6df-xqn5s\" (UID: \"84c44eb7-6483-4f83-ab59-447f04ba5d1d\") " pod="openshift-ingress/e2e-extmod-a-ca2cd5-openshift-default-59664bf6df-xqn5s"
Oct 30 19:06:30.911868 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:30.882178    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84c44eb7-6483-4f83-ab59-447f04ba5d1d-istio-data\") pod \"e2e-extmod-a-ca2cd5-openshift-default-59664bf6df-xqn5s\" (UID: \"84c44eb7-6483-4f83-ab59-447f04ba5d1d\") " pod="openshift-ingress/e2e-extmod-a-ca2cd5-openshift-default-59664bf6df-xqn5s"
Oct 30 19:06:30.911868 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:30.882210    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84c44eb7-6483-4f83-ab59-447f04ba5d1d-workload-socket\") pod \"e2e-extmod-a-ca2cd5-openshift-default-59664bf6df-xqn5s\" (UID: \"84c44eb7-6483-4f83-ab59-447f04ba5d1d\") " pod="openshift-ingress/e2e-extmod-a-ca2cd5-openshift-default-59664bf6df-xqn5s"
Oct 30 19:06:30.911868 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:30.882235    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84c44eb7-6483-4f83-ab59-447f04ba5d1d-istio-envoy\") pod \"e2e-extmod-a-ca2cd5-openshift-default-59664bf6df-xqn5s\" (UID: \"84c44eb7-6483-4f83-ab59-447f04ba5d1d\") " pod="openshift-ingress/e2e-extmod-a-ca2cd5-openshift-default-59664bf6df-xqn5s"
Oct 30 19:06:31.037703 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:31.018120    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84c44eb7-6483-4f83-ab59-447f04ba5d1d-istio-podinfo\") pod \"e2e-extmod-a-ca2cd5-openshift-default-59664bf6df-xqn5s\" (UID: \"84c44eb7-6483-4f83-ab59-447f04ba5d1d\") " pod="openshift-ingress/e2e-extmod-a-ca2cd5-openshift-default-59664bf6df-xqn5s"
Oct 30 19:06:31.037703 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:31.018272    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84c44eb7-6483-4f83-ab59-447f04ba5d1d-istiod-ca-cert\") pod \"e2e-extmod-a-ca2cd5-openshift-default-59664bf6df-xqn5s\" (UID: \"84c44eb7-6483-4f83-ab59-447f04ba5d1d\") " pod="openshift-ingress/e2e-extmod-a-ca2cd5-openshift-default-59664bf6df-xqn5s"
Oct 30 19:06:31.037703 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:31.018341    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52499\" (UniqueName: \"kubernetes.io/projected/84c44eb7-6483-4f83-ab59-447f04ba5d1d-kube-api-access-52499\") pod \"e2e-extmod-a-ca2cd5-openshift-default-59664bf6df-xqn5s\" (UID: \"84c44eb7-6483-4f83-ab59-447f04ba5d1d\") " pod="openshift-ingress/e2e-extmod-a-ca2cd5-openshift-default-59664bf6df-xqn5s"
Oct 30 19:06:31.037703 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:31.018382    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84c44eb7-6483-4f83-ab59-447f04ba5d1d-istio-token\") pod \"e2e-extmod-a-ca2cd5-openshift-default-59664bf6df-xqn5s\" (UID: \"84c44eb7-6483-4f83-ab59-447f04ba5d1d\") " pod="openshift-ingress/e2e-extmod-a-ca2cd5-openshift-default-59664bf6df-xqn5s"
Oct 30 19:06:31.037703 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:31.018542    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84c44eb7-6483-4f83-ab59-447f04ba5d1d-workload-certs\") pod \"e2e-extmod-a-ca2cd5-openshift-default-59664bf6df-xqn5s\" (UID: \"84c44eb7-6483-4f83-ab59-447f04ba5d1d\") " pod="openshift-ingress/e2e-extmod-a-ca2cd5-openshift-default-59664bf6df-xqn5s"
Oct 30 19:06:31.037703 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:31.018605    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84c44eb7-6483-4f83-ab59-447f04ba5d1d-credential-socket\") pod \"e2e-extmod-a-ca2cd5-openshift-default-59664bf6df-xqn5s\" (UID: \"84c44eb7-6483-4f83-ab59-447f04ba5d1d\") " pod="openshift-ingress/e2e-extmod-a-ca2cd5-openshift-default-59664bf6df-xqn5s"
Oct 30 19:06:31.037703 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:31.037397    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84c44eb7-6483-4f83-ab59-447f04ba5d1d-workload-certs\") pod \"e2e-extmod-a-ca2cd5-openshift-default-59664bf6df-xqn5s\" (UID: \"84c44eb7-6483-4f83-ab59-447f04ba5d1d\") " pod="openshift-ingress/e2e-extmod-a-ca2cd5-openshift-default-59664bf6df-xqn5s"
Oct 30 19:06:31.046705 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:31.038938    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84c44eb7-6483-4f83-ab59-447f04ba5d1d-istio-data\") pod \"e2e-extmod-a-ca2cd5-openshift-default-59664bf6df-xqn5s\" (UID: \"84c44eb7-6483-4f83-ab59-447f04ba5d1d\") " pod="openshift-ingress/e2e-extmod-a-ca2cd5-openshift-default-59664bf6df-xqn5s"
Oct 30 19:06:31.046705 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:31.039334    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84c44eb7-6483-4f83-ab59-447f04ba5d1d-credential-socket\") pod \"e2e-extmod-a-ca2cd5-openshift-default-59664bf6df-xqn5s\" (UID: \"84c44eb7-6483-4f83-ab59-447f04ba5d1d\") " pod="openshift-ingress/e2e-extmod-a-ca2cd5-openshift-default-59664bf6df-xqn5s"
Oct 30 19:06:31.046705 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:31.039467    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84c44eb7-6483-4f83-ab59-447f04ba5d1d-istiod-ca-cert\") pod \"e2e-extmod-a-ca2cd5-openshift-default-59664bf6df-xqn5s\" (UID: \"84c44eb7-6483-4f83-ab59-447f04ba5d1d\") " pod="openshift-ingress/e2e-extmod-a-ca2cd5-openshift-default-59664bf6df-xqn5s"
Oct 30 19:06:31.046705 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:31.038631    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84c44eb7-6483-4f83-ab59-447f04ba5d1d-istio-data\") pod \"e2e-extmod-a-ca2cd5-openshift-default-59664bf6df-xqn5s\" (UID: \"84c44eb7-6483-4f83-ab59-447f04ba5d1d\") " pod="openshift-ingress/e2e-extmod-a-ca2cd5-openshift-default-59664bf6df-xqn5s"
Oct 30 19:06:31.046705 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:31.039695    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84c44eb7-6483-4f83-ab59-447f04ba5d1d-workload-socket\") pod \"e2e-extmod-a-ca2cd5-openshift-default-59664bf6df-xqn5s\" (UID: \"84c44eb7-6483-4f83-ab59-447f04ba5d1d\") " pod="openshift-ingress/e2e-extmod-a-ca2cd5-openshift-default-59664bf6df-xqn5s"
Oct 30 19:06:31.046705 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:31.039738    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84c44eb7-6483-4f83-ab59-447f04ba5d1d-istio-envoy\") pod \"e2e-extmod-a-ca2cd5-openshift-default-59664bf6df-xqn5s\" (UID: \"84c44eb7-6483-4f83-ab59-447f04ba5d1d\") " pod="openshift-ingress/e2e-extmod-a-ca2cd5-openshift-default-59664bf6df-xqn5s"
Oct 30 19:06:31.046705 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:31.040034    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84c44eb7-6483-4f83-ab59-447f04ba5d1d-workload-socket\") pod \"e2e-extmod-a-ca2cd5-openshift-default-59664bf6df-xqn5s\" (UID: \"84c44eb7-6483-4f83-ab59-447f04ba5d1d\") " pod="openshift-ingress/e2e-extmod-a-ca2cd5-openshift-default-59664bf6df-xqn5s"
Oct 30 19:06:31.047281 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:31.047029    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84c44eb7-6483-4f83-ab59-447f04ba5d1d-istio-podinfo\") pod \"e2e-extmod-a-ca2cd5-openshift-default-59664bf6df-xqn5s\" (UID: \"84c44eb7-6483-4f83-ab59-447f04ba5d1d\") " pod="openshift-ingress/e2e-extmod-a-ca2cd5-openshift-default-59664bf6df-xqn5s"
Oct 30 19:06:31.055715 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:31.049934    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52499\" (UniqueName: \"kubernetes.io/projected/84c44eb7-6483-4f83-ab59-447f04ba5d1d-kube-api-access-52499\") pod \"e2e-extmod-a-ca2cd5-openshift-default-59664bf6df-xqn5s\" (UID: \"84c44eb7-6483-4f83-ab59-447f04ba5d1d\") " pod="openshift-ingress/e2e-extmod-a-ca2cd5-openshift-default-59664bf6df-xqn5s"
Oct 30 19:06:31.055715 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:31.050931    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84c44eb7-6483-4f83-ab59-447f04ba5d1d-istio-envoy\") pod \"e2e-extmod-a-ca2cd5-openshift-default-59664bf6df-xqn5s\" (UID: \"84c44eb7-6483-4f83-ab59-447f04ba5d1d\") " pod="openshift-ingress/e2e-extmod-a-ca2cd5-openshift-default-59664bf6df-xqn5s"
Oct 30 19:06:31.055715 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:31.055058    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84c44eb7-6483-4f83-ab59-447f04ba5d1d-istio-token\") pod \"e2e-extmod-a-ca2cd5-openshift-default-59664bf6df-xqn5s\" (UID: \"84c44eb7-6483-4f83-ab59-447f04ba5d1d\") " pod="openshift-ingress/e2e-extmod-a-ca2cd5-openshift-default-59664bf6df-xqn5s"
Oct 30 19:06:31.138705 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:31.137951    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ca2cd5-openshift-default-59664bf6df-xqn5s"
Oct 30 19:06:31.326202 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:31.326163    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aff7c585-wtlks"]
Oct 30 19:06:31.344493 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:31.343348    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aff7c585-wtlks"
Oct 30 19:06:31.347209 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:31.347182    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aff7c585-wtlks"]
Oct 30 19:06:31.348260 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:31.347704    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:06:31.348260 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:31.348204    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94hcn\""
Oct 30 19:06:31.451504 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:31.451423    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzmb2\" (UniqueName: \"kubernetes.io/projected/30f43a69-f88c-42fe-a3bc-2d89f22e590c-kube-api-access-zzmb2\") pod \"maas-api-revoke-sub-aff7c585-wtlks\" (UID: \"30f43a69-f88c-42fe-a3bc-2d89f22e590c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aff7c585-wtlks"
Oct 30 19:06:31.451676 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:31.451591    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30f43a69-f88c-42fe-a3bc-2d89f22e590c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aff7c585-wtlks\" (UID: \"30f43a69-f88c-42fe-a3bc-2d89f22e590c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aff7c585-wtlks"
Oct 30 19:06:31.451676 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:31.451659    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30f43a69-f88c-42fe-a3bc-2d89f22e590c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aff7c585-wtlks\" (UID: \"30f43a69-f88c-42fe-a3bc-2d89f22e590c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aff7c585-wtlks"
Oct 30 19:06:31.524972 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:06:31.524943    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84c44eb7_6483_4f83_ab59_447f04ba5d1d.slice/crio-802ece6cc8f0d1270248adfcfb0cf0a553ce181a1f79354b873f579a5960ead0 WatchSource:0}: Error finding container 802ece6cc8f0d1270248adfcfb0cf0a553ce181a1f79354b873f579a5960ead0: Status 404 returned error can't find the container with id 802ece6cc8f0d1270248adfcfb0cf0a553ce181a1f79354b873f579a5960ead0
Oct 30 19:06:31.525960 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:31.525933    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-ca2cd5-openshift-default-59664bf6df-xqn5s"]
Oct 30 19:06:31.527854 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:31.527821    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:06:31.527929 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:31.527909    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:06:31.527967 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:31.527949    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:06:31.554258 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:31.554230    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30f43a69-f88c-42fe-a3bc-2d89f22e590c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aff7c585-wtlks\" (UID: \"30f43a69-f88c-42fe-a3bc-2d89f22e590c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aff7c585-wtlks"
Oct 30 19:06:31.554369 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:31.554328    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30f43a69-f88c-42fe-a3bc-2d89f22e590c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aff7c585-wtlks\" (UID: \"30f43a69-f88c-42fe-a3bc-2d89f22e590c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aff7c585-wtlks"
Oct 30 19:06:31.554601 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:31.554582    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzmb2\" (UniqueName: \"kubernetes.io/projected/30f43a69-f88c-42fe-a3bc-2d89f22e590c-kube-api-access-zzmb2\") pod \"maas-api-revoke-sub-aff7c585-wtlks\" (UID: \"30f43a69-f88c-42fe-a3bc-2d89f22e590c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aff7c585-wtlks"
Oct 30 19:06:31.554938 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:31.554906    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30f43a69-f88c-42fe-a3bc-2d89f22e590c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aff7c585-wtlks\" (UID: \"30f43a69-f88c-42fe-a3bc-2d89f22e590c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aff7c585-wtlks"
Oct 30 19:06:31.557329 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:31.557311    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30f43a69-f88c-42fe-a3bc-2d89f22e590c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aff7c585-wtlks\" (UID: \"30f43a69-f88c-42fe-a3bc-2d89f22e590c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aff7c585-wtlks"
Oct 30 19:06:31.562613 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:31.562594    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzmb2\" (UniqueName: \"kubernetes.io/projected/30f43a69-f88c-42fe-a3bc-2d89f22e590c-kube-api-access-zzmb2\") pod \"maas-api-revoke-sub-aff7c585-wtlks\" (UID: \"30f43a69-f88c-42fe-a3bc-2d89f22e590c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aff7c585-wtlks"
Oct 30 19:06:31.667708 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:31.666983    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aff7c585-wtlks"
Oct 30 19:06:31.770619 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:31.770587    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3f6d48f-vqc5d"]
Oct 30 19:06:31.786246 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:31.786217    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3f6d48f-vqc5d"
Oct 30 19:06:31.787144 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:31.787120    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3f6d48f-vqc5d"]
Oct 30 19:06:31.957043 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:06:31.957008    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30f43a69_f88c_42fe_a3bc_2d89f22e590c.slice/crio-9c0e33e37e8235643962862e762e006a50617679e7e5a6e2dc89398b0442358d WatchSource:0}: Error finding container 9c0e33e37e8235643962862e762e006a50617679e7e5a6e2dc89398b0442358d: Status 404 returned error can't find the container with id 9c0e33e37e8235643962862e762e006a50617679e7e5a6e2dc89398b0442358d
Oct 30 19:06:31.957649 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:31.957624    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aff7c585-wtlks"]
Oct 30 19:06:31.976310 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:31.976286    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc3415ec-21bd-40f3-baec-cb80ebe2468f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3f6d48f-vqc5d\" (UID: \"fc3415ec-21bd-40f3-baec-cb80ebe2468f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3f6d48f-vqc5d"
Oct 30 19:06:31.976509 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:31.976485    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc3415ec-21bd-40f3-baec-cb80ebe2468f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3f6d48f-vqc5d\" (UID: \"fc3415ec-21bd-40f3-baec-cb80ebe2468f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3f6d48f-vqc5d"
Oct 30 19:06:31.976605 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:31.976588    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5whk9\" (UniqueName: \"kubernetes.io/projected/fc3415ec-21bd-40f3-baec-cb80ebe2468f-kube-api-access-5whk9\") pod \"maas-api-revoke-sub-a3f6d48f-vqc5d\" (UID: \"fc3415ec-21bd-40f3-baec-cb80ebe2468f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3f6d48f-vqc5d"
Oct 30 19:06:32.077656 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:32.077618    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc3415ec-21bd-40f3-baec-cb80ebe2468f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3f6d48f-vqc5d\" (UID: \"fc3415ec-21bd-40f3-baec-cb80ebe2468f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3f6d48f-vqc5d"
Oct 30 19:06:32.077841 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:32.077718    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5whk9\" (UniqueName: \"kubernetes.io/projected/fc3415ec-21bd-40f3-baec-cb80ebe2468f-kube-api-access-5whk9\") pod \"maas-api-revoke-sub-a3f6d48f-vqc5d\" (UID: \"fc3415ec-21bd-40f3-baec-cb80ebe2468f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3f6d48f-vqc5d"
Oct 30 19:06:32.077911 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:32.077882    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc3415ec-21bd-40f3-baec-cb80ebe2468f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3f6d48f-vqc5d\" (UID: \"fc3415ec-21bd-40f3-baec-cb80ebe2468f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3f6d48f-vqc5d"
Oct 30 19:06:32.078417 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:32.078394    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc3415ec-21bd-40f3-baec-cb80ebe2468f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3f6d48f-vqc5d\" (UID: \"fc3415ec-21bd-40f3-baec-cb80ebe2468f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3f6d48f-vqc5d"
Oct 30 19:06:32.080413 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:32.080395    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc3415ec-21bd-40f3-baec-cb80ebe2468f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3f6d48f-vqc5d\" (UID: \"fc3415ec-21bd-40f3-baec-cb80ebe2468f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3f6d48f-vqc5d"
Oct 30 19:06:32.092673 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:32.092642    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aff7c585-wtlks" event={"ID":"30f43a69-f88c-42fe-a3bc-2d89f22e590c","Type":"ContainerStarted","Data":"9c0e33e37e8235643962862e762e006a50617679e7e5a6e2dc89398b0442358d"}
Oct 30 19:06:32.094554 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:32.094527    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ca2cd5-openshift-default-59664bf6df-xqn5s" event={"ID":"84c44eb7-6483-4f83-ab59-447f04ba5d1d","Type":"ContainerStarted","Data":"aa921ad64f5ffd5ff960ec19542da03046337d91f1e9db386923174b3632c874"}
Oct 30 19:06:32.094650 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:32.094578    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ca2cd5-openshift-default-59664bf6df-xqn5s" event={"ID":"84c44eb7-6483-4f83-ab59-447f04ba5d1d","Type":"ContainerStarted","Data":"802ece6cc8f0d1270248adfcfb0cf0a553ce181a1f79354b873f579a5960ead0"}
Oct 30 19:06:32.122645 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:32.122594    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-ca2cd5-openshift-default-59664bf6df-xqn5s" podStartSLOduration=2.122578661 podStartE2EDuration="2.122578661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:06:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:06:32.117291123 +0000 UTC m=+1814.742961087" watchObservedRunningTime="2026-10-30 19:06:32.122578661 +0000 UTC m=+1814.748248625"
Oct 30 19:06:32.138668 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:32.138633    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-ca2cd5-openshift-default-59664bf6df-xqn5s"
Oct 30 19:06:32.144773 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:32.144748    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-ca2cd5-openshift-default-59664bf6df-xqn5s"
Oct 30 19:06:32.177116 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:32.177088    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5whk9\" (UniqueName: \"kubernetes.io/projected/fc3415ec-21bd-40f3-baec-cb80ebe2468f-kube-api-access-5whk9\") pod \"maas-api-revoke-sub-a3f6d48f-vqc5d\" (UID: \"fc3415ec-21bd-40f3-baec-cb80ebe2468f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3f6d48f-vqc5d"
Oct 30 19:06:32.417853 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:32.417815    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3f6d48f-vqc5d"
Oct 30 19:06:32.701655 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:06:32.701624    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc3415ec_21bd_40f3_baec_cb80ebe2468f.slice/crio-ee9962461a7f597b629c4d8e122ea22988a0ac4e7c4faed5cea9e1729c1f3a04 WatchSource:0}: Error finding container ee9962461a7f597b629c4d8e122ea22988a0ac4e7c4faed5cea9e1729c1f3a04: Status 404 returned error can't find the container with id ee9962461a7f597b629c4d8e122ea22988a0ac4e7c4faed5cea9e1729c1f3a04
Oct 30 19:06:32.702889 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:32.702865    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3f6d48f-vqc5d"]
Oct 30 19:06:33.106323 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:33.106289    4525 generic.go:356] "Generic (PLEG): container finished" podID="30f43a69-f88c-42fe-a3bc-2d89f22e590c" containerID="312e333d6833330acb4fcde0951ab2063af3f0ba5ba832bcc8921676f139bfea" exitCode=0
Oct 30 19:06:33.106763 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:33.106375    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aff7c585-wtlks" event={"ID":"30f43a69-f88c-42fe-a3bc-2d89f22e590c","Type":"ContainerDied","Data":"312e333d6833330acb4fcde0951ab2063af3f0ba5ba832bcc8921676f139bfea"}
Oct 30 19:06:33.107784 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:33.107757    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3f6d48f-vqc5d" event={"ID":"fc3415ec-21bd-40f3-baec-cb80ebe2468f","Type":"ContainerStarted","Data":"ee9962461a7f597b629c4d8e122ea22988a0ac4e7c4faed5cea9e1729c1f3a04"}
Oct 30 19:06:33.108310 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:33.108295    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-ca2cd5-openshift-default-59664bf6df-xqn5s"
Oct 30 19:06:33.110068 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:33.110050    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-ca2cd5-openshift-default-59664bf6df-xqn5s"
Oct 30 19:06:34.119401 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:34.119365    4525 generic.go:356] "Generic (PLEG): container finished" podID="fc3415ec-21bd-40f3-baec-cb80ebe2468f" containerID="88a248e06f67fae48e93b755d8f14656928e2f2db358bc2cecf2bd459687de84" exitCode=0
Oct 30 19:06:34.119871 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:34.119446    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3f6d48f-vqc5d" event={"ID":"fc3415ec-21bd-40f3-baec-cb80ebe2468f","Type":"ContainerDied","Data":"88a248e06f67fae48e93b755d8f14656928e2f2db358bc2cecf2bd459687de84"}
Oct 30 19:06:34.380124 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:34.380054    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aff7c585-wtlks"
Oct 30 19:06:34.459110 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:34.459071    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30f43a69-f88c-42fe-a3bc-2d89f22e590c-kube-api-access-zzmb2\" (UniqueName: \"kubernetes.io/projected/30f43a69-f88c-42fe-a3bc-2d89f22e590c-kube-api-access-zzmb2\") pod \"30f43a69-f88c-42fe-a3bc-2d89f22e590c\" (UID: \"30f43a69-f88c-42fe-a3bc-2d89f22e590c\") "
Oct 30 19:06:34.459309 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:34.459131    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30f43a69-f88c-42fe-a3bc-2d89f22e590c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30f43a69-f88c-42fe-a3bc-2d89f22e590c-maas-api-xks-ca\") pod \"30f43a69-f88c-42fe-a3bc-2d89f22e590c\" (UID: \"30f43a69-f88c-42fe-a3bc-2d89f22e590c\") "
Oct 30 19:06:34.459428 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:34.459413    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30f43a69-f88c-42fe-a3bc-2d89f22e590c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30f43a69-f88c-42fe-a3bc-2d89f22e590c-maas-api-openshift-ca\") pod \"30f43a69-f88c-42fe-a3bc-2d89f22e590c\" (UID: \"30f43a69-f88c-42fe-a3bc-2d89f22e590c\") "
Oct 30 19:06:34.459921 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:34.459896    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30f43a69-f88c-42fe-a3bc-2d89f22e590c-maas-api-openshift-ca" pod "30f43a69-f88c-42fe-a3bc-2d89f22e590c" (UID: "30f43a69-f88c-42fe-a3bc-2d89f22e590c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:06:34.460207 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:34.460184    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30f43a69-f88c-42fe-a3bc-2d89f22e590c-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:34.461306 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:34.461285    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30f43a69-f88c-42fe-a3bc-2d89f22e590c-maas-api-xks-ca" pod "30f43a69-f88c-42fe-a3bc-2d89f22e590c" (UID: "30f43a69-f88c-42fe-a3bc-2d89f22e590c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:06:34.461366 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:34.461288    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30f43a69-f88c-42fe-a3bc-2d89f22e590c-kube-api-access-zzmb2" pod "30f43a69-f88c-42fe-a3bc-2d89f22e590c" (UID: "30f43a69-f88c-42fe-a3bc-2d89f22e590c"). InnerVolumeSpecName "kube-api-access-zzmb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:06:34.562092 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:34.562050    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzmb2\" (UniqueName: \"kubernetes.io/projected/30f43a69-f88c-42fe-a3bc-2d89f22e590c-kube-api-access-zzmb2\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:34.562092 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:34.562083    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30f43a69-f88c-42fe-a3bc-2d89f22e590c-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:35.129615 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:35.129585    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aff7c585-wtlks"
Oct 30 19:06:35.130116 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:35.129629    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aff7c585-wtlks" event={"ID":"30f43a69-f88c-42fe-a3bc-2d89f22e590c","Type":"ContainerDied","Data":"9c0e33e37e8235643962862e762e006a50617679e7e5a6e2dc89398b0442358d"}
Oct 30 19:06:35.130116 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:35.129694    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c0e33e37e8235643962862e762e006a50617679e7e5a6e2dc89398b0442358d"
Oct 30 19:06:35.377438 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:35.377408    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3f6d48f-vqc5d"
Oct 30 19:06:35.497856 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:35.497717    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc3415ec-21bd-40f3-baec-cb80ebe2468f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc3415ec-21bd-40f3-baec-cb80ebe2468f-maas-api-xks-ca\") pod \"fc3415ec-21bd-40f3-baec-cb80ebe2468f\" (UID: \"fc3415ec-21bd-40f3-baec-cb80ebe2468f\") "
Oct 30 19:06:35.498137 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:35.497895    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc3415ec-21bd-40f3-baec-cb80ebe2468f-kube-api-access-5whk9\" (UniqueName: \"kubernetes.io/projected/fc3415ec-21bd-40f3-baec-cb80ebe2468f-kube-api-access-5whk9\") pod \"fc3415ec-21bd-40f3-baec-cb80ebe2468f\" (UID: \"fc3415ec-21bd-40f3-baec-cb80ebe2468f\") "
Oct 30 19:06:35.498137 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:35.497968    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc3415ec-21bd-40f3-baec-cb80ebe2468f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc3415ec-21bd-40f3-baec-cb80ebe2468f-maas-api-openshift-ca\") pod \"fc3415ec-21bd-40f3-baec-cb80ebe2468f\" (UID: \"fc3415ec-21bd-40f3-baec-cb80ebe2468f\") "
Oct 30 19:06:35.498654 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:35.498632    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc3415ec-21bd-40f3-baec-cb80ebe2468f-maas-api-openshift-ca" pod "fc3415ec-21bd-40f3-baec-cb80ebe2468f" (UID: "fc3415ec-21bd-40f3-baec-cb80ebe2468f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:06:35.500642 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:35.500612    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc3415ec-21bd-40f3-baec-cb80ebe2468f-maas-api-xks-ca" pod "fc3415ec-21bd-40f3-baec-cb80ebe2468f" (UID: "fc3415ec-21bd-40f3-baec-cb80ebe2468f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:06:35.500880 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:35.500858    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc3415ec-21bd-40f3-baec-cb80ebe2468f-kube-api-access-5whk9" pod "fc3415ec-21bd-40f3-baec-cb80ebe2468f" (UID: "fc3415ec-21bd-40f3-baec-cb80ebe2468f"). InnerVolumeSpecName "kube-api-access-5whk9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:06:35.600561 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:35.600523    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc3415ec-21bd-40f3-baec-cb80ebe2468f-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:35.600561 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:35.600564    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5whk9\" (UniqueName: \"kubernetes.io/projected/fc3415ec-21bd-40f3-baec-cb80ebe2468f-kube-api-access-5whk9\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:35.600848 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:35.600583    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc3415ec-21bd-40f3-baec-cb80ebe2468f-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:36.140173 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:36.140139    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3f6d48f-vqc5d"
Oct 30 19:06:36.140173 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:36.140159    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3f6d48f-vqc5d" event={"ID":"fc3415ec-21bd-40f3-baec-cb80ebe2468f","Type":"ContainerDied","Data":"ee9962461a7f597b629c4d8e122ea22988a0ac4e7c4faed5cea9e1729c1f3a04"}
Oct 30 19:06:36.140628 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:36.140224    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee9962461a7f597b629c4d8e122ea22988a0ac4e7c4faed5cea9e1729c1f3a04"
Oct 30 19:06:37.031444 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:37.031397    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44a8360d-nl2qd"]
Oct 30 19:06:37.037917 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:37.037883    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44a8360d-nl2qd"]
Oct 30 19:06:37.547460 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:37.547428    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea492c81-b0a8-4973-9e81-0bfb8d8c1c10" path="/var/lib/kubelet/pods/ea492c81-b0a8-4973-9e81-0bfb8d8c1c10/volumes"
Oct 30 19:06:38.025263 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:38.025226    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf304416-z29q5"]
Oct 30 19:06:38.029673 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:38.029649    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf304416-z29q5"]
Oct 30 19:06:39.028206 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:39.028166    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6555186c-9phwp"]
Oct 30 19:06:39.033094 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:39.033063    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6555186c-9phwp"]
Oct 30 19:06:39.545790 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:39.545760    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ff9d08b-2785-438e-a818-d217f24f247c" path="/var/lib/kubelet/pods/5ff9d08b-2785-438e-a818-d217f24f247c/volumes"
Oct 30 19:06:39.546290 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:39.546275    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7cda5cb-59d6-4ca3-b308-401c7c3de9af" path="/var/lib/kubelet/pods/d7cda5cb-59d6-4ca3-b308-401c7c3de9af/volumes"
Oct 30 19:06:43.031470 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:43.031432    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55276386-slqmv"]
Oct 30 19:06:43.035739 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:43.035704    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55276386-slqmv"]
Oct 30 19:06:43.206132 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:43.206099    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-a1d55d/auth-test-model-a1d55d-kserve-7c4f64fd9-njmlg"]
Oct 30 19:06:43.546085 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:43.546053    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1c2894f-597a-4d4a-a247-863056689d5c" path="/var/lib/kubelet/pods/c1c2894f-597a-4d4a-a247-863056689d5c/volumes"
Oct 30 19:06:45.777638 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:45.777608    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45"]
Oct 30 19:06:45.778576 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:45.778549    4525 cpu_manager.go:414] "containerMap: removing container" podUID="30f43a69-f88c-42fe-a3bc-2d89f22e590c" containerName="revoke-keys"
Oct 30 19:06:45.778576 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:45.778578    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30f43a69-f88c-42fe-a3bc-2d89f22e590c" containerName="revoke-keys"
Oct 30 19:06:45.779500 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:45.778614    4525 cpu_manager.go:414] "containerMap: removing container" podUID="fc3415ec-21bd-40f3-baec-cb80ebe2468f" containerName="revoke-keys"
Oct 30 19:06:45.779500 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:45.778621    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc3415ec-21bd-40f3-baec-cb80ebe2468f" containerName="revoke-keys"
Oct 30 19:06:45.779500 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:45.778921    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="30f43a69-f88c-42fe-a3bc-2d89f22e590c" containerName="revoke-keys"
Oct 30 19:06:45.779500 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:45.778947    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc3415ec-21bd-40f3-baec-cb80ebe2468f" containerName="revoke-keys"
Oct 30 19:06:45.784370 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:45.784345    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45"
Oct 30 19:06:45.797417 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:45.797021    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-d6f956-openshift-default-dockercfg-jmqmf\""
Oct 30 19:06:45.807368 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:45.807341    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45"]
Oct 30 19:06:45.974735 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:45.968245    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2f9a313-f069-4e48-8cdc-f75c6a285a7c-istio-podinfo\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45\" (UID: \"d2f9a313-f069-4e48-8cdc-f75c6a285a7c\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45"
Oct 30 19:06:45.974735 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:45.968330    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2f9a313-f069-4e48-8cdc-f75c6a285a7c-workload-certs\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45\" (UID: \"d2f9a313-f069-4e48-8cdc-f75c6a285a7c\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45"
Oct 30 19:06:45.974735 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:45.968365    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2f9a313-f069-4e48-8cdc-f75c6a285a7c-workload-socket\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45\" (UID: \"d2f9a313-f069-4e48-8cdc-f75c6a285a7c\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45"
Oct 30 19:06:45.974735 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:45.968425    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2f9a313-f069-4e48-8cdc-f75c6a285a7c-istio-token\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45\" (UID: \"d2f9a313-f069-4e48-8cdc-f75c6a285a7c\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45"
Oct 30 19:06:45.974735 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:45.968474    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2f9a313-f069-4e48-8cdc-f75c6a285a7c-istio-data\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45\" (UID: \"d2f9a313-f069-4e48-8cdc-f75c6a285a7c\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45"
Oct 30 19:06:45.974735 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:45.968553    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2f9a313-f069-4e48-8cdc-f75c6a285a7c-istiod-ca-cert\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45\" (UID: \"d2f9a313-f069-4e48-8cdc-f75c6a285a7c\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45"
Oct 30 19:06:45.974735 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:45.968591    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8mst\" (UniqueName: \"kubernetes.io/projected/d2f9a313-f069-4e48-8cdc-f75c6a285a7c-kube-api-access-m8mst\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45\" (UID: \"d2f9a313-f069-4e48-8cdc-f75c6a285a7c\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45"
Oct 30 19:06:45.974735 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:45.968661    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2f9a313-f069-4e48-8cdc-f75c6a285a7c-credential-socket\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45\" (UID: \"d2f9a313-f069-4e48-8cdc-f75c6a285a7c\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45"
Oct 30 19:06:45.974735 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:45.968749    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2f9a313-f069-4e48-8cdc-f75c6a285a7c-istio-envoy\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45\" (UID: \"d2f9a313-f069-4e48-8cdc-f75c6a285a7c\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45"
Oct 30 19:06:46.074763 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:46.074313    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2f9a313-f069-4e48-8cdc-f75c6a285a7c-istio-podinfo\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45\" (UID: \"d2f9a313-f069-4e48-8cdc-f75c6a285a7c\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45"
Oct 30 19:06:46.080704 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:46.075245    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2f9a313-f069-4e48-8cdc-f75c6a285a7c-workload-certs\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45\" (UID: \"d2f9a313-f069-4e48-8cdc-f75c6a285a7c\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45"
Oct 30 19:06:46.080704 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:46.075308    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2f9a313-f069-4e48-8cdc-f75c6a285a7c-workload-certs\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45\" (UID: \"d2f9a313-f069-4e48-8cdc-f75c6a285a7c\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45"
Oct 30 19:06:46.080704 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:46.075375    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2f9a313-f069-4e48-8cdc-f75c6a285a7c-workload-socket\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45\" (UID: \"d2f9a313-f069-4e48-8cdc-f75c6a285a7c\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45"
Oct 30 19:06:46.080704 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:46.075702    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2f9a313-f069-4e48-8cdc-f75c6a285a7c-workload-socket\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45\" (UID: \"d2f9a313-f069-4e48-8cdc-f75c6a285a7c\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45"
Oct 30 19:06:46.080704 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:46.075791    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2f9a313-f069-4e48-8cdc-f75c6a285a7c-istio-token\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45\" (UID: \"d2f9a313-f069-4e48-8cdc-f75c6a285a7c\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45"
Oct 30 19:06:46.080704 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:46.076865    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2f9a313-f069-4e48-8cdc-f75c6a285a7c-istio-data\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45\" (UID: \"d2f9a313-f069-4e48-8cdc-f75c6a285a7c\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45"
Oct 30 19:06:46.080704 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:46.076947    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2f9a313-f069-4e48-8cdc-f75c6a285a7c-istiod-ca-cert\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45\" (UID: \"d2f9a313-f069-4e48-8cdc-f75c6a285a7c\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45"
Oct 30 19:06:46.080704 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:46.076982    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8mst\" (UniqueName: \"kubernetes.io/projected/d2f9a313-f069-4e48-8cdc-f75c6a285a7c-kube-api-access-m8mst\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45\" (UID: \"d2f9a313-f069-4e48-8cdc-f75c6a285a7c\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45"
Oct 30 19:06:46.080704 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:46.077047    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2f9a313-f069-4e48-8cdc-f75c6a285a7c-credential-socket\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45\" (UID: \"d2f9a313-f069-4e48-8cdc-f75c6a285a7c\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45"
Oct 30 19:06:46.080704 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:46.077111    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2f9a313-f069-4e48-8cdc-f75c6a285a7c-istio-envoy\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45\" (UID: \"d2f9a313-f069-4e48-8cdc-f75c6a285a7c\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45"
Oct 30 19:06:46.080704 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:46.078181    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2f9a313-f069-4e48-8cdc-f75c6a285a7c-istiod-ca-cert\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45\" (UID: \"d2f9a313-f069-4e48-8cdc-f75c6a285a7c\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45"
Oct 30 19:06:46.080704 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:46.078411    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2f9a313-f069-4e48-8cdc-f75c6a285a7c-istio-data\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45\" (UID: \"d2f9a313-f069-4e48-8cdc-f75c6a285a7c\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45"
Oct 30 19:06:46.094703 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:46.091028    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2f9a313-f069-4e48-8cdc-f75c6a285a7c-credential-socket\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45\" (UID: \"d2f9a313-f069-4e48-8cdc-f75c6a285a7c\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45"
Oct 30 19:06:46.109707 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:46.107798    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2f9a313-f069-4e48-8cdc-f75c6a285a7c-istio-podinfo\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45\" (UID: \"d2f9a313-f069-4e48-8cdc-f75c6a285a7c\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45"
Oct 30 19:06:46.109707 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:46.108090    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2f9a313-f069-4e48-8cdc-f75c6a285a7c-istio-envoy\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45\" (UID: \"d2f9a313-f069-4e48-8cdc-f75c6a285a7c\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45"
Oct 30 19:06:46.119716 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:46.118229    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2f9a313-f069-4e48-8cdc-f75c6a285a7c-istio-token\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45\" (UID: \"d2f9a313-f069-4e48-8cdc-f75c6a285a7c\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45"
Oct 30 19:06:46.131760 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:46.131425    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8mst\" (UniqueName: \"kubernetes.io/projected/d2f9a313-f069-4e48-8cdc-f75c6a285a7c-kube-api-access-m8mst\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45\" (UID: \"d2f9a313-f069-4e48-8cdc-f75c6a285a7c\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45"
Oct 30 19:06:46.407606 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:46.407569    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45"
Oct 30 19:06:46.775139 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:46.774720    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45"]
Oct 30 19:06:46.777601 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:06:46.776660    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2f9a313_f069_4e48_8cdc_f75c6a285a7c.slice/crio-2bd83764225aa9567618eb597150f429110d4ce37258e87d8c1b0fde256819e7 WatchSource:0}: Error finding container 2bd83764225aa9567618eb597150f429110d4ce37258e87d8c1b0fde256819e7: Status 404 returned error can't find the container with id 2bd83764225aa9567618eb597150f429110d4ce37258e87d8c1b0fde256819e7
Oct 30 19:06:46.780383 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:46.780343    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:06:46.780768 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:46.780430    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:06:46.780768 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:46.780470    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:06:47.396906 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:47.396862    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45" event={"ID":"d2f9a313-f069-4e48-8cdc-f75c6a285a7c","Type":"ContainerStarted","Data":"4e0c7213e1372f284371fd7df3e88f0b367fecefa141b4f62b24c7e75b267acf"}
Oct 30 19:06:47.397127 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:47.396941    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45" event={"ID":"d2f9a313-f069-4e48-8cdc-f75c6a285a7c","Type":"ContainerStarted","Data":"2bd83764225aa9567618eb597150f429110d4ce37258e87d8c1b0fde256819e7"}
Oct 30 19:06:47.407835 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:47.407812    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45"
Oct 30 19:06:47.422748 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:47.420548    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45" podStartSLOduration=2.420529377 podStartE2EDuration="2.420529377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:06:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:06:47.418559928 +0000 UTC m=+1830.044229893" watchObservedRunningTime="2026-10-30 19:06:47.420529377 +0000 UTC m=+1830.046199342"
Oct 30 19:06:47.423258 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:06:47.423213    4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d49l" containerName="extractor"
Oct 30 19:06:47.424301 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:47.424279    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45"
Oct 30 19:06:48.405414 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:48.405376    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45"
Oct 30 19:06:48.406297 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:48.406272    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45"
Oct 30 19:06:51.069936 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:51.069904    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-ef6b56/auth-test-model-ef6b56-kserve-66c7fd94b9-qgqj4"]
Oct 30 19:06:54.040673 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:54.040618    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06319aae-h4mgm"]
Oct 30 19:06:54.045850 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:54.045821    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06319aae-h4mgm"]
Oct 30 19:06:54.909538 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:54.909484    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e2a74ee-8rn8t"]
Oct 30 19:06:54.915498 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:54.915470    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e2a74ee-8rn8t"
Oct 30 19:06:54.917731 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:54.917705    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e2a74ee-8rn8t"]
Oct 30 19:06:54.921938 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:54.921911    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:06:54.922375 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:54.922352    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94hcn\""
Oct 30 19:06:55.109672 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:55.109634    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7feb7ae1-63f7-499a-ab49-cc561b2a2ca5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e2a74ee-8rn8t\" (UID: \"7feb7ae1-63f7-499a-ab49-cc561b2a2ca5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e2a74ee-8rn8t"
Oct 30 19:06:55.110137 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:55.109886    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7feb7ae1-63f7-499a-ab49-cc561b2a2ca5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e2a74ee-8rn8t\" (UID: \"7feb7ae1-63f7-499a-ab49-cc561b2a2ca5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e2a74ee-8rn8t"
Oct 30 19:06:55.110137 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:55.110045    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8p4f\" (UniqueName: \"kubernetes.io/projected/7feb7ae1-63f7-499a-ab49-cc561b2a2ca5-kube-api-access-l8p4f\") pod \"maas-api-revoke-sub-3e2a74ee-8rn8t\" (UID: \"7feb7ae1-63f7-499a-ab49-cc561b2a2ca5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e2a74ee-8rn8t"
Oct 30 19:06:55.211524 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:55.211432    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8p4f\" (UniqueName: \"kubernetes.io/projected/7feb7ae1-63f7-499a-ab49-cc561b2a2ca5-kube-api-access-l8p4f\") pod \"maas-api-revoke-sub-3e2a74ee-8rn8t\" (UID: \"7feb7ae1-63f7-499a-ab49-cc561b2a2ca5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e2a74ee-8rn8t"
Oct 30 19:06:55.211690 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:55.211565    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7feb7ae1-63f7-499a-ab49-cc561b2a2ca5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e2a74ee-8rn8t\" (UID: \"7feb7ae1-63f7-499a-ab49-cc561b2a2ca5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e2a74ee-8rn8t"
Oct 30 19:06:55.211772 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:55.211744    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7feb7ae1-63f7-499a-ab49-cc561b2a2ca5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e2a74ee-8rn8t\" (UID: \"7feb7ae1-63f7-499a-ab49-cc561b2a2ca5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e2a74ee-8rn8t"
Oct 30 19:06:55.212382 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:55.212362    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7feb7ae1-63f7-499a-ab49-cc561b2a2ca5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e2a74ee-8rn8t\" (UID: \"7feb7ae1-63f7-499a-ab49-cc561b2a2ca5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e2a74ee-8rn8t"
Oct 30 19:06:55.214227 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:55.214205    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7feb7ae1-63f7-499a-ab49-cc561b2a2ca5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e2a74ee-8rn8t\" (UID: \"7feb7ae1-63f7-499a-ab49-cc561b2a2ca5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e2a74ee-8rn8t"
Oct 30 19:06:55.221463 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:55.221442    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8p4f\" (UniqueName: \"kubernetes.io/projected/7feb7ae1-63f7-499a-ab49-cc561b2a2ca5-kube-api-access-l8p4f\") pod \"maas-api-revoke-sub-3e2a74ee-8rn8t\" (UID: \"7feb7ae1-63f7-499a-ab49-cc561b2a2ca5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e2a74ee-8rn8t"
Oct 30 19:06:55.244578 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:55.244555    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e2a74ee-8rn8t"
Oct 30 19:06:55.552322 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:55.552240    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="334444c1-d724-4938-b69e-331a3cacefd2" path="/var/lib/kubelet/pods/334444c1-d724-4938-b69e-331a3cacefd2/volumes"
Oct 30 19:06:55.563929 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:55.563788    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e2a74ee-8rn8t"]
Oct 30 19:06:55.564661 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:06:55.564630    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7feb7ae1_63f7_499a_ab49_cc561b2a2ca5.slice/crio-5ffabdf6825e507718d2fee7c03121dda23f28deceef73d23daf33f46cd1ea70 WatchSource:0}: Error finding container 5ffabdf6825e507718d2fee7c03121dda23f28deceef73d23daf33f46cd1ea70: Status 404 returned error can't find the container with id 5ffabdf6825e507718d2fee7c03121dda23f28deceef73d23daf33f46cd1ea70
Oct 30 19:06:56.546647 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:56.546550    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e2a74ee-8rn8t" event={"ID":"7feb7ae1-63f7-499a-ab49-cc561b2a2ca5","Type":"ContainerStarted","Data":"ca7a3ef6e3ed3e2f7e6aa4eb9b83abf888232928ac66586c7f4a951d28676c88"}
Oct 30 19:06:56.546647 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:56.546631    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e2a74ee-8rn8t" event={"ID":"7feb7ae1-63f7-499a-ab49-cc561b2a2ca5","Type":"ContainerStarted","Data":"5ffabdf6825e507718d2fee7c03121dda23f28deceef73d23daf33f46cd1ea70"}
Oct 30 19:06:56.568642 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:56.568597    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e2a74ee-8rn8t" podStartSLOduration=2.568583531 podStartE2EDuration="2.568583531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:06:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:06:56.562885495 +0000 UTC m=+1839.188555451" watchObservedRunningTime="2026-10-30 19:06:56.568583531 +0000 UTC m=+1839.194253495"
Oct 30 19:06:57.557835 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:57.557799    4525 generic.go:356] "Generic (PLEG): container finished" podID="7feb7ae1-63f7-499a-ab49-cc561b2a2ca5" containerID="ca7a3ef6e3ed3e2f7e6aa4eb9b83abf888232928ac66586c7f4a951d28676c88" exitCode=0
Oct 30 19:06:57.558332 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:57.557875    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e2a74ee-8rn8t" event={"ID":"7feb7ae1-63f7-499a-ab49-cc561b2a2ca5","Type":"ContainerDied","Data":"ca7a3ef6e3ed3e2f7e6aa4eb9b83abf888232928ac66586c7f4a951d28676c88"}
Oct 30 19:06:58.230037 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:58.229985    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-a1d55d/auth-test-model-a1d55d-kserve-7c4f64fd9-njmlg" podUID="72b01806-d7cc-4452-896b-32d64e92af97" containerName="main" containerID="cri-o://60fba0878041766f2b49adbac4b18a306253254ca7d10037190ae03cc4d1f0d3" gracePeriod=45
Oct 30 19:06:58.572084 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:58.571658    4525 generic.go:356] "Generic (PLEG): container finished" podID="72b01806-d7cc-4452-896b-32d64e92af97" containerID="60fba0878041766f2b49adbac4b18a306253254ca7d10037190ae03cc4d1f0d3" exitCode=0
Oct 30 19:06:58.572084 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:58.571865    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-a1d55d/auth-test-model-a1d55d-kserve-7c4f64fd9-njmlg" event={"ID":"72b01806-d7cc-4452-896b-32d64e92af97","Type":"ContainerDied","Data":"60fba0878041766f2b49adbac4b18a306253254ca7d10037190ae03cc4d1f0d3"}
Oct 30 19:06:58.635407 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:58.635375    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-a1d55d/auth-test-model-a1d55d-kserve-7c4f64fd9-njmlg"
Oct 30 19:06:58.707864 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:58.707826    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72b01806-d7cc-4452-896b-32d64e92af97-tls-certs\" (UniqueName: \"kubernetes.io/secret/72b01806-d7cc-4452-896b-32d64e92af97-tls-certs\") pod \"72b01806-d7cc-4452-896b-32d64e92af97\" (UID: \"72b01806-d7cc-4452-896b-32d64e92af97\") "
Oct 30 19:06:58.708056 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:58.707992    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72b01806-d7cc-4452-896b-32d64e92af97-kube-api-access-q2mz7\" (UniqueName: \"kubernetes.io/projected/72b01806-d7cc-4452-896b-32d64e92af97-kube-api-access-q2mz7\") pod \"72b01806-d7cc-4452-896b-32d64e92af97\" (UID: \"72b01806-d7cc-4452-896b-32d64e92af97\") "
Oct 30 19:06:58.708129 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:58.708093    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72b01806-d7cc-4452-896b-32d64e92af97-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72b01806-d7cc-4452-896b-32d64e92af97-model-cache\") pod \"72b01806-d7cc-4452-896b-32d64e92af97\" (UID: \"72b01806-d7cc-4452-896b-32d64e92af97\") "
Oct 30 19:06:58.708195 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:58.708149    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72b01806-d7cc-4452-896b-32d64e92af97-home\" (UniqueName: \"kubernetes.io/empty-dir/72b01806-d7cc-4452-896b-32d64e92af97-home\") pod \"72b01806-d7cc-4452-896b-32d64e92af97\" (UID: \"72b01806-d7cc-4452-896b-32d64e92af97\") "
Oct 30 19:06:58.708195 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:58.708183    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72b01806-d7cc-4452-896b-32d64e92af97-dshm\" (UniqueName: \"kubernetes.io/empty-dir/72b01806-d7cc-4452-896b-32d64e92af97-dshm\") pod \"72b01806-d7cc-4452-896b-32d64e92af97\" (UID: \"72b01806-d7cc-4452-896b-32d64e92af97\") "
Oct 30 19:06:58.708302 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:58.708272    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72b01806-d7cc-4452-896b-32d64e92af97-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72b01806-d7cc-4452-896b-32d64e92af97-tmp-dir\") pod \"72b01806-d7cc-4452-896b-32d64e92af97\" (UID: \"72b01806-d7cc-4452-896b-32d64e92af97\") "
Oct 30 19:06:58.708954 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:58.708918    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72b01806-d7cc-4452-896b-32d64e92af97-home" pod "72b01806-d7cc-4452-896b-32d64e92af97" (UID: "72b01806-d7cc-4452-896b-32d64e92af97"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:06:58.709111 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:58.708987    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72b01806-d7cc-4452-896b-32d64e92af97-model-cache" pod "72b01806-d7cc-4452-896b-32d64e92af97" (UID: "72b01806-d7cc-4452-896b-32d64e92af97"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:06:58.709185 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:58.709118    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72b01806-d7cc-4452-896b-32d64e92af97-tmp-dir" pod "72b01806-d7cc-4452-896b-32d64e92af97" (UID: "72b01806-d7cc-4452-896b-32d64e92af97"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:06:58.709342 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:58.709294    4525 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72b01806-d7cc-4452-896b-32d64e92af97-model-cache\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:58.709342 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:58.709326    4525 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72b01806-d7cc-4452-896b-32d64e92af97-home\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:58.709342 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:58.709350    4525 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72b01806-d7cc-4452-896b-32d64e92af97-tmp-dir\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:58.718076 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:58.718044    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72b01806-d7cc-4452-896b-32d64e92af97-kube-api-access-q2mz7" pod "72b01806-d7cc-4452-896b-32d64e92af97" (UID: "72b01806-d7cc-4452-896b-32d64e92af97"). InnerVolumeSpecName "kube-api-access-q2mz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:06:58.718959 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:58.718914    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72b01806-d7cc-4452-896b-32d64e92af97-tls-certs" pod "72b01806-d7cc-4452-896b-32d64e92af97" (UID: "72b01806-d7cc-4452-896b-32d64e92af97"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:06:58.719284 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:58.719246    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72b01806-d7cc-4452-896b-32d64e92af97-dshm" pod "72b01806-d7cc-4452-896b-32d64e92af97" (UID: "72b01806-d7cc-4452-896b-32d64e92af97"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:06:58.811342 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:58.810847    4525 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72b01806-d7cc-4452-896b-32d64e92af97-tls-certs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:58.811342 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:58.810873    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2mz7\" (UniqueName: \"kubernetes.io/projected/72b01806-d7cc-4452-896b-32d64e92af97-kube-api-access-q2mz7\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:58.811342 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:58.810883    4525 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72b01806-d7cc-4452-896b-32d64e92af97-dshm\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:58.818092 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:58.818060    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e2a74ee-8rn8t"
Oct 30 19:06:58.912027 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:58.911988    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7feb7ae1-63f7-499a-ab49-cc561b2a2ca5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7feb7ae1-63f7-499a-ab49-cc561b2a2ca5-maas-api-xks-ca\") pod \"7feb7ae1-63f7-499a-ab49-cc561b2a2ca5\" (UID: \"7feb7ae1-63f7-499a-ab49-cc561b2a2ca5\") "
Oct 30 19:06:58.912222 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:58.912092    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7feb7ae1-63f7-499a-ab49-cc561b2a2ca5-kube-api-access-l8p4f\" (UniqueName: \"kubernetes.io/projected/7feb7ae1-63f7-499a-ab49-cc561b2a2ca5-kube-api-access-l8p4f\") pod \"7feb7ae1-63f7-499a-ab49-cc561b2a2ca5\" (UID: \"7feb7ae1-63f7-499a-ab49-cc561b2a2ca5\") "
Oct 30 19:06:58.912489 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:58.912439    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7feb7ae1-63f7-499a-ab49-cc561b2a2ca5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7feb7ae1-63f7-499a-ab49-cc561b2a2ca5-maas-api-openshift-ca\") pod \"7feb7ae1-63f7-499a-ab49-cc561b2a2ca5\" (UID: \"7feb7ae1-63f7-499a-ab49-cc561b2a2ca5\") "
Oct 30 19:06:58.912923 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:58.912881    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7feb7ae1-63f7-499a-ab49-cc561b2a2ca5-maas-api-openshift-ca" pod "7feb7ae1-63f7-499a-ab49-cc561b2a2ca5" (UID: "7feb7ae1-63f7-499a-ab49-cc561b2a2ca5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:06:58.913382 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:58.913355    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7feb7ae1-63f7-499a-ab49-cc561b2a2ca5-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:58.914558 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:58.914529    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7feb7ae1-63f7-499a-ab49-cc561b2a2ca5-maas-api-xks-ca" pod "7feb7ae1-63f7-499a-ab49-cc561b2a2ca5" (UID: "7feb7ae1-63f7-499a-ab49-cc561b2a2ca5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:06:58.915251 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:58.915226    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7feb7ae1-63f7-499a-ab49-cc561b2a2ca5-kube-api-access-l8p4f" pod "7feb7ae1-63f7-499a-ab49-cc561b2a2ca5" (UID: "7feb7ae1-63f7-499a-ab49-cc561b2a2ca5"). InnerVolumeSpecName "kube-api-access-l8p4f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:06:59.015474 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:59.015433    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7feb7ae1-63f7-499a-ab49-cc561b2a2ca5-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:59.015474 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:59.015471    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8p4f\" (UniqueName: \"kubernetes.io/projected/7feb7ae1-63f7-499a-ab49-cc561b2a2ca5-kube-api-access-l8p4f\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:59.052382 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:59.052336    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-248f423d-bxtt8"]
Oct 30 19:06:59.059527 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:59.059489    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-248f423d-bxtt8"]
Oct 30 19:06:59.546994 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:59.546909    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1aa70b7e-acd9-48f1-bd02-0db2ea932d98" path="/var/lib/kubelet/pods/1aa70b7e-acd9-48f1-bd02-0db2ea932d98/volumes"
Oct 30 19:06:59.585349 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:59.585322    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-a1d55d/auth-test-model-a1d55d-kserve-7c4f64fd9-njmlg"
Oct 30 19:06:59.585886 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:59.585346    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-a1d55d/auth-test-model-a1d55d-kserve-7c4f64fd9-njmlg" event={"ID":"72b01806-d7cc-4452-896b-32d64e92af97","Type":"ContainerDied","Data":"ebcdc7ea7cb04d4540c7cef67aa23acb114b885f57081243c7230ecf485063da"}
Oct 30 19:06:59.585886 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:59.585402    4525 scope.go:122] "RemoveContainer" containerID="60fba0878041766f2b49adbac4b18a306253254ca7d10037190ae03cc4d1f0d3"
Oct 30 19:06:59.588559 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:59.588532    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e2a74ee-8rn8t"
Oct 30 19:06:59.588714 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:59.588623    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e2a74ee-8rn8t" event={"ID":"7feb7ae1-63f7-499a-ab49-cc561b2a2ca5","Type":"ContainerDied","Data":"5ffabdf6825e507718d2fee7c03121dda23f28deceef73d23daf33f46cd1ea70"}
Oct 30 19:06:59.588810 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:59.588792    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ffabdf6825e507718d2fee7c03121dda23f28deceef73d23daf33f46cd1ea70"
Oct 30 19:06:59.607322 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:59.607283    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-a1d55d/auth-test-model-a1d55d-kserve-7c4f64fd9-njmlg"]
Oct 30 19:06:59.618166 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:06:59.618130    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-a1d55d/auth-test-model-a1d55d-kserve-7c4f64fd9-njmlg"]
Oct 30 19:07:01.548086 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:01.548052    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72b01806-d7cc-4452-896b-32d64e92af97" path="/var/lib/kubelet/pods/72b01806-d7cc-4452-896b-32d64e92af97/volumes"
Oct 30 19:07:04.046130 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:04.046093    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx"]
Oct 30 19:07:04.047256 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:04.047219    4525 cpu_manager.go:414] "containerMap: removing container" podUID="72b01806-d7cc-4452-896b-32d64e92af97" containerName="main"
Oct 30 19:07:04.047256 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:04.047245    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72b01806-d7cc-4452-896b-32d64e92af97" containerName="main"
Oct 30 19:07:04.047470 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:04.047284    4525 cpu_manager.go:414] "containerMap: removing container" podUID="7feb7ae1-63f7-499a-ab49-cc561b2a2ca5" containerName="revoke-keys"
Oct 30 19:07:04.047470 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:04.047293    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7feb7ae1-63f7-499a-ab49-cc561b2a2ca5" containerName="revoke-keys"
Oct 30 19:07:04.047587 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:04.047479    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="72b01806-d7cc-4452-896b-32d64e92af97" containerName="main"
Oct 30 19:07:04.047587 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:04.047496    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="7feb7ae1-63f7-499a-ab49-cc561b2a2ca5" containerName="revoke-keys"
Oct 30 19:07:04.067326 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:04.066499    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx"]
Oct 30 19:07:04.067326 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:04.066622    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx"
Oct 30 19:07:04.076094 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:04.076057    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-1dd09d-openshift-default-dockercfg-cj522\""
Oct 30 19:07:04.160505 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:04.160472    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2099828-c41e-43c1-b174-63e8eb826c05-workload-certs\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx\" (UID: \"b2099828-c41e-43c1-b174-63e8eb826c05\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx"
Oct 30 19:07:04.160720 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:04.160545    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2099828-c41e-43c1-b174-63e8eb826c05-istiod-ca-cert\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx\" (UID: \"b2099828-c41e-43c1-b174-63e8eb826c05\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx"
Oct 30 19:07:04.160720 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:04.160636    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2099828-c41e-43c1-b174-63e8eb826c05-istio-token\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx\" (UID: \"b2099828-c41e-43c1-b174-63e8eb826c05\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx"
Oct 30 19:07:04.160854 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:04.160787    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2099828-c41e-43c1-b174-63e8eb826c05-istio-envoy\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx\" (UID: \"b2099828-c41e-43c1-b174-63e8eb826c05\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx"
Oct 30 19:07:04.160854 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:04.160841    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2099828-c41e-43c1-b174-63e8eb826c05-istio-podinfo\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx\" (UID: \"b2099828-c41e-43c1-b174-63e8eb826c05\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx"
Oct 30 19:07:04.160973 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:04.160924    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2099828-c41e-43c1-b174-63e8eb826c05-credential-socket\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx\" (UID: \"b2099828-c41e-43c1-b174-63e8eb826c05\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx"
Oct 30 19:07:04.160973 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:04.160952    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fr8k\" (UniqueName: \"kubernetes.io/projected/b2099828-c41e-43c1-b174-63e8eb826c05-kube-api-access-7fr8k\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx\" (UID: \"b2099828-c41e-43c1-b174-63e8eb826c05\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx"
Oct 30 19:07:04.161095 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:04.160975    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2099828-c41e-43c1-b174-63e8eb826c05-istio-data\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx\" (UID: \"b2099828-c41e-43c1-b174-63e8eb826c05\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx"
Oct 30 19:07:04.161158 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:04.161131    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2099828-c41e-43c1-b174-63e8eb826c05-workload-socket\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx\" (UID: \"b2099828-c41e-43c1-b174-63e8eb826c05\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx"
Oct 30 19:07:04.361758 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:04.266090    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2099828-c41e-43c1-b174-63e8eb826c05-workload-socket\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx\" (UID: \"b2099828-c41e-43c1-b174-63e8eb826c05\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx"
Oct 30 19:07:04.361758 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:04.266161    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2099828-c41e-43c1-b174-63e8eb826c05-workload-certs\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx\" (UID: \"b2099828-c41e-43c1-b174-63e8eb826c05\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx"
Oct 30 19:07:04.361758 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:04.266240    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2099828-c41e-43c1-b174-63e8eb826c05-istiod-ca-cert\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx\" (UID: \"b2099828-c41e-43c1-b174-63e8eb826c05\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx"
Oct 30 19:07:04.361758 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:04.266269    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2099828-c41e-43c1-b174-63e8eb826c05-istio-token\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx\" (UID: \"b2099828-c41e-43c1-b174-63e8eb826c05\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx"
Oct 30 19:07:04.361758 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:04.266409    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2099828-c41e-43c1-b174-63e8eb826c05-istio-envoy\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx\" (UID: \"b2099828-c41e-43c1-b174-63e8eb826c05\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx"
Oct 30 19:07:04.361758 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:04.266484    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2099828-c41e-43c1-b174-63e8eb826c05-istio-podinfo\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx\" (UID: \"b2099828-c41e-43c1-b174-63e8eb826c05\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx"
Oct 30 19:07:04.361758 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:04.266545    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2099828-c41e-43c1-b174-63e8eb826c05-credential-socket\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx\" (UID: \"b2099828-c41e-43c1-b174-63e8eb826c05\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx"
Oct 30 19:07:04.361758 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:04.266572    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fr8k\" (UniqueName: \"kubernetes.io/projected/b2099828-c41e-43c1-b174-63e8eb826c05-kube-api-access-7fr8k\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx\" (UID: \"b2099828-c41e-43c1-b174-63e8eb826c05\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx"
Oct 30 19:07:04.361758 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:04.266606    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2099828-c41e-43c1-b174-63e8eb826c05-istio-data\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx\" (UID: \"b2099828-c41e-43c1-b174-63e8eb826c05\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx"
Oct 30 19:07:04.361758 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:04.267182    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2099828-c41e-43c1-b174-63e8eb826c05-istio-data\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx\" (UID: \"b2099828-c41e-43c1-b174-63e8eb826c05\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx"
Oct 30 19:07:04.361758 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:04.267426    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2099828-c41e-43c1-b174-63e8eb826c05-workload-socket\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx\" (UID: \"b2099828-c41e-43c1-b174-63e8eb826c05\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx"
Oct 30 19:07:04.361758 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:04.267647    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2099828-c41e-43c1-b174-63e8eb826c05-workload-certs\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx\" (UID: \"b2099828-c41e-43c1-b174-63e8eb826c05\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx"
Oct 30 19:07:04.361758 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:04.268272    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2099828-c41e-43c1-b174-63e8eb826c05-istiod-ca-cert\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx\" (UID: \"b2099828-c41e-43c1-b174-63e8eb826c05\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx"
Oct 30 19:07:04.361758 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:04.350958    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2099828-c41e-43c1-b174-63e8eb826c05-istio-podinfo\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx\" (UID: \"b2099828-c41e-43c1-b174-63e8eb826c05\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx"
Oct 30 19:07:04.361758 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:04.352312    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fr8k\" (UniqueName: \"kubernetes.io/projected/b2099828-c41e-43c1-b174-63e8eb826c05-kube-api-access-7fr8k\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx\" (UID: \"b2099828-c41e-43c1-b174-63e8eb826c05\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx"
Oct 30 19:07:04.383706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:04.364351    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2099828-c41e-43c1-b174-63e8eb826c05-istio-token\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx\" (UID: \"b2099828-c41e-43c1-b174-63e8eb826c05\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx"
Oct 30 19:07:04.383706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:04.365904    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2099828-c41e-43c1-b174-63e8eb826c05-istio-envoy\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx\" (UID: \"b2099828-c41e-43c1-b174-63e8eb826c05\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx"
Oct 30 19:07:04.383706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:04.366190    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2099828-c41e-43c1-b174-63e8eb826c05-credential-socket\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx\" (UID: \"b2099828-c41e-43c1-b174-63e8eb826c05\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx"
Oct 30 19:07:04.417005 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:04.416970    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx"
Oct 30 19:07:04.851556 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:04.845705    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx"]
Oct 30 19:07:04.851556 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:04.850154    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:07:04.851556 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:04.850235    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:07:04.851556 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:04.850264    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:07:05.217636 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:05.217539    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1af7d73c-njhbh"]
Oct 30 19:07:05.232907 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:05.232879    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1af7d73c-njhbh"]
Oct 30 19:07:05.233063 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:05.233002    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1af7d73c-njhbh"
Oct 30 19:07:05.250666 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:05.238391    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94hcn\""
Oct 30 19:07:05.250666 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:05.238453    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:07:05.375121 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:05.375082    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8df5475-2c04-4c22-b53e-f01d9c4e6e32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1af7d73c-njhbh\" (UID: \"e8df5475-2c04-4c22-b53e-f01d9c4e6e32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1af7d73c-njhbh"
Oct 30 19:07:05.376038 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:05.375341    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8df5475-2c04-4c22-b53e-f01d9c4e6e32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1af7d73c-njhbh\" (UID: \"e8df5475-2c04-4c22-b53e-f01d9c4e6e32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1af7d73c-njhbh"
Oct 30 19:07:05.376038 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:05.375464    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcdwj\" (UniqueName: \"kubernetes.io/projected/e8df5475-2c04-4c22-b53e-f01d9c4e6e32-kube-api-access-tcdwj\") pod \"maas-api-revoke-sub-1af7d73c-njhbh\" (UID: \"e8df5475-2c04-4c22-b53e-f01d9c4e6e32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1af7d73c-njhbh"
Oct 30 19:07:05.476723 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:05.476589    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8df5475-2c04-4c22-b53e-f01d9c4e6e32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1af7d73c-njhbh\" (UID: \"e8df5475-2c04-4c22-b53e-f01d9c4e6e32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1af7d73c-njhbh"
Oct 30 19:07:05.476723 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:05.476701    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcdwj\" (UniqueName: \"kubernetes.io/projected/e8df5475-2c04-4c22-b53e-f01d9c4e6e32-kube-api-access-tcdwj\") pod \"maas-api-revoke-sub-1af7d73c-njhbh\" (UID: \"e8df5475-2c04-4c22-b53e-f01d9c4e6e32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1af7d73c-njhbh"
Oct 30 19:07:05.476970 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:05.476945    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8df5475-2c04-4c22-b53e-f01d9c4e6e32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1af7d73c-njhbh\" (UID: \"e8df5475-2c04-4c22-b53e-f01d9c4e6e32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1af7d73c-njhbh"
Oct 30 19:07:05.478023 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:05.477995    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8df5475-2c04-4c22-b53e-f01d9c4e6e32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1af7d73c-njhbh\" (UID: \"e8df5475-2c04-4c22-b53e-f01d9c4e6e32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1af7d73c-njhbh"
Oct 30 19:07:05.479606 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:05.479581    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8df5475-2c04-4c22-b53e-f01d9c4e6e32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1af7d73c-njhbh\" (UID: \"e8df5475-2c04-4c22-b53e-f01d9c4e6e32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1af7d73c-njhbh"
Oct 30 19:07:05.485213 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:05.485185    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcdwj\" (UniqueName: \"kubernetes.io/projected/e8df5475-2c04-4c22-b53e-f01d9c4e6e32-kube-api-access-tcdwj\") pod \"maas-api-revoke-sub-1af7d73c-njhbh\" (UID: \"e8df5475-2c04-4c22-b53e-f01d9c4e6e32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1af7d73c-njhbh"
Oct 30 19:07:05.601715 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:05.575625    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1af7d73c-njhbh"
Oct 30 19:07:05.773981 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:05.773771    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx" event={"ID":"b2099828-c41e-43c1-b174-63e8eb826c05","Type":"ContainerStarted","Data":"4e4ca10e76fb3b83340b3f05a3a5934faae758882759cd4acbf5258f4c00f1a7"}
Oct 30 19:07:05.776694 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:05.776650    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx" event={"ID":"b2099828-c41e-43c1-b174-63e8eb826c05","Type":"ContainerStarted","Data":"0e59cb69624ab07f933963b8efe2d091c529805cad6d5d671d9668648da390f4"}
Oct 30 19:07:05.811434 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:05.811370    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx" podStartSLOduration=2.811347594 podStartE2EDuration="2.811347594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:07:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:07:05.805096678 +0000 UTC m=+1848.430766643" watchObservedRunningTime="2026-10-30 19:07:05.811347594 +0000 UTC m=+1848.437017559"
Oct 30 19:07:06.083082 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:07:06.083048    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8df5475_2c04_4c22_b53e_f01d9c4e6e32.slice/crio-6be8cd4ca4b86b6142ff6707923b58a224cd748d0d85d5d05266d27ae727d855 WatchSource:0}: Error finding container 6be8cd4ca4b86b6142ff6707923b58a224cd748d0d85d5d05266d27ae727d855: Status 404 returned error can't find the container with id 6be8cd4ca4b86b6142ff6707923b58a224cd748d0d85d5d05266d27ae727d855
Oct 30 19:07:06.087026 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:06.087001    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1af7d73c-njhbh"]
Oct 30 19:07:06.092878 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:06.092813    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-ef6b56/auth-test-model-ef6b56-kserve-66c7fd94b9-qgqj4" podUID="ec0f645b-289a-4b05-a984-f0f66e896357" containerName="main" containerID="cri-o://a10bda61492f43d5ea9ee8f82b401bbf4fa5082b8e78a3ecbd39be5aea40410f" gracePeriod=45
Oct 30 19:07:06.467817 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:06.438594    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx"
Oct 30 19:07:06.467817 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:06.444926    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx"
Oct 30 19:07:06.483878 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:06.483849    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-ef6b56/auth-test-model-ef6b56-kserve-66c7fd94b9-qgqj4"
Oct 30 19:07:06.621856 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:06.621821    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-f5ct2"]
Oct 30 19:07:06.623171 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:06.623143    4525 cpu_manager.go:414] "containerMap: removing container" podUID="ec0f645b-289a-4b05-a984-f0f66e896357" containerName="main"
Oct 30 19:07:06.623171 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:06.623169    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec0f645b-289a-4b05-a984-f0f66e896357" containerName="main"
Oct 30 19:07:06.623412 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:06.623398    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec0f645b-289a-4b05-a984-f0f66e896357" containerName="main"
Oct 30 19:07:06.641808 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:06.637847    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-f5ct2"
Oct 30 19:07:06.662897 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:06.650540    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec0f645b-289a-4b05-a984-f0f66e896357-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec0f645b-289a-4b05-a984-f0f66e896357-model-cache\") pod \"ec0f645b-289a-4b05-a984-f0f66e896357\" (UID: \"ec0f645b-289a-4b05-a984-f0f66e896357\") "
Oct 30 19:07:06.662897 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:06.650642    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec0f645b-289a-4b05-a984-f0f66e896357-home\" (UniqueName: \"kubernetes.io/empty-dir/ec0f645b-289a-4b05-a984-f0f66e896357-home\") pod \"ec0f645b-289a-4b05-a984-f0f66e896357\" (UID: \"ec0f645b-289a-4b05-a984-f0f66e896357\") "
Oct 30 19:07:06.662897 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:06.650751    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec0f645b-289a-4b05-a984-f0f66e896357-tls-certs\" (UniqueName: \"kubernetes.io/secret/ec0f645b-289a-4b05-a984-f0f66e896357-tls-certs\") pod \"ec0f645b-289a-4b05-a984-f0f66e896357\" (UID: \"ec0f645b-289a-4b05-a984-f0f66e896357\") "
Oct 30 19:07:06.662897 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:06.650834    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec0f645b-289a-4b05-a984-f0f66e896357-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec0f645b-289a-4b05-a984-f0f66e896357-tmp-dir\") pod \"ec0f645b-289a-4b05-a984-f0f66e896357\" (UID: \"ec0f645b-289a-4b05-a984-f0f66e896357\") "
Oct 30 19:07:06.662897 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:06.650902    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec0f645b-289a-4b05-a984-f0f66e896357-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec0f645b-289a-4b05-a984-f0f66e896357-dshm\") pod \"ec0f645b-289a-4b05-a984-f0f66e896357\" (UID: \"ec0f645b-289a-4b05-a984-f0f66e896357\") "
Oct 30 19:07:06.662897 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:06.650967    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec0f645b-289a-4b05-a984-f0f66e896357-kube-api-access-mn25j\" (UniqueName: \"kubernetes.io/projected/ec0f645b-289a-4b05-a984-f0f66e896357-kube-api-access-mn25j\") pod \"ec0f645b-289a-4b05-a984-f0f66e896357\" (UID: \"ec0f645b-289a-4b05-a984-f0f66e896357\") "
Oct 30 19:07:06.662897 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:06.652697    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-f3d6a5-openshift-default-dockercfg-8dmt5\""
Oct 30 19:07:06.662897 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:06.656532    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec0f645b-289a-4b05-a984-f0f66e896357-model-cache" pod "ec0f645b-289a-4b05-a984-f0f66e896357" (UID: "ec0f645b-289a-4b05-a984-f0f66e896357"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:07:06.662897 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:06.656900    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec0f645b-289a-4b05-a984-f0f66e896357-home" pod "ec0f645b-289a-4b05-a984-f0f66e896357" (UID: "ec0f645b-289a-4b05-a984-f0f66e896357"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:07:06.662897 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:06.657611    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec0f645b-289a-4b05-a984-f0f66e896357-tmp-dir" pod "ec0f645b-289a-4b05-a984-f0f66e896357" (UID: "ec0f645b-289a-4b05-a984-f0f66e896357"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:07:06.662897 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:06.658284    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec0f645b-289a-4b05-a984-f0f66e896357-tls-certs" pod "ec0f645b-289a-4b05-a984-f0f66e896357" (UID: "ec0f645b-289a-4b05-a984-f0f66e896357"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:07:06.673572 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:06.673524    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec0f645b-289a-4b05-a984-f0f66e896357-kube-api-access-mn25j" pod "ec0f645b-289a-4b05-a984-f0f66e896357" (UID: "ec0f645b-289a-4b05-a984-f0f66e896357"). InnerVolumeSpecName "kube-api-access-mn25j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:07:06.681520 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:06.681486    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-f5ct2"]
Oct 30 19:07:06.753597 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:06.753564    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9094695-97bc-4924-9526-45b9e8892662-istio-token\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-f5ct2\" (UID: \"d9094695-97bc-4924-9526-45b9e8892662\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-f5ct2"
Oct 30 19:07:06.753597 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:06.753601    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9094695-97bc-4924-9526-45b9e8892662-workload-certs\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-f5ct2\" (UID: \"d9094695-97bc-4924-9526-45b9e8892662\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-f5ct2"
Oct 30 19:07:06.753843 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:06.753742    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9094695-97bc-4924-9526-45b9e8892662-istio-envoy\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-f5ct2\" (UID: \"d9094695-97bc-4924-9526-45b9e8892662\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-f5ct2"
Oct 30 19:07:06.753883 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:06.753854    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9094695-97bc-4924-9526-45b9e8892662-istio-data\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-f5ct2\" (UID: \"d9094695-97bc-4924-9526-45b9e8892662\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-f5ct2"
Oct 30 19:07:06.753933 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:06.753920    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9094695-97bc-4924-9526-45b9e8892662-istiod-ca-cert\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-f5ct2\" (UID: \"d9094695-97bc-4924-9526-45b9e8892662\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-f5ct2"
Oct 30 19:07:06.753974 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:06.753956    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9094695-97bc-4924-9526-45b9e8892662-workload-socket\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-f5ct2\" (UID: \"d9094695-97bc-4924-9526-45b9e8892662\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-f5ct2"
Oct 30 19:07:06.754064 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:06.754047    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjxkq\" (UniqueName: \"kubernetes.io/projected/d9094695-97bc-4924-9526-45b9e8892662-kube-api-access-gjxkq\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-f5ct2\" (UID: \"d9094695-97bc-4924-9526-45b9e8892662\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-f5ct2"
Oct 30 19:07:06.754102 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:06.754094    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9094695-97bc-4924-9526-45b9e8892662-istio-podinfo\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-f5ct2\" (UID: \"d9094695-97bc-4924-9526-45b9e8892662\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-f5ct2"
Oct 30 19:07:06.754243 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:06.754226    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9094695-97bc-4924-9526-45b9e8892662-credential-socket\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-f5ct2\" (UID: \"d9094695-97bc-4924-9526-45b9e8892662\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-f5ct2"
Oct 30 19:07:06.754361 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:06.754349    4525 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec0f645b-289a-4b05-a984-f0f66e896357-model-cache\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:06.754413 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:06.754364    4525 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec0f645b-289a-4b05-a984-f0f66e896357-home\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:06.754413 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:06.754374    4525 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec0f645b-289a-4b05-a984-f0f66e896357-tls-certs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:06.754413 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:06.754382    4525 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec0f645b-289a-4b05-a984-f0f66e896357-tmp-dir\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:06.754413 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:06.754392    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mn25j\" (UniqueName: \"kubernetes.io/projected/ec0f645b-289a-4b05-a984-f0f66e896357-kube-api-access-mn25j\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:06.776129 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:06.776098    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec0f645b-289a-4b05-a984-f0f66e896357-dshm" pod "ec0f645b-289a-4b05-a984-f0f66e896357" (UID: "ec0f645b-289a-4b05-a984-f0f66e896357"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:07:06.785894 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:06.785863    4525 generic.go:356] "Generic (PLEG): container finished" podID="ec0f645b-289a-4b05-a984-f0f66e896357" containerID="a10bda61492f43d5ea9ee8f82b401bbf4fa5082b8e78a3ecbd39be5aea40410f" exitCode=0
Oct 30 19:07:06.786064 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:06.785920    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-ef6b56/auth-test-model-ef6b56-kserve-66c7fd94b9-qgqj4"
Oct 30 19:07:06.786064 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:06.785944    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-ef6b56/auth-test-model-ef6b56-kserve-66c7fd94b9-qgqj4" event={"ID":"ec0f645b-289a-4b05-a984-f0f66e896357","Type":"ContainerDied","Data":"a10bda61492f43d5ea9ee8f82b401bbf4fa5082b8e78a3ecbd39be5aea40410f"}
Oct 30 19:07:06.786064 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:06.785989    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-ef6b56/auth-test-model-ef6b56-kserve-66c7fd94b9-qgqj4" event={"ID":"ec0f645b-289a-4b05-a984-f0f66e896357","Type":"ContainerDied","Data":"779ce506fd37208bc6176de96803e4b686620d8f63a84b09c6852b84a3c6f17b"}
Oct 30 19:07:06.786064 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:06.786013    4525 scope.go:122] "RemoveContainer" containerID="a10bda61492f43d5ea9ee8f82b401bbf4fa5082b8e78a3ecbd39be5aea40410f"
Oct 30 19:07:06.788124 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:06.788105    4525 generic.go:356] "Generic (PLEG): container finished" podID="7ec64e5e-2c2b-47d8-8070-296ce41ada09" containerID="6370674b13dd336f02fd228a6ea078f74a5fbbdef9f26eedd9354ada3ed2fdbd" exitCode=0
Oct 30 19:07:06.788237 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:06.788184    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-46385c-openshift-default-6ccdb978d5-4gmh4" event={"ID":"7ec64e5e-2c2b-47d8-8070-296ce41ada09","Type":"ContainerDied","Data":"6370674b13dd336f02fd228a6ea078f74a5fbbdef9f26eedd9354ada3ed2fdbd"}
Oct 30 19:07:06.788927 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:06.788906    4525 scope.go:122] "RemoveContainer" containerID="6370674b13dd336f02fd228a6ea078f74a5fbbdef9f26eedd9354ada3ed2fdbd"
Oct 30 19:07:06.790418 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:06.790398    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1af7d73c-njhbh" event={"ID":"e8df5475-2c04-4c22-b53e-f01d9c4e6e32","Type":"ContainerStarted","Data":"c1b1837a21fb838f29057657902b42219cde8c8aa30b962129613f976b555ed2"}
Oct 30 19:07:06.790527 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:06.790439    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1af7d73c-njhbh" event={"ID":"e8df5475-2c04-4c22-b53e-f01d9c4e6e32","Type":"ContainerStarted","Data":"6be8cd4ca4b86b6142ff6707923b58a224cd748d0d85d5d05266d27ae727d855"}
Oct 30 19:07:06.790788 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:06.790762    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:07:06.790981 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:06.790958    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:07:06.791153 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:06.791132    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:07:06.791367 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:06.790787    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx"
Oct 30 19:07:06.791757 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:06.791738    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx"
Oct 30 19:07:06.799795 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:06.799776    4525 scope.go:122] "RemoveContainer" containerID="a10bda61492f43d5ea9ee8f82b401bbf4fa5082b8e78a3ecbd39be5aea40410f"
Oct 30 19:07:06.800095 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:07:06.800074    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a10bda61492f43d5ea9ee8f82b401bbf4fa5082b8e78a3ecbd39be5aea40410f\": container with ID starting with a10bda61492f43d5ea9ee8f82b401bbf4fa5082b8e78a3ecbd39be5aea40410f not found: ID does not exist" containerID="a10bda61492f43d5ea9ee8f82b401bbf4fa5082b8e78a3ecbd39be5aea40410f"
Oct 30 19:07:06.800172 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:06.800104    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a10bda61492f43d5ea9ee8f82b401bbf4fa5082b8e78a3ecbd39be5aea40410f"} err="failed to get container status \"a10bda61492f43d5ea9ee8f82b401bbf4fa5082b8e78a3ecbd39be5aea40410f\": rpc error: code = NotFound desc = could not find container \"a10bda61492f43d5ea9ee8f82b401bbf4fa5082b8e78a3ecbd39be5aea40410f\": container with ID starting with a10bda61492f43d5ea9ee8f82b401bbf4fa5082b8e78a3ecbd39be5aea40410f not found: ID does not exist"
Oct 30 19:07:06.817009 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:06.816951    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1af7d73c-njhbh" podStartSLOduration=1.816933445 podStartE2EDuration="1.816933445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:07:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:07:06.814984136 +0000 UTC m=+1849.440654100" watchObservedRunningTime="2026-10-30 19:07:06.816933445 +0000 UTC m=+1849.442603412"
Oct 30 19:07:06.924899 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:06.910037    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9094695-97bc-4924-9526-45b9e8892662-credential-socket\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-f5ct2\" (UID: \"d9094695-97bc-4924-9526-45b9e8892662\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-f5ct2"
Oct 30 19:07:06.924899 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:06.910242    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9094695-97bc-4924-9526-45b9e8892662-istio-token\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-f5ct2\" (UID: \"d9094695-97bc-4924-9526-45b9e8892662\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-f5ct2"
Oct 30 19:07:06.924899 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:06.910276    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9094695-97bc-4924-9526-45b9e8892662-workload-certs\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-f5ct2\" (UID: \"d9094695-97bc-4924-9526-45b9e8892662\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-f5ct2"
Oct 30 19:07:06.924899 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:06.910331    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9094695-97bc-4924-9526-45b9e8892662-istio-envoy\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-f5ct2\" (UID: \"d9094695-97bc-4924-9526-45b9e8892662\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-f5ct2"
Oct 30 19:07:06.924899 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:06.910436    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9094695-97bc-4924-9526-45b9e8892662-istio-data\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-f5ct2\" (UID: \"d9094695-97bc-4924-9526-45b9e8892662\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-f5ct2"
Oct 30 19:07:06.924899 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:06.910546    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9094695-97bc-4924-9526-45b9e8892662-istiod-ca-cert\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-f5ct2\" (UID: \"d9094695-97bc-4924-9526-45b9e8892662\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-f5ct2"
Oct 30 19:07:06.924899 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:06.910589    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9094695-97bc-4924-9526-45b9e8892662-workload-socket\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-f5ct2\" (UID: \"d9094695-97bc-4924-9526-45b9e8892662\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-f5ct2"
Oct 30 19:07:06.924899 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:06.910657    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjxkq\" (UniqueName: \"kubernetes.io/projected/d9094695-97bc-4924-9526-45b9e8892662-kube-api-access-gjxkq\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-f5ct2\" (UID: \"d9094695-97bc-4924-9526-45b9e8892662\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-f5ct2"
Oct 30 19:07:06.924899 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:06.910747    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9094695-97bc-4924-9526-45b9e8892662-istio-podinfo\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-f5ct2\" (UID: \"d9094695-97bc-4924-9526-45b9e8892662\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-f5ct2"
Oct 30 19:07:06.952707 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:06.929350    4525 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec0f645b-289a-4b05-a984-f0f66e896357-dshm\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:06.952707 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:06.944367    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9094695-97bc-4924-9526-45b9e8892662-credential-socket\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-f5ct2\" (UID: \"d9094695-97bc-4924-9526-45b9e8892662\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-f5ct2"
Oct 30 19:07:06.952707 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:06.944801    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9094695-97bc-4924-9526-45b9e8892662-workload-certs\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-f5ct2\" (UID: \"d9094695-97bc-4924-9526-45b9e8892662\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-f5ct2"
Oct 30 19:07:06.952707 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:06.945365    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9094695-97bc-4924-9526-45b9e8892662-istiod-ca-cert\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-f5ct2\" (UID: \"d9094695-97bc-4924-9526-45b9e8892662\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-f5ct2"
Oct 30 19:07:06.964707 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:06.954046    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9094695-97bc-4924-9526-45b9e8892662-workload-socket\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-f5ct2\" (UID: \"d9094695-97bc-4924-9526-45b9e8892662\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-f5ct2"
Oct 30 19:07:06.964707 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:06.956069    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9094695-97bc-4924-9526-45b9e8892662-istio-data\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-f5ct2\" (UID: \"d9094695-97bc-4924-9526-45b9e8892662\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-f5ct2"
Oct 30 19:07:07.067805 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:07.061108    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9094695-97bc-4924-9526-45b9e8892662-istio-token\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-f5ct2\" (UID: \"d9094695-97bc-4924-9526-45b9e8892662\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-f5ct2"
Oct 30 19:07:07.107845 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:07.107749    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-ef6b56/auth-test-model-ef6b56-kserve-66c7fd94b9-qgqj4"]
Oct 30 19:07:07.242358 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:07.242280    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9094695-97bc-4924-9526-45b9e8892662-istio-envoy\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-f5ct2\" (UID: \"d9094695-97bc-4924-9526-45b9e8892662\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-f5ct2"
Oct 30 19:07:07.242546 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:07.242451    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9094695-97bc-4924-9526-45b9e8892662-istio-podinfo\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-f5ct2\" (UID: \"d9094695-97bc-4924-9526-45b9e8892662\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-f5ct2"
Oct 30 19:07:07.248755 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:07.248727    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-ef6b56/auth-test-model-ef6b56-kserve-66c7fd94b9-qgqj4"]
Oct 30 19:07:07.266037 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:07.265790    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjxkq\" (UniqueName: \"kubernetes.io/projected/d9094695-97bc-4924-9526-45b9e8892662-kube-api-access-gjxkq\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-f5ct2\" (UID: \"d9094695-97bc-4924-9526-45b9e8892662\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-f5ct2"
Oct 30 19:07:07.293908 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:07.293874    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-f5ct2"
Oct 30 19:07:07.445148 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:07.445109    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-ef6b56/auth-test-model-ef6b56-kserve-66c7fd94b9-qgqj4" podUID="ec0f645b-289a-4b05-a984-f0f66e896357" containerName="main" probeResult="failure" output="Get \"https://10.133.0.199:8000/ready\": context deadline exceeded"
Oct 30 19:07:07.548262 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:07.548184    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec0f645b-289a-4b05-a984-f0f66e896357" path="/var/lib/kubelet/pods/ec0f645b-289a-4b05-a984-f0f66e896357/volumes"
Oct 30 19:07:07.607328 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:07.607297    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-f5ct2"]
Oct 30 19:07:07.607911 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:07:07.607886    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9094695_97bc_4924_9526_45b9e8892662.slice/crio-94f38ffacf702d2e801acf0d1dc31eb30c924937b8f973785e6b9202554eb8da WatchSource:0}: Error finding container 94f38ffacf702d2e801acf0d1dc31eb30c924937b8f973785e6b9202554eb8da: Status 404 returned error can't find the container with id 94f38ffacf702d2e801acf0d1dc31eb30c924937b8f973785e6b9202554eb8da
Oct 30 19:07:07.611516 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:07.611480    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:07:07.611591 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:07.611552    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:07:07.611591 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:07.611582    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:07:07.821154 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:07.821061    4525 generic.go:356] "Generic (PLEG): container finished" podID="e8df5475-2c04-4c22-b53e-f01d9c4e6e32" containerID="c1b1837a21fb838f29057657902b42219cde8c8aa30b962129613f976b555ed2" exitCode=0
Oct 30 19:07:07.821336 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:07.821165    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1af7d73c-njhbh" event={"ID":"e8df5475-2c04-4c22-b53e-f01d9c4e6e32","Type":"ContainerDied","Data":"c1b1837a21fb838f29057657902b42219cde8c8aa30b962129613f976b555ed2"}
Oct 30 19:07:07.824645 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:07.824621    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-f5ct2" event={"ID":"d9094695-97bc-4924-9526-45b9e8892662","Type":"ContainerStarted","Data":"e2d446f40cf1dd6c12c1badf1f879711f7cb234dd8dca27e13a442572a339ac4"}
Oct 30 19:07:07.824825 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:07.824706    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-f5ct2" event={"ID":"d9094695-97bc-4924-9526-45b9e8892662","Type":"ContainerStarted","Data":"94f38ffacf702d2e801acf0d1dc31eb30c924937b8f973785e6b9202554eb8da"}
Oct 30 19:07:07.829613 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:07.829596    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-46385c-openshift-default-6ccdb978d5-4gmh4" event={"ID":"7ec64e5e-2c2b-47d8-8070-296ce41ada09","Type":"ContainerStarted","Data":"fbd1aac2be1a8e31597e2f8ead67cba4c839c6c061cd745c17b63ed13c0d0f2a"}
Oct 30 19:07:08.302707 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:08.302603    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-f5ct2"
Oct 30 19:07:08.308579 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:08.308554    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-f5ct2"
Oct 30 19:07:08.342820 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:08.342759    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-f5ct2" podStartSLOduration=2.342743232 podStartE2EDuration="2.342743232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:07:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:07:07.898742119 +0000 UTC m=+1850.524412087" watchObservedRunningTime="2026-10-30 19:07:08.342743232 +0000 UTC m=+1850.968413196"
Oct 30 19:07:08.765070 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:08.765037    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-46385c-openshift-default-6ccdb978d5-4gmh4"
Oct 30 19:07:08.766404 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:08.766371    4525 patch_prober.go:29] interesting pod/e2e-shared-b-w1-46385c-openshift-default-6ccdb978d5-4gmh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body=
Oct 30 19:07:08.766550 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:08.766426    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-46385c-openshift-default-6ccdb978d5-4gmh4" podUID="7ec64e5e-2c2b-47d8-8070-296ce41ada09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused"
Oct 30 19:07:08.955840 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:08.955804    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-f5ct2"
Oct 30 19:07:08.971196 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:08.971162    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-f5ct2"
Oct 30 19:07:09.237928 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:09.237898    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1af7d73c-njhbh"
Oct 30 19:07:09.357601 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:09.357571    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8df5475-2c04-4c22-b53e-f01d9c4e6e32-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8df5475-2c04-4c22-b53e-f01d9c4e6e32-maas-api-openshift-ca\") pod \"e8df5475-2c04-4c22-b53e-f01d9c4e6e32\" (UID: \"e8df5475-2c04-4c22-b53e-f01d9c4e6e32\") "
Oct 30 19:07:09.357917 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:09.357894    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8df5475-2c04-4c22-b53e-f01d9c4e6e32-kube-api-access-tcdwj\" (UniqueName: \"kubernetes.io/projected/e8df5475-2c04-4c22-b53e-f01d9c4e6e32-kube-api-access-tcdwj\") pod \"e8df5475-2c04-4c22-b53e-f01d9c4e6e32\" (UID: \"e8df5475-2c04-4c22-b53e-f01d9c4e6e32\") "
Oct 30 19:07:09.357983 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:09.357958    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8df5475-2c04-4c22-b53e-f01d9c4e6e32-maas-api-openshift-ca" pod "e8df5475-2c04-4c22-b53e-f01d9c4e6e32" (UID: "e8df5475-2c04-4c22-b53e-f01d9c4e6e32"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:07:09.358025 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:09.357998    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8df5475-2c04-4c22-b53e-f01d9c4e6e32-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8df5475-2c04-4c22-b53e-f01d9c4e6e32-maas-api-xks-ca\") pod \"e8df5475-2c04-4c22-b53e-f01d9c4e6e32\" (UID: \"e8df5475-2c04-4c22-b53e-f01d9c4e6e32\") "
Oct 30 19:07:09.358766 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:09.358633    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8df5475-2c04-4c22-b53e-f01d9c4e6e32-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:09.360238 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:09.360218    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8df5475-2c04-4c22-b53e-f01d9c4e6e32-kube-api-access-tcdwj" pod "e8df5475-2c04-4c22-b53e-f01d9c4e6e32" (UID: "e8df5475-2c04-4c22-b53e-f01d9c4e6e32"). InnerVolumeSpecName "kube-api-access-tcdwj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:07:09.360238 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:09.360230    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8df5475-2c04-4c22-b53e-f01d9c4e6e32-maas-api-xks-ca" pod "e8df5475-2c04-4c22-b53e-f01d9c4e6e32" (UID: "e8df5475-2c04-4c22-b53e-f01d9c4e6e32"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:07:09.460091 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:09.460060    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcdwj\" (UniqueName: \"kubernetes.io/projected/e8df5475-2c04-4c22-b53e-f01d9c4e6e32-kube-api-access-tcdwj\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:09.460091 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:09.460089    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8df5475-2c04-4c22-b53e-f01d9c4e6e32-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:09.766768 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:09.766050    4525 patch_prober.go:29] interesting pod/e2e-shared-b-w1-46385c-openshift-default-6ccdb978d5-4gmh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body=
Oct 30 19:07:09.766768 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:09.766128    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-46385c-openshift-default-6ccdb978d5-4gmh4" podUID="7ec64e5e-2c2b-47d8-8070-296ce41ada09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused"
Oct 30 19:07:10.238695 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:10.238639    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1af7d73c-njhbh"
Oct 30 19:07:10.244140 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:10.244110    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1af7d73c-njhbh" event={"ID":"e8df5475-2c04-4c22-b53e-f01d9c4e6e32","Type":"ContainerDied","Data":"6be8cd4ca4b86b6142ff6707923b58a224cd748d0d85d5d05266d27ae727d855"}
Oct 30 19:07:10.247289 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:10.247254    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6be8cd4ca4b86b6142ff6707923b58a224cd748d0d85d5d05266d27ae727d855"
Oct 30 19:07:10.765202 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:10.765169    4525 patch_prober.go:29] interesting pod/e2e-shared-b-w1-46385c-openshift-default-6ccdb978d5-4gmh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body=
Oct 30 19:07:10.765381 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:10.765222    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-46385c-openshift-default-6ccdb978d5-4gmh4" podUID="7ec64e5e-2c2b-47d8-8070-296ce41ada09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused"
Oct 30 19:07:11.773726 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:11.771517    4525 patch_prober.go:29] interesting pod/e2e-shared-b-w1-46385c-openshift-default-6ccdb978d5-4gmh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body=
Oct 30 19:07:11.773726 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:11.771580    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-46385c-openshift-default-6ccdb978d5-4gmh4" podUID="7ec64e5e-2c2b-47d8-8070-296ce41ada09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused"
Oct 30 19:07:12.390852 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:12.390816    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-46385c-openshift-default-6ccdb978d5-4gmh4"
Oct 30 19:07:12.764671 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:12.764584    4525 patch_prober.go:29] interesting pod/e2e-shared-b-w1-46385c-openshift-default-6ccdb978d5-4gmh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body=
Oct 30 19:07:12.764671 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:12.764649    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-46385c-openshift-default-6ccdb978d5-4gmh4" podUID="7ec64e5e-2c2b-47d8-8070-296ce41ada09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused"
Oct 30 19:07:13.764378 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:13.764342    4525 patch_prober.go:29] interesting pod/e2e-shared-b-w1-46385c-openshift-default-6ccdb978d5-4gmh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body=
Oct 30 19:07:13.764837 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:13.764393    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-46385c-openshift-default-6ccdb978d5-4gmh4" podUID="7ec64e5e-2c2b-47d8-8070-296ce41ada09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused"
Oct 30 19:07:14.769866 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:14.769834    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-46385c-openshift-default-6ccdb978d5-4gmh4"
Oct 30 19:07:14.771156 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:14.771134    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-46385c-openshift-default-6ccdb978d5-4gmh4"
Oct 30 19:07:15.437803 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:15.437769    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dhv28"]
Oct 30 19:07:15.438001 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:15.437966    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dhv28" podUID="bd3970a9-8e49-456f-b7c7-b6af7320dfab" containerName="istio-proxy" containerID="cri-o://660496b207a9a7640e6fe57760be261b29f5df1864cbcbfea328fc946d68dcae" gracePeriod=30
Oct 30 19:07:17.018161 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:17.018131    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c2e3f9/auth-test-model-c2e3f9-kserve-6b9f58d9cc-pmkg2"]
Oct 30 19:07:17.402741 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:07:17.402702    4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d49l" containerName="extractor"
Oct 30 19:07:20.488529 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:20.488504    4525 scope.go:122] "RemoveContainer" containerID="aaeaeb3aa4f109147bd96c39e1544aa5951de9b7f5912e3adaeb28c56b78c883"
Oct 30 19:07:20.552467 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:20.552440    4525 scope.go:122] "RemoveContainer" containerID="7310e87ef76e1f0498b0be40911bcbd08b7610a3ba2e5e7ca2ff884346550777"
Oct 30 19:07:20.562734 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:20.562714    4525 scope.go:122] "RemoveContainer" containerID="c99b64138a16b37233f5a8674187e9132786288e6c839c483a9ef40ae4d3825b"
Oct 30 19:07:20.573224 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:20.573109    4525 scope.go:122] "RemoveContainer" containerID="7b81547120503d9584661d9fb11bd06c14b52c2bc240f1e2eb2c1ff9402e778e"
Oct 30 19:07:20.583896 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:20.583873    4525 scope.go:122] "RemoveContainer" containerID="41ad94f62444fcc6d76752b311e26d4cc4e5839d15448c1202b8a47b62b95266"
Oct 30 19:07:20.594549 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:20.594524    4525 scope.go:122] "RemoveContainer" containerID="35e92a1dce5ec9c65b77fc41286d318e6d9512a8d7369abcd0b1d38f11ee4a81"
Oct 30 19:07:20.604924 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:20.604904    4525 scope.go:122] "RemoveContainer" containerID="76e15e6ca54c0aab216dd95559367fe1e5e4bbf8cbcfc604ab2a64b4b9dac0eb"
Oct 30 19:07:20.721554 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:20.721505    4525 scope.go:122] "RemoveContainer" containerID="acbc54293ceb9cc43d9cbd294ad19a5eba04ff2443138d843c01ec0e2a600d8e"
Oct 30 19:07:20.755010 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:20.754981    4525 scope.go:122] "RemoveContainer" containerID="2f4509a742eef5429f304848bc20f625eb2177c435666b97307d5d6fe9a62dd2"
Oct 30 19:07:20.773853 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:20.773825    4525 scope.go:122] "RemoveContainer" containerID="7aa78e891057a9e05eacf34765ccb80b48f34138a556bcc7b80535eff42db3c3"
Oct 30 19:07:20.788441 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:20.788416    4525 scope.go:122] "RemoveContainer" containerID="cabb95626b839ca4d6c8cfd21fed0970f54fa7b02d355f56745abc3070fd5c63"
Oct 30 19:07:20.802506 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:20.802464    4525 scope.go:122] "RemoveContainer" containerID="d6e3d41bacb177aa6e14ac9966710b901a3708bdbfae153d48a48ee14d727789"
Oct 30 19:07:20.839845 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:20.838840    4525 scope.go:122] "RemoveContainer" containerID="15a05a403aa48a93970925ee05c193139594e86595bc6b8d4429aaa68f79d782"
Oct 30 19:07:20.839845 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:20.839280    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dhv28"
Oct 30 19:07:20.850480 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:20.850464    4525 scope.go:122] "RemoveContainer" containerID="4f7c188b7b4e5a51dc6d7099519dd4bc1436ec1781642754f3c9c3d29ca11009"
Oct 30 19:07:20.861238 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:20.861210    4525 scope.go:122] "RemoveContainer" containerID="5dd62a8df86f68ce7a012b10637fc9189959475bcb961817382d06a4312cceff"
Oct 30 19:07:20.945008 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:20.944984    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd3970a9-8e49-456f-b7c7-b6af7320dfab-istio-token\" (UniqueName: \"kubernetes.io/projected/bd3970a9-8e49-456f-b7c7-b6af7320dfab-istio-token\") pod \"bd3970a9-8e49-456f-b7c7-b6af7320dfab\" (UID: \"bd3970a9-8e49-456f-b7c7-b6af7320dfab\") "
Oct 30 19:07:20.945091 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:20.945052    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd3970a9-8e49-456f-b7c7-b6af7320dfab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd3970a9-8e49-456f-b7c7-b6af7320dfab-istiod-ca-cert\") pod \"bd3970a9-8e49-456f-b7c7-b6af7320dfab\" (UID: \"bd3970a9-8e49-456f-b7c7-b6af7320dfab\") "
Oct 30 19:07:20.945161 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:20.945087    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd3970a9-8e49-456f-b7c7-b6af7320dfab-kube-api-access-vg7c5\" (UniqueName: \"kubernetes.io/projected/bd3970a9-8e49-456f-b7c7-b6af7320dfab-kube-api-access-vg7c5\") pod \"bd3970a9-8e49-456f-b7c7-b6af7320dfab\" (UID: \"bd3970a9-8e49-456f-b7c7-b6af7320dfab\") "
Oct 30 19:07:20.945161 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:20.945156    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd3970a9-8e49-456f-b7c7-b6af7320dfab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd3970a9-8e49-456f-b7c7-b6af7320dfab-istio-data\") pod \"bd3970a9-8e49-456f-b7c7-b6af7320dfab\" (UID: \"bd3970a9-8e49-456f-b7c7-b6af7320dfab\") "
Oct 30 19:07:20.945277 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:20.945225    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd3970a9-8e49-456f-b7c7-b6af7320dfab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd3970a9-8e49-456f-b7c7-b6af7320dfab-istio-envoy\") pod \"bd3970a9-8e49-456f-b7c7-b6af7320dfab\" (UID: \"bd3970a9-8e49-456f-b7c7-b6af7320dfab\") "
Oct 30 19:07:20.945359 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:20.945338    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bd3970a9-8e49-456f-b7c7-b6af7320dfab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd3970a9-8e49-456f-b7c7-b6af7320dfab-istio-podinfo\") pod \"bd3970a9-8e49-456f-b7c7-b6af7320dfab\" (UID: \"bd3970a9-8e49-456f-b7c7-b6af7320dfab\") "
Oct 30 19:07:20.945639 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:20.945452    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd3970a9-8e49-456f-b7c7-b6af7320dfab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd3970a9-8e49-456f-b7c7-b6af7320dfab-credential-socket\") pod \"bd3970a9-8e49-456f-b7c7-b6af7320dfab\" (UID: \"bd3970a9-8e49-456f-b7c7-b6af7320dfab\") "
Oct 30 19:07:20.945639 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:20.945484    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd3970a9-8e49-456f-b7c7-b6af7320dfab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd3970a9-8e49-456f-b7c7-b6af7320dfab-workload-certs\") pod \"bd3970a9-8e49-456f-b7c7-b6af7320dfab\" (UID: \"bd3970a9-8e49-456f-b7c7-b6af7320dfab\") "
Oct 30 19:07:20.945639 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:20.945531    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd3970a9-8e49-456f-b7c7-b6af7320dfab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd3970a9-8e49-456f-b7c7-b6af7320dfab-workload-socket\") pod \"bd3970a9-8e49-456f-b7c7-b6af7320dfab\" (UID: \"bd3970a9-8e49-456f-b7c7-b6af7320dfab\") "
Oct 30 19:07:20.945877 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:20.945478    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd3970a9-8e49-456f-b7c7-b6af7320dfab-istiod-ca-cert" pod "bd3970a9-8e49-456f-b7c7-b6af7320dfab" (UID: "bd3970a9-8e49-456f-b7c7-b6af7320dfab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:07:20.945877 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:20.945737    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd3970a9-8e49-456f-b7c7-b6af7320dfab-credential-socket" pod "bd3970a9-8e49-456f-b7c7-b6af7320dfab" (UID: "bd3970a9-8e49-456f-b7c7-b6af7320dfab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:07:20.945877 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:20.945719    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd3970a9-8e49-456f-b7c7-b6af7320dfab-istio-data" pod "bd3970a9-8e49-456f-b7c7-b6af7320dfab" (UID: "bd3970a9-8e49-456f-b7c7-b6af7320dfab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:07:20.946046 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:20.945880    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd3970a9-8e49-456f-b7c7-b6af7320dfab-workload-certs" pod "bd3970a9-8e49-456f-b7c7-b6af7320dfab" (UID: "bd3970a9-8e49-456f-b7c7-b6af7320dfab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:07:20.946046 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:20.946013    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd3970a9-8e49-456f-b7c7-b6af7320dfab-workload-socket" pod "bd3970a9-8e49-456f-b7c7-b6af7320dfab" (UID: "bd3970a9-8e49-456f-b7c7-b6af7320dfab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:07:20.946433 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:20.946408    4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd3970a9-8e49-456f-b7c7-b6af7320dfab-istiod-ca-cert\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:20.946522 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:20.946436    4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd3970a9-8e49-456f-b7c7-b6af7320dfab-istio-data\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:20.946522 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:20.946451    4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd3970a9-8e49-456f-b7c7-b6af7320dfab-credential-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:20.946522 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:20.946464    4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd3970a9-8e49-456f-b7c7-b6af7320dfab-workload-certs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:20.946522 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:20.946476    4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd3970a9-8e49-456f-b7c7-b6af7320dfab-workload-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:20.947780 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:20.947754    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd3970a9-8e49-456f-b7c7-b6af7320dfab-istio-token" pod "bd3970a9-8e49-456f-b7c7-b6af7320dfab" (UID: "bd3970a9-8e49-456f-b7c7-b6af7320dfab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:07:20.948318 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:20.948294    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bd3970a9-8e49-456f-b7c7-b6af7320dfab-istio-podinfo" pod "bd3970a9-8e49-456f-b7c7-b6af7320dfab" (UID: "bd3970a9-8e49-456f-b7c7-b6af7320dfab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:07:20.948520 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:20.948498    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd3970a9-8e49-456f-b7c7-b6af7320dfab-kube-api-access-vg7c5" pod "bd3970a9-8e49-456f-b7c7-b6af7320dfab" (UID: "bd3970a9-8e49-456f-b7c7-b6af7320dfab"). InnerVolumeSpecName "kube-api-access-vg7c5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:07:20.952076 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:20.949694    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd3970a9-8e49-456f-b7c7-b6af7320dfab-istio-envoy" pod "bd3970a9-8e49-456f-b7c7-b6af7320dfab" (UID: "bd3970a9-8e49-456f-b7c7-b6af7320dfab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:07:21.047970 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:21.047936    4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd3970a9-8e49-456f-b7c7-b6af7320dfab-istio-token\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:21.047970 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:21.047965    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vg7c5\" (UniqueName: \"kubernetes.io/projected/bd3970a9-8e49-456f-b7c7-b6af7320dfab-kube-api-access-vg7c5\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:21.048127 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:21.047981    4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd3970a9-8e49-456f-b7c7-b6af7320dfab-istio-envoy\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:21.048127 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:21.047994    4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd3970a9-8e49-456f-b7c7-b6af7320dfab-istio-podinfo\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:21.767990 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:21.765394    4525 generic.go:356] "Generic (PLEG): container finished" podID="bd3970a9-8e49-456f-b7c7-b6af7320dfab" containerID="660496b207a9a7640e6fe57760be261b29f5df1864cbcbfea328fc946d68dcae" exitCode=0
Oct 30 19:07:21.767990 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:21.765617    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dhv28"
Oct 30 19:07:21.824347 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:21.823762    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dhv28" event={"ID":"bd3970a9-8e49-456f-b7c7-b6af7320dfab","Type":"ContainerDied","Data":"660496b207a9a7640e6fe57760be261b29f5df1864cbcbfea328fc946d68dcae"}
Oct 30 19:07:21.824347 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:21.823841    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dhv28" event={"ID":"bd3970a9-8e49-456f-b7c7-b6af7320dfab","Type":"ContainerDied","Data":"06166f2f56488c3c9ba3c28cc0b565a2937e61e6995419e97d605498dca091f5"}
Oct 30 19:07:21.824347 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:21.823890    4525 scope.go:122] "RemoveContainer" containerID="660496b207a9a7640e6fe57760be261b29f5df1864cbcbfea328fc946d68dcae"
Oct 30 19:07:21.850488 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:21.850441    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dhv28"]
Oct 30 19:07:21.853291 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:21.853273    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-dhv28"]
Oct 30 19:07:21.853372 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:21.853351    4525 scope.go:122] "RemoveContainer" containerID="660496b207a9a7640e6fe57760be261b29f5df1864cbcbfea328fc946d68dcae"
Oct 30 19:07:21.853732 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:07:21.853711    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"660496b207a9a7640e6fe57760be261b29f5df1864cbcbfea328fc946d68dcae\": container with ID starting with 660496b207a9a7640e6fe57760be261b29f5df1864cbcbfea328fc946d68dcae not found: ID does not exist" containerID="660496b207a9a7640e6fe57760be261b29f5df1864cbcbfea328fc946d68dcae"
Oct 30 19:07:21.853812 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:21.853747    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"660496b207a9a7640e6fe57760be261b29f5df1864cbcbfea328fc946d68dcae"} err="failed to get container status \"660496b207a9a7640e6fe57760be261b29f5df1864cbcbfea328fc946d68dcae\": rpc error: code = NotFound desc = could not find container \"660496b207a9a7640e6fe57760be261b29f5df1864cbcbfea328fc946d68dcae\": container with ID starting with 660496b207a9a7640e6fe57760be261b29f5df1864cbcbfea328fc946d68dcae not found: ID does not exist"
Oct 30 19:07:23.373717 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:23.371094    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b"]
Oct 30 19:07:23.373717 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:23.372185    4525 cpu_manager.go:414] "containerMap: removing container" podUID="e8df5475-2c04-4c22-b53e-f01d9c4e6e32" containerName="revoke-keys"
Oct 30 19:07:23.373717 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:23.372207    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8df5475-2c04-4c22-b53e-f01d9c4e6e32" containerName="revoke-keys"
Oct 30 19:07:23.373717 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:23.372345    4525 cpu_manager.go:414] "containerMap: removing container" podUID="bd3970a9-8e49-456f-b7c7-b6af7320dfab" containerName="istio-proxy"
Oct 30 19:07:23.373717 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:23.372357    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd3970a9-8e49-456f-b7c7-b6af7320dfab" containerName="istio-proxy"
Oct 30 19:07:23.373717 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:23.372589    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd3970a9-8e49-456f-b7c7-b6af7320dfab" containerName="istio-proxy"
Oct 30 19:07:23.373717 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:23.372607    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8df5475-2c04-4c22-b53e-f01d9c4e6e32" containerName="revoke-keys"
Oct 30 19:07:23.379729 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:23.379673    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b"
Oct 30 19:07:23.382013 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:23.381984    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b"]
Oct 30 19:07:23.394166 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:23.393137    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-e51c10-openshift-default-dockercfg-s9hfx\""
Oct 30 19:07:23.551705 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:23.548737    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8e73ad5-17f6-4205-bc5f-8965afa49f57-istio-podinfo\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b\" (UID: \"a8e73ad5-17f6-4205-bc5f-8965afa49f57\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b"
Oct 30 19:07:23.551705 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:23.548813    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8e73ad5-17f6-4205-bc5f-8965afa49f57-istiod-ca-cert\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b\" (UID: \"a8e73ad5-17f6-4205-bc5f-8965afa49f57\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b"
Oct 30 19:07:23.551705 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:23.548844    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdlfj\" (UniqueName: \"kubernetes.io/projected/a8e73ad5-17f6-4205-bc5f-8965afa49f57-kube-api-access-hdlfj\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b\" (UID: \"a8e73ad5-17f6-4205-bc5f-8965afa49f57\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b"
Oct 30 19:07:23.551705 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:23.548967    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8e73ad5-17f6-4205-bc5f-8965afa49f57-workload-socket\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b\" (UID: \"a8e73ad5-17f6-4205-bc5f-8965afa49f57\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b"
Oct 30 19:07:23.551705 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:23.548999    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8e73ad5-17f6-4205-bc5f-8965afa49f57-credential-socket\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b\" (UID: \"a8e73ad5-17f6-4205-bc5f-8965afa49f57\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b"
Oct 30 19:07:23.551705 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:23.549138    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8e73ad5-17f6-4205-bc5f-8965afa49f57-istio-data\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b\" (UID: \"a8e73ad5-17f6-4205-bc5f-8965afa49f57\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b"
Oct 30 19:07:23.551705 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:23.549192    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8e73ad5-17f6-4205-bc5f-8965afa49f57-workload-certs\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b\" (UID: \"a8e73ad5-17f6-4205-bc5f-8965afa49f57\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b"
Oct 30 19:07:23.551705 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:23.549221    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8e73ad5-17f6-4205-bc5f-8965afa49f57-istio-envoy\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b\" (UID: \"a8e73ad5-17f6-4205-bc5f-8965afa49f57\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b"
Oct 30 19:07:23.551705 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:23.549318    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8e73ad5-17f6-4205-bc5f-8965afa49f57-istio-token\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b\" (UID: \"a8e73ad5-17f6-4205-bc5f-8965afa49f57\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b"
Oct 30 19:07:23.612701 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:23.607348    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd3970a9-8e49-456f-b7c7-b6af7320dfab" path="/var/lib/kubelet/pods/bd3970a9-8e49-456f-b7c7-b6af7320dfab/volumes"
Oct 30 19:07:23.655765 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:23.654807    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8e73ad5-17f6-4205-bc5f-8965afa49f57-workload-socket\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b\" (UID: \"a8e73ad5-17f6-4205-bc5f-8965afa49f57\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b"
Oct 30 19:07:23.655765 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:23.654871    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8e73ad5-17f6-4205-bc5f-8965afa49f57-credential-socket\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b\" (UID: \"a8e73ad5-17f6-4205-bc5f-8965afa49f57\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b"
Oct 30 19:07:23.655765 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:23.655010    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8e73ad5-17f6-4205-bc5f-8965afa49f57-istio-data\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b\" (UID: \"a8e73ad5-17f6-4205-bc5f-8965afa49f57\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b"
Oct 30 19:07:23.655765 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:23.655062    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8e73ad5-17f6-4205-bc5f-8965afa49f57-workload-certs\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b\" (UID: \"a8e73ad5-17f6-4205-bc5f-8965afa49f57\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b"
Oct 30 19:07:23.655765 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:23.655096    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8e73ad5-17f6-4205-bc5f-8965afa49f57-istio-envoy\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b\" (UID: \"a8e73ad5-17f6-4205-bc5f-8965afa49f57\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b"
Oct 30 19:07:23.655765 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:23.655190    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8e73ad5-17f6-4205-bc5f-8965afa49f57-istio-token\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b\" (UID: \"a8e73ad5-17f6-4205-bc5f-8965afa49f57\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b"
Oct 30 19:07:23.655765 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:23.655269    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8e73ad5-17f6-4205-bc5f-8965afa49f57-istio-podinfo\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b\" (UID: \"a8e73ad5-17f6-4205-bc5f-8965afa49f57\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b"
Oct 30 19:07:23.655765 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:23.655316    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8e73ad5-17f6-4205-bc5f-8965afa49f57-istiod-ca-cert\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b\" (UID: \"a8e73ad5-17f6-4205-bc5f-8965afa49f57\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b"
Oct 30 19:07:23.655765 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:23.655347    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdlfj\" (UniqueName: \"kubernetes.io/projected/a8e73ad5-17f6-4205-bc5f-8965afa49f57-kube-api-access-hdlfj\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b\" (UID: \"a8e73ad5-17f6-4205-bc5f-8965afa49f57\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b"
Oct 30 19:07:23.656967 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:23.656363    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8e73ad5-17f6-4205-bc5f-8965afa49f57-workload-socket\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b\" (UID: \"a8e73ad5-17f6-4205-bc5f-8965afa49f57\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b"
Oct 30 19:07:23.656967 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:23.656604    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8e73ad5-17f6-4205-bc5f-8965afa49f57-credential-socket\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b\" (UID: \"a8e73ad5-17f6-4205-bc5f-8965afa49f57\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b"
Oct 30 19:07:23.656967 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:23.656841    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8e73ad5-17f6-4205-bc5f-8965afa49f57-istio-data\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b\" (UID: \"a8e73ad5-17f6-4205-bc5f-8965afa49f57\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b"
Oct 30 19:07:23.657141 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:23.657071    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8e73ad5-17f6-4205-bc5f-8965afa49f57-workload-certs\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b\" (UID: \"a8e73ad5-17f6-4205-bc5f-8965afa49f57\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b"
Oct 30 19:07:23.660711 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:23.660617    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8e73ad5-17f6-4205-bc5f-8965afa49f57-istiod-ca-cert\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b\" (UID: \"a8e73ad5-17f6-4205-bc5f-8965afa49f57\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b"
Oct 30 19:07:23.705749 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:23.696990    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8e73ad5-17f6-4205-bc5f-8965afa49f57-istio-envoy\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b\" (UID: \"a8e73ad5-17f6-4205-bc5f-8965afa49f57\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b"
Oct 30 19:07:23.705749 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:23.697458    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8e73ad5-17f6-4205-bc5f-8965afa49f57-istio-token\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b\" (UID: \"a8e73ad5-17f6-4205-bc5f-8965afa49f57\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b"
Oct 30 19:07:23.705749 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:23.698187    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdlfj\" (UniqueName: \"kubernetes.io/projected/a8e73ad5-17f6-4205-bc5f-8965afa49f57-kube-api-access-hdlfj\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b\" (UID: \"a8e73ad5-17f6-4205-bc5f-8965afa49f57\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b"
Oct 30 19:07:23.705749 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:23.702860    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8e73ad5-17f6-4205-bc5f-8965afa49f57-istio-podinfo\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b\" (UID: \"a8e73ad5-17f6-4205-bc5f-8965afa49f57\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b"
Oct 30 19:07:23.998296 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:23.998217    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b"
Oct 30 19:07:24.525307 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:24.525263    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:07:24.525716 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:24.525358    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:07:24.525716 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:24.525412    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:07:24.566862 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:24.566830    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b"]
Oct 30 19:07:24.889704 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:24.884156    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b" event={"ID":"a8e73ad5-17f6-4205-bc5f-8965afa49f57","Type":"ContainerStarted","Data":"1cfe61de8850135e71e9c6cd76641bf7efac9ef158b8006ebc4be79165fb08e3"}
Oct 30 19:07:24.889704 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:24.884252    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b" event={"ID":"a8e73ad5-17f6-4205-bc5f-8965afa49f57","Type":"ContainerStarted","Data":"974c48842cdd2e0e53e65c05d6dd060e7865303ea74d8976ce5a8ab5ca56e8c0"}
Oct 30 19:07:24.922727 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:24.921328    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b" podStartSLOduration=1.921307495 podStartE2EDuration="1.921307495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:07:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:07:24.919423498 +0000 UTC m=+1867.545093463" watchObservedRunningTime="2026-10-30 19:07:24.921307495 +0000 UTC m=+1867.546977459"
Oct 30 19:07:25.005887 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:25.005786    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b"
Oct 30 19:07:25.010751 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:25.009964    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body=
Oct 30 19:07:25.010751 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:25.010020    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b" podUID="a8e73ad5-17f6-4205-bc5f-8965afa49f57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused"
Oct 30 19:07:26.003964 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:26.003932    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b"
Oct 30 19:07:26.952354 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:26.952323    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b"
Oct 30 19:07:26.953404 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:26.953380    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b"
Oct 30 19:07:27.060488 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:27.060437    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4"]
Oct 30 19:07:27.066139 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:27.066112    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4"
Oct 30 19:07:27.072475 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:27.072435    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-4a477d-openshift-default-dockercfg-6bds5\""
Oct 30 19:07:27.123729 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:27.114831    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4"]
Oct 30 19:07:27.147540 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:27.147503    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0650ae0c-36c1-4b6e-9212-12264a92c920-istio-token\") pod \"e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4\" (UID: \"0650ae0c-36c1-4b6e-9212-12264a92c920\") " pod="openshift-ingress/e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4"
Oct 30 19:07:27.147758 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:27.147550    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0650ae0c-36c1-4b6e-9212-12264a92c920-workload-socket\") pod \"e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4\" (UID: \"0650ae0c-36c1-4b6e-9212-12264a92c920\") " pod="openshift-ingress/e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4"
Oct 30 19:07:27.147758 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:27.147699    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0650ae0c-36c1-4b6e-9212-12264a92c920-workload-certs\") pod \"e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4\" (UID: \"0650ae0c-36c1-4b6e-9212-12264a92c920\") " pod="openshift-ingress/e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4"
Oct 30 19:07:27.147919 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:27.147765    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0650ae0c-36c1-4b6e-9212-12264a92c920-istiod-ca-cert\") pod \"e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4\" (UID: \"0650ae0c-36c1-4b6e-9212-12264a92c920\") " pod="openshift-ingress/e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4"
Oct 30 19:07:27.147919 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:27.147813    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0650ae0c-36c1-4b6e-9212-12264a92c920-istio-envoy\") pod \"e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4\" (UID: \"0650ae0c-36c1-4b6e-9212-12264a92c920\") " pod="openshift-ingress/e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4"
Oct 30 19:07:27.147919 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:27.147894    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0650ae0c-36c1-4b6e-9212-12264a92c920-istio-data\") pod \"e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4\" (UID: \"0650ae0c-36c1-4b6e-9212-12264a92c920\") " pod="openshift-ingress/e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4"
Oct 30 19:07:27.148101 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:27.147931    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjnk5\" (UniqueName: \"kubernetes.io/projected/0650ae0c-36c1-4b6e-9212-12264a92c920-kube-api-access-wjnk5\") pod \"e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4\" (UID: \"0650ae0c-36c1-4b6e-9212-12264a92c920\") " pod="openshift-ingress/e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4"
Oct 30 19:07:27.148101 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:27.147981    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0650ae0c-36c1-4b6e-9212-12264a92c920-credential-socket\") pod \"e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4\" (UID: \"0650ae0c-36c1-4b6e-9212-12264a92c920\") " pod="openshift-ingress/e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4"
Oct 30 19:07:27.148101 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:27.148081    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0650ae0c-36c1-4b6e-9212-12264a92c920-istio-podinfo\") pod \"e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4\" (UID: \"0650ae0c-36c1-4b6e-9212-12264a92c920\") " pod="openshift-ingress/e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4"
Oct 30 19:07:27.250697 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:27.249982    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0650ae0c-36c1-4b6e-9212-12264a92c920-workload-certs\") pod \"e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4\" (UID: \"0650ae0c-36c1-4b6e-9212-12264a92c920\") " pod="openshift-ingress/e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4"
Oct 30 19:07:27.250697 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:27.250092    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0650ae0c-36c1-4b6e-9212-12264a92c920-istiod-ca-cert\") pod \"e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4\" (UID: \"0650ae0c-36c1-4b6e-9212-12264a92c920\") " pod="openshift-ingress/e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4"
Oct 30 19:07:27.250697 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:27.250151    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0650ae0c-36c1-4b6e-9212-12264a92c920-istio-envoy\") pod \"e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4\" (UID: \"0650ae0c-36c1-4b6e-9212-12264a92c920\") " pod="openshift-ingress/e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4"
Oct 30 19:07:27.250697 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:27.250232    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0650ae0c-36c1-4b6e-9212-12264a92c920-istio-data\") pod \"e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4\" (UID: \"0650ae0c-36c1-4b6e-9212-12264a92c920\") " pod="openshift-ingress/e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4"
Oct 30 19:07:27.250697 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:27.250317    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjnk5\" (UniqueName: \"kubernetes.io/projected/0650ae0c-36c1-4b6e-9212-12264a92c920-kube-api-access-wjnk5\") pod \"e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4\" (UID: \"0650ae0c-36c1-4b6e-9212-12264a92c920\") " pod="openshift-ingress/e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4"
Oct 30 19:07:27.250697 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:27.250369    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0650ae0c-36c1-4b6e-9212-12264a92c920-credential-socket\") pod \"e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4\" (UID: \"0650ae0c-36c1-4b6e-9212-12264a92c920\") " pod="openshift-ingress/e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4"
Oct 30 19:07:27.250697 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:27.250420    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0650ae0c-36c1-4b6e-9212-12264a92c920-istio-podinfo\") pod \"e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4\" (UID: \"0650ae0c-36c1-4b6e-9212-12264a92c920\") " pod="openshift-ingress/e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4"
Oct 30 19:07:27.250697 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:27.250602    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0650ae0c-36c1-4b6e-9212-12264a92c920-istio-token\") pod \"e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4\" (UID: \"0650ae0c-36c1-4b6e-9212-12264a92c920\") " pod="openshift-ingress/e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4"
Oct 30 19:07:27.250697 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:27.250634    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0650ae0c-36c1-4b6e-9212-12264a92c920-workload-socket\") pod \"e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4\" (UID: \"0650ae0c-36c1-4b6e-9212-12264a92c920\") " pod="openshift-ingress/e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4"
Oct 30 19:07:27.251284 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:27.251153    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0650ae0c-36c1-4b6e-9212-12264a92c920-workload-socket\") pod \"e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4\" (UID: \"0650ae0c-36c1-4b6e-9212-12264a92c920\") " pod="openshift-ingress/e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4"
Oct 30 19:07:27.262733 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:27.251797    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0650ae0c-36c1-4b6e-9212-12264a92c920-workload-certs\") pod \"e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4\" (UID: \"0650ae0c-36c1-4b6e-9212-12264a92c920\") " pod="openshift-ingress/e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4"
Oct 30 19:07:27.262733 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:27.252383    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0650ae0c-36c1-4b6e-9212-12264a92c920-istiod-ca-cert\") pod \"e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4\" (UID: \"0650ae0c-36c1-4b6e-9212-12264a92c920\") " pod="openshift-ingress/e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4"
Oct 30 19:07:27.262733 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:27.260288    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0650ae0c-36c1-4b6e-9212-12264a92c920-istio-envoy\") pod \"e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4\" (UID: \"0650ae0c-36c1-4b6e-9212-12264a92c920\") " pod="openshift-ingress/e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4"
Oct 30 19:07:27.262733 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:27.260550    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0650ae0c-36c1-4b6e-9212-12264a92c920-istio-data\") pod \"e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4\" (UID: \"0650ae0c-36c1-4b6e-9212-12264a92c920\") " pod="openshift-ingress/e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4"
Oct 30 19:07:27.262733 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:27.261135    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0650ae0c-36c1-4b6e-9212-12264a92c920-credential-socket\") pod \"e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4\" (UID: \"0650ae0c-36c1-4b6e-9212-12264a92c920\") " pod="openshift-ingress/e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4"
Oct 30 19:07:27.295707 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:27.285864    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjnk5\" (UniqueName: \"kubernetes.io/projected/0650ae0c-36c1-4b6e-9212-12264a92c920-kube-api-access-wjnk5\") pod \"e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4\" (UID: \"0650ae0c-36c1-4b6e-9212-12264a92c920\") " pod="openshift-ingress/e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4"
Oct 30 19:07:27.356161 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:27.353913    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0650ae0c-36c1-4b6e-9212-12264a92c920-istio-podinfo\") pod \"e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4\" (UID: \"0650ae0c-36c1-4b6e-9212-12264a92c920\") " pod="openshift-ingress/e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4"
Oct 30 19:07:27.356161 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:27.355478    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0650ae0c-36c1-4b6e-9212-12264a92c920-istio-token\") pod \"e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4\" (UID: \"0650ae0c-36c1-4b6e-9212-12264a92c920\") " pod="openshift-ingress/e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4"
Oct 30 19:07:27.443725 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:27.440046    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4"
Oct 30 19:07:28.065096 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:07:28.065054    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0650ae0c_36c1_4b6e_9212_12264a92c920.slice/crio-62c6d60a54184c45b9f3bee2bc6561554c860c80a8ce722ce63b658a934f1184 WatchSource:0}: Error finding container 62c6d60a54184c45b9f3bee2bc6561554c860c80a8ce722ce63b658a934f1184: Status 404 returned error can't find the container with id 62c6d60a54184c45b9f3bee2bc6561554c860c80a8ce722ce63b658a934f1184
Oct 30 19:07:28.068994 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:28.068961    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4"]
Oct 30 19:07:28.069382 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:28.069332    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:07:28.069552 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:28.069535    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:07:28.069642 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:28.069631    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:07:29.010164 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:29.010129    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4" event={"ID":"0650ae0c-36c1-4b6e-9212-12264a92c920","Type":"ContainerStarted","Data":"ddde0bd43d503e1026196638a207f5250f5d53e85a6bd1119d79bc22736da3a3"}
Oct 30 19:07:29.010372 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:29.010185    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4" event={"ID":"0650ae0c-36c1-4b6e-9212-12264a92c920","Type":"ContainerStarted","Data":"62c6d60a54184c45b9f3bee2bc6561554c860c80a8ce722ce63b658a934f1184"}
Oct 30 19:07:29.161704 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:29.158849    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4" podStartSLOduration=2.158828175 podStartE2EDuration="2.158828175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:07:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:07:29.030283716 +0000 UTC m=+1871.655953697" watchObservedRunningTime="2026-10-30 19:07:29.158828175 +0000 UTC m=+1871.784498143"
Oct 30 19:07:29.445329 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:29.445298    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4"
Oct 30 19:07:29.537113 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:29.537082    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4"
Oct 30 19:07:30.019673 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:30.019639    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4"
Oct 30 19:07:30.020839 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:30.020813    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4"
Oct 30 19:07:32.016718 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:32.013775    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cf3c4bb-4zdf4"]
Oct 30 19:07:32.055948 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:32.048189    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-c2e3f9/auth-test-model-c2e3f9-kserve-6b9f58d9cc-pmkg2" podUID="fedf2771-d37f-4d43-83f4-a04290b9fa9c" containerName="main" containerID="cri-o://db0a0c54e501691d218078b68d968fd9e3da84b12465ce8c84d2b75d0ef38dfb" gracePeriod=45
Oct 30 19:07:32.081016 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:32.080480    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf3c4bb-4zdf4"
Oct 30 19:07:32.102407 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:32.102381    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:07:32.114007 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:32.113978    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cf3c4bb-4zdf4"]
Oct 30 19:07:32.117005 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:32.116974    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94hcn\""
Oct 30 19:07:32.147573 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:32.147338    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-c2e3f9/auth-test-model-c2e3f9-kserve-6b9f58d9cc-pmkg2" podUID="fedf2771-d37f-4d43-83f4-a04290b9fa9c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.203:8000/ready\": dial tcp 10.133.0.203:8000: connect: connection refused"
Oct 30 19:07:32.269755 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:32.268809    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b963374-1226-411e-87b0-642297fad691-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5cf3c4bb-4zdf4\" (UID: \"6b963374-1226-411e-87b0-642297fad691\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf3c4bb-4zdf4"
Oct 30 19:07:32.269755 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:32.269021    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7t4x\" (UniqueName: \"kubernetes.io/projected/6b963374-1226-411e-87b0-642297fad691-kube-api-access-m7t4x\") pod \"maas-api-revoke-sub-5cf3c4bb-4zdf4\" (UID: \"6b963374-1226-411e-87b0-642297fad691\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf3c4bb-4zdf4"
Oct 30 19:07:32.269755 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:32.269745    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b963374-1226-411e-87b0-642297fad691-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5cf3c4bb-4zdf4\" (UID: \"6b963374-1226-411e-87b0-642297fad691\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf3c4bb-4zdf4"
Oct 30 19:07:32.372057 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:32.371235    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b963374-1226-411e-87b0-642297fad691-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5cf3c4bb-4zdf4\" (UID: \"6b963374-1226-411e-87b0-642297fad691\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf3c4bb-4zdf4"
Oct 30 19:07:32.372057 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:32.371418    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b963374-1226-411e-87b0-642297fad691-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5cf3c4bb-4zdf4\" (UID: \"6b963374-1226-411e-87b0-642297fad691\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf3c4bb-4zdf4"
Oct 30 19:07:32.372057 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:32.371575    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7t4x\" (UniqueName: \"kubernetes.io/projected/6b963374-1226-411e-87b0-642297fad691-kube-api-access-m7t4x\") pod \"maas-api-revoke-sub-5cf3c4bb-4zdf4\" (UID: \"6b963374-1226-411e-87b0-642297fad691\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf3c4bb-4zdf4"
Oct 30 19:07:32.373744 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:32.373696    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b963374-1226-411e-87b0-642297fad691-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5cf3c4bb-4zdf4\" (UID: \"6b963374-1226-411e-87b0-642297fad691\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf3c4bb-4zdf4"
Oct 30 19:07:32.412387 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:32.412350    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b963374-1226-411e-87b0-642297fad691-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5cf3c4bb-4zdf4\" (UID: \"6b963374-1226-411e-87b0-642297fad691\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf3c4bb-4zdf4"
Oct 30 19:07:32.413038 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:32.413013    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7t4x\" (UniqueName: \"kubernetes.io/projected/6b963374-1226-411e-87b0-642297fad691-kube-api-access-m7t4x\") pod \"maas-api-revoke-sub-5cf3c4bb-4zdf4\" (UID: \"6b963374-1226-411e-87b0-642297fad691\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf3c4bb-4zdf4"
Oct 30 19:07:32.424970 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:32.424942    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf3c4bb-4zdf4"
Oct 30 19:07:32.503708 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:32.501995    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-c2e3f9/auth-test-model-c2e3f9-kserve-6b9f58d9cc-pmkg2"
Oct 30 19:07:32.707733 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:32.681716    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fedf2771-d37f-4d43-83f4-a04290b9fa9c-tls-certs\" (UniqueName: \"kubernetes.io/secret/fedf2771-d37f-4d43-83f4-a04290b9fa9c-tls-certs\") pod \"fedf2771-d37f-4d43-83f4-a04290b9fa9c\" (UID: \"fedf2771-d37f-4d43-83f4-a04290b9fa9c\") "
Oct 30 19:07:32.707733 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:32.681808    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fedf2771-d37f-4d43-83f4-a04290b9fa9c-kube-api-access-z4t9m\" (UniqueName: \"kubernetes.io/projected/fedf2771-d37f-4d43-83f4-a04290b9fa9c-kube-api-access-z4t9m\") pod \"fedf2771-d37f-4d43-83f4-a04290b9fa9c\" (UID: \"fedf2771-d37f-4d43-83f4-a04290b9fa9c\") "
Oct 30 19:07:32.707733 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:32.681855    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fedf2771-d37f-4d43-83f4-a04290b9fa9c-home\" (UniqueName: \"kubernetes.io/empty-dir/fedf2771-d37f-4d43-83f4-a04290b9fa9c-home\") pod \"fedf2771-d37f-4d43-83f4-a04290b9fa9c\" (UID: \"fedf2771-d37f-4d43-83f4-a04290b9fa9c\") "
Oct 30 19:07:32.707733 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:32.682056    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fedf2771-d37f-4d43-83f4-a04290b9fa9c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fedf2771-d37f-4d43-83f4-a04290b9fa9c-model-cache\") pod \"fedf2771-d37f-4d43-83f4-a04290b9fa9c\" (UID: \"fedf2771-d37f-4d43-83f4-a04290b9fa9c\") "
Oct 30 19:07:32.707733 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:32.682250    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fedf2771-d37f-4d43-83f4-a04290b9fa9c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fedf2771-d37f-4d43-83f4-a04290b9fa9c-tmp-dir\") pod \"fedf2771-d37f-4d43-83f4-a04290b9fa9c\" (UID: \"fedf2771-d37f-4d43-83f4-a04290b9fa9c\") "
Oct 30 19:07:32.707733 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:32.682282    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fedf2771-d37f-4d43-83f4-a04290b9fa9c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fedf2771-d37f-4d43-83f4-a04290b9fa9c-dshm\") pod \"fedf2771-d37f-4d43-83f4-a04290b9fa9c\" (UID: \"fedf2771-d37f-4d43-83f4-a04290b9fa9c\") "
Oct 30 19:07:32.707733 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:32.705288    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fedf2771-d37f-4d43-83f4-a04290b9fa9c-home" pod "fedf2771-d37f-4d43-83f4-a04290b9fa9c" (UID: "fedf2771-d37f-4d43-83f4-a04290b9fa9c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:07:32.707733 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:32.707410    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fedf2771-d37f-4d43-83f4-a04290b9fa9c-model-cache" pod "fedf2771-d37f-4d43-83f4-a04290b9fa9c" (UID: "fedf2771-d37f-4d43-83f4-a04290b9fa9c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:07:32.707733 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:32.707592    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fedf2771-d37f-4d43-83f4-a04290b9fa9c-tmp-dir" pod "fedf2771-d37f-4d43-83f4-a04290b9fa9c" (UID: "fedf2771-d37f-4d43-83f4-a04290b9fa9c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:07:32.708842 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:32.708817    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fedf2771-d37f-4d43-83f4-a04290b9fa9c-dshm" pod "fedf2771-d37f-4d43-83f4-a04290b9fa9c" (UID: "fedf2771-d37f-4d43-83f4-a04290b9fa9c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:07:32.709421 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:32.709392    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fedf2771-d37f-4d43-83f4-a04290b9fa9c-tls-certs" pod "fedf2771-d37f-4d43-83f4-a04290b9fa9c" (UID: "fedf2771-d37f-4d43-83f4-a04290b9fa9c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:07:32.712385 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:32.712111    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fedf2771-d37f-4d43-83f4-a04290b9fa9c-kube-api-access-z4t9m" pod "fedf2771-d37f-4d43-83f4-a04290b9fa9c" (UID: "fedf2771-d37f-4d43-83f4-a04290b9fa9c"). InnerVolumeSpecName "kube-api-access-z4t9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:07:32.748028 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:07:32.747998    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b963374_1226_411e_87b0_642297fad691.slice/crio-90231477c7bc97660a59f7a1bc06038c1c17e1ecbffc25179506833e81efbcf8 WatchSource:0}: Error finding container 90231477c7bc97660a59f7a1bc06038c1c17e1ecbffc25179506833e81efbcf8: Status 404 returned error can't find the container with id 90231477c7bc97660a59f7a1bc06038c1c17e1ecbffc25179506833e81efbcf8
Oct 30 19:07:32.749533 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:32.749515    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cf3c4bb-4zdf4"]
Oct 30 19:07:32.785197 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:32.784668    4525 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fedf2771-d37f-4d43-83f4-a04290b9fa9c-model-cache\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:32.785197 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:32.784720    4525 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fedf2771-d37f-4d43-83f4-a04290b9fa9c-tmp-dir\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:32.785197 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:32.784734    4525 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fedf2771-d37f-4d43-83f4-a04290b9fa9c-dshm\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:32.785197 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:32.784747    4525 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fedf2771-d37f-4d43-83f4-a04290b9fa9c-tls-certs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:32.785197 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:32.784761    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4t9m\" (UniqueName: \"kubernetes.io/projected/fedf2771-d37f-4d43-83f4-a04290b9fa9c-kube-api-access-z4t9m\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:32.785197 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:32.784775    4525 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fedf2771-d37f-4d43-83f4-a04290b9fa9c-home\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:33.237136 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:33.237042    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf3c4bb-4zdf4" event={"ID":"6b963374-1226-411e-87b0-642297fad691","Type":"ContainerStarted","Data":"90231477c7bc97660a59f7a1bc06038c1c17e1ecbffc25179506833e81efbcf8"}
Oct 30 19:07:33.260779 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:33.253484    4525 generic.go:356] "Generic (PLEG): container finished" podID="fedf2771-d37f-4d43-83f4-a04290b9fa9c" containerID="db0a0c54e501691d218078b68d968fd9e3da84b12465ce8c84d2b75d0ef38dfb" exitCode=0
Oct 30 19:07:33.260779 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:33.253673    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-c2e3f9/auth-test-model-c2e3f9-kserve-6b9f58d9cc-pmkg2" event={"ID":"fedf2771-d37f-4d43-83f4-a04290b9fa9c","Type":"ContainerDied","Data":"db0a0c54e501691d218078b68d968fd9e3da84b12465ce8c84d2b75d0ef38dfb"}
Oct 30 19:07:33.260779 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:33.253735    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-c2e3f9/auth-test-model-c2e3f9-kserve-6b9f58d9cc-pmkg2" event={"ID":"fedf2771-d37f-4d43-83f4-a04290b9fa9c","Type":"ContainerDied","Data":"b5e9a46ffd085278d35584f89511bbacaeee5b42909d9db2427c8dad5722a1d9"}
Oct 30 19:07:33.260779 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:33.253757    4525 scope.go:122] "RemoveContainer" containerID="db0a0c54e501691d218078b68d968fd9e3da84b12465ce8c84d2b75d0ef38dfb"
Oct 30 19:07:33.260779 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:33.253961    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-c2e3f9/auth-test-model-c2e3f9-kserve-6b9f58d9cc-pmkg2"
Oct 30 19:07:33.303829 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:33.303807    4525 scope.go:122] "RemoveContainer" containerID="db0a0c54e501691d218078b68d968fd9e3da84b12465ce8c84d2b75d0ef38dfb"
Oct 30 19:07:33.304284 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:07:33.304225    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db0a0c54e501691d218078b68d968fd9e3da84b12465ce8c84d2b75d0ef38dfb\": container with ID starting with db0a0c54e501691d218078b68d968fd9e3da84b12465ce8c84d2b75d0ef38dfb not found: ID does not exist" containerID="db0a0c54e501691d218078b68d968fd9e3da84b12465ce8c84d2b75d0ef38dfb"
Oct 30 19:07:33.304403 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:33.304297    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db0a0c54e501691d218078b68d968fd9e3da84b12465ce8c84d2b75d0ef38dfb"} err="failed to get container status \"db0a0c54e501691d218078b68d968fd9e3da84b12465ce8c84d2b75d0ef38dfb\": rpc error: code = NotFound desc = could not find container \"db0a0c54e501691d218078b68d968fd9e3da84b12465ce8c84d2b75d0ef38dfb\": container with ID starting with db0a0c54e501691d218078b68d968fd9e3da84b12465ce8c84d2b75d0ef38dfb not found: ID does not exist"
Oct 30 19:07:33.312077 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:33.312053    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c2e3f9/auth-test-model-c2e3f9-kserve-6b9f58d9cc-pmkg2"]
Oct 30 19:07:33.318168 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:33.318143    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c2e3f9/auth-test-model-c2e3f9-kserve-6b9f58d9cc-pmkg2"]
Oct 30 19:07:33.546829 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:33.546752    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fedf2771-d37f-4d43-83f4-a04290b9fa9c" path="/var/lib/kubelet/pods/fedf2771-d37f-4d43-83f4-a04290b9fa9c/volumes"
Oct 30 19:07:34.264005 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:34.263918    4525 generic.go:356] "Generic (PLEG): container finished" podID="6b963374-1226-411e-87b0-642297fad691" containerID="fe25cbc30a6ea4051a77fd6b9acd67aaccee9d58d48e9b3cb7226e971f822317" exitCode=0
Oct 30 19:07:34.264440 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:34.263999    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf3c4bb-4zdf4" event={"ID":"6b963374-1226-411e-87b0-642297fad691","Type":"ContainerDied","Data":"fe25cbc30a6ea4051a77fd6b9acd67aaccee9d58d48e9b3cb7226e971f822317"}
Oct 30 19:07:35.881427 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:35.881396    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf3c4bb-4zdf4"
Oct 30 19:07:35.953705 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:35.945303    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b963374-1226-411e-87b0-642297fad691-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b963374-1226-411e-87b0-642297fad691-maas-api-openshift-ca\") pod \"6b963374-1226-411e-87b0-642297fad691\" (UID: \"6b963374-1226-411e-87b0-642297fad691\") "
Oct 30 19:07:35.953705 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:35.945390    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b963374-1226-411e-87b0-642297fad691-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b963374-1226-411e-87b0-642297fad691-maas-api-xks-ca\") pod \"6b963374-1226-411e-87b0-642297fad691\" (UID: \"6b963374-1226-411e-87b0-642297fad691\") "
Oct 30 19:07:35.953705 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:35.945450    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b963374-1226-411e-87b0-642297fad691-kube-api-access-m7t4x\" (UniqueName: \"kubernetes.io/projected/6b963374-1226-411e-87b0-642297fad691-kube-api-access-m7t4x\") pod \"6b963374-1226-411e-87b0-642297fad691\" (UID: \"6b963374-1226-411e-87b0-642297fad691\") "
Oct 30 19:07:35.970341 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:35.968911    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b963374-1226-411e-87b0-642297fad691-maas-api-openshift-ca" pod "6b963374-1226-411e-87b0-642297fad691" (UID: "6b963374-1226-411e-87b0-642297fad691"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:07:35.971012 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:35.970971    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b963374-1226-411e-87b0-642297fad691-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:36.004104 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:36.004021    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b963374-1226-411e-87b0-642297fad691-kube-api-access-m7t4x" pod "6b963374-1226-411e-87b0-642297fad691" (UID: "6b963374-1226-411e-87b0-642297fad691"). InnerVolumeSpecName "kube-api-access-m7t4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:07:36.004489 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:36.004469    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b963374-1226-411e-87b0-642297fad691-maas-api-xks-ca" pod "6b963374-1226-411e-87b0-642297fad691" (UID: "6b963374-1226-411e-87b0-642297fad691"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:07:36.073202 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:36.073160    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b963374-1226-411e-87b0-642297fad691-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:36.073202 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:36.073202    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7t4x\" (UniqueName: \"kubernetes.io/projected/6b963374-1226-411e-87b0-642297fad691-kube-api-access-m7t4x\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:36.304648 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:36.304549    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf3c4bb-4zdf4" event={"ID":"6b963374-1226-411e-87b0-642297fad691","Type":"ContainerDied","Data":"90231477c7bc97660a59f7a1bc06038c1c17e1ecbffc25179506833e81efbcf8"}
Oct 30 19:07:36.304648 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:36.304613    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90231477c7bc97660a59f7a1bc06038c1c17e1ecbffc25179506833e81efbcf8"
Oct 30 19:07:36.304908 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:36.304725    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf3c4bb-4zdf4"
Oct 30 19:07:39.394120 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:39.393911    4525 generic.go:356] "Generic (PLEG): container finished" podID="84c44eb7-6483-4f83-ab59-447f04ba5d1d" containerID="aa921ad64f5ffd5ff960ec19542da03046337d91f1e9db386923174b3632c874" exitCode=0
Oct 30 19:07:39.394120 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:39.394023    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ca2cd5-openshift-default-59664bf6df-xqn5s" event={"ID":"84c44eb7-6483-4f83-ab59-447f04ba5d1d","Type":"ContainerDied","Data":"aa921ad64f5ffd5ff960ec19542da03046337d91f1e9db386923174b3632c874"}
Oct 30 19:07:39.394729 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:39.394652    4525 scope.go:122] "RemoveContainer" containerID="aa921ad64f5ffd5ff960ec19542da03046337d91f1e9db386923174b3632c874"
Oct 30 19:07:39.419879 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:39.419494    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:07:39.420224 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:39.420190    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:07:39.420417 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:39.420397    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:07:40.410933 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:40.410900    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ca2cd5-openshift-default-59664bf6df-xqn5s" event={"ID":"84c44eb7-6483-4f83-ab59-447f04ba5d1d","Type":"ContainerStarted","Data":"6842a25642aa14d8c33d61180051c1fc518646a16433f9ea1fc5b7ac15e5364a"}
Oct 30 19:07:41.138660 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:41.138630    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-ca2cd5-openshift-default-59664bf6df-xqn5s"
Oct 30 19:07:41.138876 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:41.138804    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-ca2cd5-openshift-default-59664bf6df-xqn5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body=
Oct 30 19:07:41.138876 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:41.138855    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ca2cd5-openshift-default-59664bf6df-xqn5s" podUID="84c44eb7-6483-4f83-ab59-447f04ba5d1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused"
Oct 30 19:07:42.155048 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:42.153024    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-ca2cd5-openshift-default-59664bf6df-xqn5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body=
Oct 30 19:07:42.155048 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:42.153093    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ca2cd5-openshift-default-59664bf6df-xqn5s" podUID="84c44eb7-6483-4f83-ab59-447f04ba5d1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused"
Oct 30 19:07:42.259493 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:42.259454    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf63472c-v2vbc"]
Oct 30 19:07:42.260202 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:42.260180    4525 cpu_manager.go:414] "containerMap: removing container" podUID="fedf2771-d37f-4d43-83f4-a04290b9fa9c" containerName="main"
Oct 30 19:07:42.260202 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:42.260203    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fedf2771-d37f-4d43-83f4-a04290b9fa9c" containerName="main"
Oct 30 19:07:42.260503 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:42.260275    4525 cpu_manager.go:414] "containerMap: removing container" podUID="6b963374-1226-411e-87b0-642297fad691" containerName="revoke-keys"
Oct 30 19:07:42.260503 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:42.260286    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b963374-1226-411e-87b0-642297fad691" containerName="revoke-keys"
Oct 30 19:07:42.260503 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:42.260460    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="fedf2771-d37f-4d43-83f4-a04290b9fa9c" containerName="main"
Oct 30 19:07:42.260503 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:42.260488    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b963374-1226-411e-87b0-642297fad691" containerName="revoke-keys"
Oct 30 19:07:42.264281 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:42.264262    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf63472c-v2vbc"
Oct 30 19:07:42.271467 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:42.271371    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94hcn\""
Oct 30 19:07:42.271467 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:42.271441    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:07:42.289712 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:42.276114    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf63472c-v2vbc"]
Oct 30 19:07:42.449602 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:42.449517    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/363cd59e-8ddd-43a6-91e9-f2c1b577948b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf63472c-v2vbc\" (UID: \"363cd59e-8ddd-43a6-91e9-f2c1b577948b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf63472c-v2vbc"
Oct 30 19:07:42.449783 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:42.449755    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhqr6\" (UniqueName: \"kubernetes.io/projected/363cd59e-8ddd-43a6-91e9-f2c1b577948b-kube-api-access-rhqr6\") pod \"maas-api-revoke-sub-cf63472c-v2vbc\" (UID: \"363cd59e-8ddd-43a6-91e9-f2c1b577948b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf63472c-v2vbc"
Oct 30 19:07:42.449860 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:42.449846    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/363cd59e-8ddd-43a6-91e9-f2c1b577948b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf63472c-v2vbc\" (UID: \"363cd59e-8ddd-43a6-91e9-f2c1b577948b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf63472c-v2vbc"
Oct 30 19:07:42.552623 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:42.552584    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/363cd59e-8ddd-43a6-91e9-f2c1b577948b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf63472c-v2vbc\" (UID: \"363cd59e-8ddd-43a6-91e9-f2c1b577948b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf63472c-v2vbc"
Oct 30 19:07:42.552960 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:42.552938    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhqr6\" (UniqueName: \"kubernetes.io/projected/363cd59e-8ddd-43a6-91e9-f2c1b577948b-kube-api-access-rhqr6\") pod \"maas-api-revoke-sub-cf63472c-v2vbc\" (UID: \"363cd59e-8ddd-43a6-91e9-f2c1b577948b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf63472c-v2vbc"
Oct 30 19:07:42.553034 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:42.553025    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/363cd59e-8ddd-43a6-91e9-f2c1b577948b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf63472c-v2vbc\" (UID: \"363cd59e-8ddd-43a6-91e9-f2c1b577948b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf63472c-v2vbc"
Oct 30 19:07:42.554145 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:42.554117    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/363cd59e-8ddd-43a6-91e9-f2c1b577948b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf63472c-v2vbc\" (UID: \"363cd59e-8ddd-43a6-91e9-f2c1b577948b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf63472c-v2vbc"
Oct 30 19:07:42.555469 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:42.555430    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/363cd59e-8ddd-43a6-91e9-f2c1b577948b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf63472c-v2vbc\" (UID: \"363cd59e-8ddd-43a6-91e9-f2c1b577948b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf63472c-v2vbc"
Oct 30 19:07:42.566752 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:42.566732    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhqr6\" (UniqueName: \"kubernetes.io/projected/363cd59e-8ddd-43a6-91e9-f2c1b577948b-kube-api-access-rhqr6\") pod \"maas-api-revoke-sub-cf63472c-v2vbc\" (UID: \"363cd59e-8ddd-43a6-91e9-f2c1b577948b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf63472c-v2vbc"
Oct 30 19:07:42.611530 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:42.609960    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf63472c-v2vbc"
Oct 30 19:07:43.139248 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:43.139212    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-ca2cd5-openshift-default-59664bf6df-xqn5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body=
Oct 30 19:07:43.139464 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:43.139286    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ca2cd5-openshift-default-59664bf6df-xqn5s" podUID="84c44eb7-6483-4f83-ab59-447f04ba5d1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused"
Oct 30 19:07:43.214807 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:43.214642    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf63472c-v2vbc"]
Oct 30 19:07:43.218388 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:07:43.217494    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod363cd59e_8ddd_43a6_91e9_f2c1b577948b.slice/crio-2420b37fcc58ffccfd87021188b0c6d04adba3c529742a47031dc7b12761e902 WatchSource:0}: Error finding container 2420b37fcc58ffccfd87021188b0c6d04adba3c529742a47031dc7b12761e902: Status 404 returned error can't find the container with id 2420b37fcc58ffccfd87021188b0c6d04adba3c529742a47031dc7b12761e902
Oct 30 19:07:43.436220 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:43.436062    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf63472c-v2vbc" event={"ID":"363cd59e-8ddd-43a6-91e9-f2c1b577948b","Type":"ContainerStarted","Data":"2420b37fcc58ffccfd87021188b0c6d04adba3c529742a47031dc7b12761e902"}
Oct 30 19:07:44.141049 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:44.139544    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-ca2cd5-openshift-default-59664bf6df-xqn5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body=
Oct 30 19:07:44.141049 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:44.139603    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ca2cd5-openshift-default-59664bf6df-xqn5s" podUID="84c44eb7-6483-4f83-ab59-447f04ba5d1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused"
Oct 30 19:07:44.526581 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:44.514384    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf63472c-v2vbc" event={"ID":"363cd59e-8ddd-43a6-91e9-f2c1b577948b","Type":"ContainerStarted","Data":"f92e360403e1fbcf91c9c7b5a9948880a6617e8b5d1571c517ec4df81624912a"}
Oct 30 19:07:44.607044 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:44.598375    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf63472c-v2vbc" podStartSLOduration=2.598356937 podStartE2EDuration="2.598356937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:07:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:07:44.571121576 +0000 UTC m=+1887.196791582" watchObservedRunningTime="2026-10-30 19:07:44.598356937 +0000 UTC m=+1887.224026904"
Oct 30 19:07:45.138628 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:45.138580    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-ca2cd5-openshift-default-59664bf6df-xqn5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body=
Oct 30 19:07:45.138910 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:45.138663    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ca2cd5-openshift-default-59664bf6df-xqn5s" podUID="84c44eb7-6483-4f83-ab59-447f04ba5d1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused"
Oct 30 19:07:45.417620 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:45.417551    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7"]
Oct 30 19:07:45.451733 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:45.447035    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7"
Oct 30 19:07:45.455995 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:45.455968    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7"]
Oct 30 19:07:45.459699 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:45.459651    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-e67d95-openshift-default-dockercfg-jdrgt\""
Oct 30 19:07:45.542707 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:45.519567    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77adecb1-af53-4dd9-bb52-d8e22ebb03e4-istio-podinfo\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7\" (UID: \"77adecb1-af53-4dd9-bb52-d8e22ebb03e4\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7"
Oct 30 19:07:45.542707 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:45.519648    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77adecb1-af53-4dd9-bb52-d8e22ebb03e4-credential-socket\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7\" (UID: \"77adecb1-af53-4dd9-bb52-d8e22ebb03e4\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7"
Oct 30 19:07:45.542707 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:45.519846    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77adecb1-af53-4dd9-bb52-d8e22ebb03e4-istiod-ca-cert\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7\" (UID: \"77adecb1-af53-4dd9-bb52-d8e22ebb03e4\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7"
Oct 30 19:07:45.542707 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:45.519878    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzx5n\" (UniqueName: \"kubernetes.io/projected/77adecb1-af53-4dd9-bb52-d8e22ebb03e4-kube-api-access-nzx5n\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7\" (UID: \"77adecb1-af53-4dd9-bb52-d8e22ebb03e4\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7"
Oct 30 19:07:45.542707 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:45.519906    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77adecb1-af53-4dd9-bb52-d8e22ebb03e4-istio-envoy\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7\" (UID: \"77adecb1-af53-4dd9-bb52-d8e22ebb03e4\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7"
Oct 30 19:07:45.542707 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:45.519948    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77adecb1-af53-4dd9-bb52-d8e22ebb03e4-workload-certs\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7\" (UID: \"77adecb1-af53-4dd9-bb52-d8e22ebb03e4\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7"
Oct 30 19:07:45.542707 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:45.519973    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77adecb1-af53-4dd9-bb52-d8e22ebb03e4-istio-data\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7\" (UID: \"77adecb1-af53-4dd9-bb52-d8e22ebb03e4\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7"
Oct 30 19:07:45.542707 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:45.520013    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77adecb1-af53-4dd9-bb52-d8e22ebb03e4-workload-socket\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7\" (UID: \"77adecb1-af53-4dd9-bb52-d8e22ebb03e4\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7"
Oct 30 19:07:45.542707 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:45.520289    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77adecb1-af53-4dd9-bb52-d8e22ebb03e4-istio-token\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7\" (UID: \"77adecb1-af53-4dd9-bb52-d8e22ebb03e4\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7"
Oct 30 19:07:45.649706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:45.644091    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77adecb1-af53-4dd9-bb52-d8e22ebb03e4-istiod-ca-cert\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7\" (UID: \"77adecb1-af53-4dd9-bb52-d8e22ebb03e4\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7"
Oct 30 19:07:45.649706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:45.644149    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzx5n\" (UniqueName: \"kubernetes.io/projected/77adecb1-af53-4dd9-bb52-d8e22ebb03e4-kube-api-access-nzx5n\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7\" (UID: \"77adecb1-af53-4dd9-bb52-d8e22ebb03e4\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7"
Oct 30 19:07:45.649706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:45.644179    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77adecb1-af53-4dd9-bb52-d8e22ebb03e4-istio-envoy\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7\" (UID: \"77adecb1-af53-4dd9-bb52-d8e22ebb03e4\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7"
Oct 30 19:07:45.649706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:45.644283    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77adecb1-af53-4dd9-bb52-d8e22ebb03e4-workload-certs\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7\" (UID: \"77adecb1-af53-4dd9-bb52-d8e22ebb03e4\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7"
Oct 30 19:07:45.649706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:45.644311    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77adecb1-af53-4dd9-bb52-d8e22ebb03e4-istio-data\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7\" (UID: \"77adecb1-af53-4dd9-bb52-d8e22ebb03e4\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7"
Oct 30 19:07:45.649706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:45.644354    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77adecb1-af53-4dd9-bb52-d8e22ebb03e4-workload-socket\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7\" (UID: \"77adecb1-af53-4dd9-bb52-d8e22ebb03e4\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7"
Oct 30 19:07:45.649706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:45.644509    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77adecb1-af53-4dd9-bb52-d8e22ebb03e4-istio-token\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7\" (UID: \"77adecb1-af53-4dd9-bb52-d8e22ebb03e4\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7"
Oct 30 19:07:45.649706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:45.644607    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77adecb1-af53-4dd9-bb52-d8e22ebb03e4-istio-podinfo\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7\" (UID: \"77adecb1-af53-4dd9-bb52-d8e22ebb03e4\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7"
Oct 30 19:07:45.649706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:45.644667    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77adecb1-af53-4dd9-bb52-d8e22ebb03e4-credential-socket\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7\" (UID: \"77adecb1-af53-4dd9-bb52-d8e22ebb03e4\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7"
Oct 30 19:07:45.726806 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:45.698543    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77adecb1-af53-4dd9-bb52-d8e22ebb03e4-workload-certs\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7\" (UID: \"77adecb1-af53-4dd9-bb52-d8e22ebb03e4\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7"
Oct 30 19:07:45.726806 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:45.699147    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77adecb1-af53-4dd9-bb52-d8e22ebb03e4-istiod-ca-cert\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7\" (UID: \"77adecb1-af53-4dd9-bb52-d8e22ebb03e4\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7"
Oct 30 19:07:45.742116 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:45.741992    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77adecb1-af53-4dd9-bb52-d8e22ebb03e4-workload-socket\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7\" (UID: \"77adecb1-af53-4dd9-bb52-d8e22ebb03e4\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7"
Oct 30 19:07:45.771744 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:45.771707    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77adecb1-af53-4dd9-bb52-d8e22ebb03e4-credential-socket\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7\" (UID: \"77adecb1-af53-4dd9-bb52-d8e22ebb03e4\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7"
Oct 30 19:07:45.829444 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:45.783139    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77adecb1-af53-4dd9-bb52-d8e22ebb03e4-istio-data\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7\" (UID: \"77adecb1-af53-4dd9-bb52-d8e22ebb03e4\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7"
Oct 30 19:07:45.924511 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:45.921223    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77adecb1-af53-4dd9-bb52-d8e22ebb03e4-istio-token\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7\" (UID: \"77adecb1-af53-4dd9-bb52-d8e22ebb03e4\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7"
Oct 30 19:07:45.946730 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:45.946355    4525 generic.go:356] "Generic (PLEG): container finished" podID="363cd59e-8ddd-43a6-91e9-f2c1b577948b" containerID="f92e360403e1fbcf91c9c7b5a9948880a6617e8b5d1571c517ec4df81624912a" exitCode=0
Oct 30 19:07:45.964011 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:45.950157    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf63472c-v2vbc" event={"ID":"363cd59e-8ddd-43a6-91e9-f2c1b577948b","Type":"ContainerDied","Data":"f92e360403e1fbcf91c9c7b5a9948880a6617e8b5d1571c517ec4df81624912a"}
Oct 30 19:07:45.964011 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:45.962342    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77adecb1-af53-4dd9-bb52-d8e22ebb03e4-istio-podinfo\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7\" (UID: \"77adecb1-af53-4dd9-bb52-d8e22ebb03e4\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7"
Oct 30 19:07:46.007950 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:46.007858    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77adecb1-af53-4dd9-bb52-d8e22ebb03e4-istio-envoy\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7\" (UID: \"77adecb1-af53-4dd9-bb52-d8e22ebb03e4\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7"
Oct 30 19:07:46.031731 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:46.011500    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzx5n\" (UniqueName: \"kubernetes.io/projected/77adecb1-af53-4dd9-bb52-d8e22ebb03e4-kube-api-access-nzx5n\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7\" (UID: \"77adecb1-af53-4dd9-bb52-d8e22ebb03e4\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7"
Oct 30 19:07:46.097747 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:46.079746    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7"
Oct 30 19:07:46.143734 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:46.140823    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-ca2cd5-openshift-default-59664bf6df-xqn5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body=
Oct 30 19:07:46.143734 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:46.140888    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ca2cd5-openshift-default-59664bf6df-xqn5s" podUID="84c44eb7-6483-4f83-ab59-447f04ba5d1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused"
Oct 30 19:07:46.410315 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:46.410280    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7"]
Oct 30 19:07:46.411302 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:07:46.411273    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77adecb1_af53_4dd9_bb52_d8e22ebb03e4.slice/crio-8463bda4408213f0eb97c7e2bb4af7faca48abf10a01e5b15707f925908e05b0 WatchSource:0}: Error finding container 8463bda4408213f0eb97c7e2bb4af7faca48abf10a01e5b15707f925908e05b0: Status 404 returned error can't find the container with id 8463bda4408213f0eb97c7e2bb4af7faca48abf10a01e5b15707f925908e05b0
Oct 30 19:07:46.415189 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:46.415135    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:07:46.415301 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:46.415215    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:07:46.415301 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:46.415248    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:07:46.985706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:46.977962    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7" event={"ID":"77adecb1-af53-4dd9-bb52-d8e22ebb03e4","Type":"ContainerStarted","Data":"052da6685f672c65b36233b5ccaf20546eb09e6184d1ec4d7da5fc1a377ff3ec"}
Oct 30 19:07:46.985706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:46.978020    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7" event={"ID":"77adecb1-af53-4dd9-bb52-d8e22ebb03e4","Type":"ContainerStarted","Data":"8463bda4408213f0eb97c7e2bb4af7faca48abf10a01e5b15707f925908e05b0"}
Oct 30 19:07:47.005755 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:47.002164    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7" podStartSLOduration=2.002145109 podStartE2EDuration="2.002145109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:07:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:07:46.998580901 +0000 UTC m=+1889.624250880" watchObservedRunningTime="2026-10-30 19:07:47.002145109 +0000 UTC m=+1889.627815110"
Oct 30 19:07:47.081171 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:47.081130    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7"
Oct 30 19:07:47.082609 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:47.082577    4525 patch_prober.go:29] interesting pod/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" start-of-body=
Oct 30 19:07:47.082776 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:47.082635    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7" podUID="77adecb1-af53-4dd9-bb52-d8e22ebb03e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused"
Oct 30 19:07:47.226097 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:47.226050    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-ca2cd5-openshift-default-59664bf6df-xqn5s"
Oct 30 19:07:47.228310 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:47.228284    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-ca2cd5-openshift-default-59664bf6df-xqn5s"
Oct 30 19:07:47.230696 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:47.230656    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-ca2cd5-openshift-default-59664bf6df-xqn5s"
Oct 30 19:07:47.268907 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:47.268879    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf63472c-v2vbc"
Oct 30 19:07:47.289383 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:47.289345    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/363cd59e-8ddd-43a6-91e9-f2c1b577948b-kube-api-access-rhqr6\" (UniqueName: \"kubernetes.io/projected/363cd59e-8ddd-43a6-91e9-f2c1b577948b-kube-api-access-rhqr6\") pod \"363cd59e-8ddd-43a6-91e9-f2c1b577948b\" (UID: \"363cd59e-8ddd-43a6-91e9-f2c1b577948b\") "
Oct 30 19:07:47.289529 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:47.289460    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/363cd59e-8ddd-43a6-91e9-f2c1b577948b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/363cd59e-8ddd-43a6-91e9-f2c1b577948b-maas-api-xks-ca\") pod \"363cd59e-8ddd-43a6-91e9-f2c1b577948b\" (UID: \"363cd59e-8ddd-43a6-91e9-f2c1b577948b\") "
Oct 30 19:07:47.289651 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:47.289626    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/363cd59e-8ddd-43a6-91e9-f2c1b577948b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/363cd59e-8ddd-43a6-91e9-f2c1b577948b-maas-api-openshift-ca\") pod \"363cd59e-8ddd-43a6-91e9-f2c1b577948b\" (UID: \"363cd59e-8ddd-43a6-91e9-f2c1b577948b\") "
Oct 30 19:07:47.289959 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:47.289936    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/363cd59e-8ddd-43a6-91e9-f2c1b577948b-maas-api-openshift-ca" pod "363cd59e-8ddd-43a6-91e9-f2c1b577948b" (UID: "363cd59e-8ddd-43a6-91e9-f2c1b577948b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:07:47.290460 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:47.290421    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/363cd59e-8ddd-43a6-91e9-f2c1b577948b-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:47.291796 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:47.291778    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/363cd59e-8ddd-43a6-91e9-f2c1b577948b-maas-api-xks-ca" pod "363cd59e-8ddd-43a6-91e9-f2c1b577948b" (UID: "363cd59e-8ddd-43a6-91e9-f2c1b577948b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:07:47.292117 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:47.292100    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/363cd59e-8ddd-43a6-91e9-f2c1b577948b-kube-api-access-rhqr6" pod "363cd59e-8ddd-43a6-91e9-f2c1b577948b" (UID: "363cd59e-8ddd-43a6-91e9-f2c1b577948b"). InnerVolumeSpecName "kube-api-access-rhqr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:07:47.391791 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:47.391761    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhqr6\" (UniqueName: \"kubernetes.io/projected/363cd59e-8ddd-43a6-91e9-f2c1b577948b-kube-api-access-rhqr6\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:47.391791 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:47.391789    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/363cd59e-8ddd-43a6-91e9-f2c1b577948b-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:47.402457 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:07:47.402425    4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d49l" containerName="extractor"
Oct 30 19:07:48.037748 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:48.036610    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf63472c-v2vbc"
Oct 30 19:07:48.037748 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:48.037083    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf63472c-v2vbc" event={"ID":"363cd59e-8ddd-43a6-91e9-f2c1b577948b","Type":"ContainerDied","Data":"2420b37fcc58ffccfd87021188b0c6d04adba3c529742a47031dc7b12761e902"}
Oct 30 19:07:48.037748 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:48.037176    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2420b37fcc58ffccfd87021188b0c6d04adba3c529742a47031dc7b12761e902"
Oct 30 19:07:48.086276 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:48.086246    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7"
Oct 30 19:07:49.046129 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:49.046084    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7"
Oct 30 19:07:49.046889 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:49.046869    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7"
Oct 30 19:07:54.657468 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:54.657415    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-baa504/auth-test-model-baa504-kserve-64c89767d5-d4pq4"]
Oct 30 19:07:56.899100 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:56.899063    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-68b466/auth-test-model-68b466-kserve-6dd5b87d8f-ddr8r"]
Oct 30 19:07:57.935703 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:57.935647    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4dd4v"]
Oct 30 19:07:57.936722 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:57.936695    4525 cpu_manager.go:414] "containerMap: removing container" podUID="363cd59e-8ddd-43a6-91e9-f2c1b577948b" containerName="revoke-keys"
Oct 30 19:07:57.936722 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:57.936722    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="363cd59e-8ddd-43a6-91e9-f2c1b577948b" containerName="revoke-keys"
Oct 30 19:07:57.936990 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:57.936974    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="363cd59e-8ddd-43a6-91e9-f2c1b577948b" containerName="revoke-keys"
Oct 30 19:07:57.942459 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:57.942435    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4dd4v"
Oct 30 19:07:57.946787 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:57.946763    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-c164d6-openshift-default-dockercfg-c42sr\""
Oct 30 19:07:57.949758 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:57.949733    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4dd4v"]
Oct 30 19:07:57.995007 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:57.994961    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/feea5f3c-8630-4e8f-8832-2998030bf2e1-credential-socket\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-4dd4v\" (UID: \"feea5f3c-8630-4e8f-8832-2998030bf2e1\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4dd4v"
Oct 30 19:07:57.995216 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:57.995020    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/feea5f3c-8630-4e8f-8832-2998030bf2e1-istio-data\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-4dd4v\" (UID: \"feea5f3c-8630-4e8f-8832-2998030bf2e1\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4dd4v"
Oct 30 19:07:57.995294 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:57.995211    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/feea5f3c-8630-4e8f-8832-2998030bf2e1-workload-certs\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-4dd4v\" (UID: \"feea5f3c-8630-4e8f-8832-2998030bf2e1\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4dd4v"
Oct 30 19:07:57.995354 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:57.995328    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/feea5f3c-8630-4e8f-8832-2998030bf2e1-istio-token\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-4dd4v\" (UID: \"feea5f3c-8630-4e8f-8832-2998030bf2e1\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4dd4v"
Oct 30 19:07:57.995418 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:57.995382    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/feea5f3c-8630-4e8f-8832-2998030bf2e1-istio-podinfo\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-4dd4v\" (UID: \"feea5f3c-8630-4e8f-8832-2998030bf2e1\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4dd4v"
Oct 30 19:07:57.995495 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:57.995472    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/feea5f3c-8630-4e8f-8832-2998030bf2e1-istio-envoy\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-4dd4v\" (UID: \"feea5f3c-8630-4e8f-8832-2998030bf2e1\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4dd4v"
Oct 30 19:07:57.995623 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:57.995609    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/feea5f3c-8630-4e8f-8832-2998030bf2e1-workload-socket\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-4dd4v\" (UID: \"feea5f3c-8630-4e8f-8832-2998030bf2e1\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4dd4v"
Oct 30 19:07:57.995728 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:57.995640    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vm9r\" (UniqueName: \"kubernetes.io/projected/feea5f3c-8630-4e8f-8832-2998030bf2e1-kube-api-access-7vm9r\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-4dd4v\" (UID: \"feea5f3c-8630-4e8f-8832-2998030bf2e1\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4dd4v"
Oct 30 19:07:57.995807 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:57.995752    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/feea5f3c-8630-4e8f-8832-2998030bf2e1-istiod-ca-cert\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-4dd4v\" (UID: \"feea5f3c-8630-4e8f-8832-2998030bf2e1\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4dd4v"
Oct 30 19:07:58.038041 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:58.038004    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9b222d4-cshq9"]
Oct 30 19:07:58.042839 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:58.042815    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9b222d4-cshq9"]
Oct 30 19:07:58.096959 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:58.096925    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/feea5f3c-8630-4e8f-8832-2998030bf2e1-istio-token\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-4dd4v\" (UID: \"feea5f3c-8630-4e8f-8832-2998030bf2e1\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4dd4v"
Oct 30 19:07:58.097125 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:58.096994    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/feea5f3c-8630-4e8f-8832-2998030bf2e1-istio-podinfo\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-4dd4v\" (UID: \"feea5f3c-8630-4e8f-8832-2998030bf2e1\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4dd4v"
Oct 30 19:07:58.097125 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:58.097046    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/feea5f3c-8630-4e8f-8832-2998030bf2e1-istio-envoy\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-4dd4v\" (UID: \"feea5f3c-8630-4e8f-8832-2998030bf2e1\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4dd4v"
Oct 30 19:07:58.097215 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:58.097168    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/feea5f3c-8630-4e8f-8832-2998030bf2e1-workload-socket\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-4dd4v\" (UID: \"feea5f3c-8630-4e8f-8832-2998030bf2e1\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4dd4v"
Oct 30 19:07:58.097215 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:58.097197    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vm9r\" (UniqueName: \"kubernetes.io/projected/feea5f3c-8630-4e8f-8832-2998030bf2e1-kube-api-access-7vm9r\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-4dd4v\" (UID: \"feea5f3c-8630-4e8f-8832-2998030bf2e1\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4dd4v"
Oct 30 19:07:58.097378 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:58.097239    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/feea5f3c-8630-4e8f-8832-2998030bf2e1-istiod-ca-cert\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-4dd4v\" (UID: \"feea5f3c-8630-4e8f-8832-2998030bf2e1\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4dd4v"
Oct 30 19:07:58.097378 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:58.097276    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/feea5f3c-8630-4e8f-8832-2998030bf2e1-credential-socket\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-4dd4v\" (UID: \"feea5f3c-8630-4e8f-8832-2998030bf2e1\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4dd4v"
Oct 30 19:07:58.097562 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:58.097544    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/feea5f3c-8630-4e8f-8832-2998030bf2e1-istio-data\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-4dd4v\" (UID: \"feea5f3c-8630-4e8f-8832-2998030bf2e1\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4dd4v"
Oct 30 19:07:58.097649 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:58.097563    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/feea5f3c-8630-4e8f-8832-2998030bf2e1-workload-socket\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-4dd4v\" (UID: \"feea5f3c-8630-4e8f-8832-2998030bf2e1\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4dd4v"
Oct 30 19:07:58.097764 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:58.097663    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/feea5f3c-8630-4e8f-8832-2998030bf2e1-credential-socket\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-4dd4v\" (UID: \"feea5f3c-8630-4e8f-8832-2998030bf2e1\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4dd4v"
Oct 30 19:07:58.097764 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:58.097704    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/feea5f3c-8630-4e8f-8832-2998030bf2e1-workload-certs\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-4dd4v\" (UID: \"feea5f3c-8630-4e8f-8832-2998030bf2e1\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4dd4v"
Oct 30 19:07:58.097894 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:58.097832    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/feea5f3c-8630-4e8f-8832-2998030bf2e1-istio-data\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-4dd4v\" (UID: \"feea5f3c-8630-4e8f-8832-2998030bf2e1\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4dd4v"
Oct 30 19:07:58.097985 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:58.097959    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/feea5f3c-8630-4e8f-8832-2998030bf2e1-workload-certs\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-4dd4v\" (UID: \"feea5f3c-8630-4e8f-8832-2998030bf2e1\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4dd4v"
Oct 30 19:07:58.098193 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:58.098168    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/feea5f3c-8630-4e8f-8832-2998030bf2e1-istiod-ca-cert\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-4dd4v\" (UID: \"feea5f3c-8630-4e8f-8832-2998030bf2e1\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4dd4v"
Oct 30 19:07:58.099691 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:58.099656    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/feea5f3c-8630-4e8f-8832-2998030bf2e1-istio-envoy\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-4dd4v\" (UID: \"feea5f3c-8630-4e8f-8832-2998030bf2e1\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4dd4v"
Oct 30 19:07:58.099764 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:58.099730    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/feea5f3c-8630-4e8f-8832-2998030bf2e1-istio-podinfo\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-4dd4v\" (UID: \"feea5f3c-8630-4e8f-8832-2998030bf2e1\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4dd4v"
Oct 30 19:07:58.104984 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:58.104960    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/feea5f3c-8630-4e8f-8832-2998030bf2e1-istio-token\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-4dd4v\" (UID: \"feea5f3c-8630-4e8f-8832-2998030bf2e1\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4dd4v"
Oct 30 19:07:58.105075 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:58.105007    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vm9r\" (UniqueName: \"kubernetes.io/projected/feea5f3c-8630-4e8f-8832-2998030bf2e1-kube-api-access-7vm9r\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-4dd4v\" (UID: \"feea5f3c-8630-4e8f-8832-2998030bf2e1\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4dd4v"
Oct 30 19:07:58.277034 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:58.262020    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4dd4v"
Oct 30 19:07:58.789245 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:58.789211    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-7njf5"]
Oct 30 19:07:58.789437 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:58.789409    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-7njf5" podUID="0d90f627-6806-4883-823a-1bb50c0afcf6" containerName="istio-proxy" containerID="cri-o://3e0aac395c4b21cc200a5b5506f7ba373b8df4734ba1676f18f6df7daf201d49" gracePeriod=30
Oct 30 19:07:58.843376 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:07:58.843347    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfeea5f3c_8630_4e8f_8832_2998030bf2e1.slice/crio-b37993812cb1e45807a5ec4a43fb1c1776a5cb0f71d9996273cce1b4a61da62d WatchSource:0}: Error finding container b37993812cb1e45807a5ec4a43fb1c1776a5cb0f71d9996273cce1b4a61da62d: Status 404 returned error can't find the container with id b37993812cb1e45807a5ec4a43fb1c1776a5cb0f71d9996273cce1b4a61da62d
Oct 30 19:07:58.845068 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:58.845042    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4dd4v"]
Oct 30 19:07:58.846376 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:58.846342    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:07:58.846489 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:58.846419    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:07:58.846489 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:58.846463    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:07:59.441833 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:59.441797    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4dd4v" event={"ID":"feea5f3c-8630-4e8f-8832-2998030bf2e1","Type":"ContainerStarted","Data":"5355dfafbeb18323c6c9b14ebabc1c273fab1826acfb5c40e6382e56248a21da"}
Oct 30 19:07:59.442286 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:59.441857    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4dd4v" event={"ID":"feea5f3c-8630-4e8f-8832-2998030bf2e1","Type":"ContainerStarted","Data":"b37993812cb1e45807a5ec4a43fb1c1776a5cb0f71d9996273cce1b4a61da62d"}
Oct 30 19:07:59.465247 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:59.465194    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4dd4v" podStartSLOduration=2.465179301 podStartE2EDuration="2.465179301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:07:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:07:59.4608557 +0000 UTC m=+1902.086525664" watchObservedRunningTime="2026-10-30 19:07:59.465179301 +0000 UTC m=+1902.090849263"
Oct 30 19:07:59.546879 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:07:59.546840    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aaf6f1db-cc65-4e6b-ac33-b4848b527f8a" path="/var/lib/kubelet/pods/aaf6f1db-cc65-4e6b-ac33-b4848b527f8a/volumes"
Oct 30 19:08:00.267234 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:00.267201    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4dd4v"
Oct 30 19:08:00.272961 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:00.272939    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4dd4v"
Oct 30 19:08:00.450314 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:00.450276    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4dd4v"
Oct 30 19:08:00.451222 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:00.451203    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4dd4v"
Oct 30 19:08:04.950281 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:04.950253    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-7njf5"
Oct 30 19:08:05.140406 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:05.140345    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d90f627-6806-4883-823a-1bb50c0afcf6-istio-token\" (UniqueName: \"kubernetes.io/projected/0d90f627-6806-4883-823a-1bb50c0afcf6-istio-token\") pod \"0d90f627-6806-4883-823a-1bb50c0afcf6\" (UID: \"0d90f627-6806-4883-823a-1bb50c0afcf6\") "
Oct 30 19:08:05.141217 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:05.141189    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d90f627-6806-4883-823a-1bb50c0afcf6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d90f627-6806-4883-823a-1bb50c0afcf6-istio-envoy\") pod \"0d90f627-6806-4883-823a-1bb50c0afcf6\" (UID: \"0d90f627-6806-4883-823a-1bb50c0afcf6\") "
Oct 30 19:08:05.146272 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:05.146248    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0d90f627-6806-4883-823a-1bb50c0afcf6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d90f627-6806-4883-823a-1bb50c0afcf6-istio-podinfo\") pod \"0d90f627-6806-4883-823a-1bb50c0afcf6\" (UID: \"0d90f627-6806-4883-823a-1bb50c0afcf6\") "
Oct 30 19:08:05.146613 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:05.146590    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d90f627-6806-4883-823a-1bb50c0afcf6-kube-api-access-9n26x\" (UniqueName: \"kubernetes.io/projected/0d90f627-6806-4883-823a-1bb50c0afcf6-kube-api-access-9n26x\") pod \"0d90f627-6806-4883-823a-1bb50c0afcf6\" (UID: \"0d90f627-6806-4883-823a-1bb50c0afcf6\") "
Oct 30 19:08:05.146956 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:05.146936    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d90f627-6806-4883-823a-1bb50c0afcf6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d90f627-6806-4883-823a-1bb50c0afcf6-workload-socket\") pod \"0d90f627-6806-4883-823a-1bb50c0afcf6\" (UID: \"0d90f627-6806-4883-823a-1bb50c0afcf6\") "
Oct 30 19:08:05.147090 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:05.147074    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d90f627-6806-4883-823a-1bb50c0afcf6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d90f627-6806-4883-823a-1bb50c0afcf6-istio-data\") pod \"0d90f627-6806-4883-823a-1bb50c0afcf6\" (UID: \"0d90f627-6806-4883-823a-1bb50c0afcf6\") "
Oct 30 19:08:05.147224 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:05.147208    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d90f627-6806-4883-823a-1bb50c0afcf6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d90f627-6806-4883-823a-1bb50c0afcf6-credential-socket\") pod \"0d90f627-6806-4883-823a-1bb50c0afcf6\" (UID: \"0d90f627-6806-4883-823a-1bb50c0afcf6\") "
Oct 30 19:08:05.147353 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:05.147340    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d90f627-6806-4883-823a-1bb50c0afcf6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d90f627-6806-4883-823a-1bb50c0afcf6-workload-certs\") pod \"0d90f627-6806-4883-823a-1bb50c0afcf6\" (UID: \"0d90f627-6806-4883-823a-1bb50c0afcf6\") "
Oct 30 19:08:05.147462 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:05.147449    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d90f627-6806-4883-823a-1bb50c0afcf6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d90f627-6806-4883-823a-1bb50c0afcf6-istiod-ca-cert\") pod \"0d90f627-6806-4883-823a-1bb50c0afcf6\" (UID: \"0d90f627-6806-4883-823a-1bb50c0afcf6\") "
Oct 30 19:08:05.148806 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:05.148784    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d90f627-6806-4883-823a-1bb50c0afcf6-istiod-ca-cert" pod "0d90f627-6806-4883-823a-1bb50c0afcf6" (UID: "0d90f627-6806-4883-823a-1bb50c0afcf6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:08:05.156644 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:05.156440    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d90f627-6806-4883-823a-1bb50c0afcf6-istio-data" pod "0d90f627-6806-4883-823a-1bb50c0afcf6" (UID: "0d90f627-6806-4883-823a-1bb50c0afcf6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:08:05.156644 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:05.156454    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d90f627-6806-4883-823a-1bb50c0afcf6-credential-socket" pod "0d90f627-6806-4883-823a-1bb50c0afcf6" (UID: "0d90f627-6806-4883-823a-1bb50c0afcf6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:08:05.156839 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:05.156671    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d90f627-6806-4883-823a-1bb50c0afcf6-workload-certs" pod "0d90f627-6806-4883-823a-1bb50c0afcf6" (UID: "0d90f627-6806-4883-823a-1bb50c0afcf6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:08:05.156903 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:05.156884    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d90f627-6806-4883-823a-1bb50c0afcf6-workload-socket" pod "0d90f627-6806-4883-823a-1bb50c0afcf6" (UID: "0d90f627-6806-4883-823a-1bb50c0afcf6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:08:05.161390 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:05.161319    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d90f627-6806-4883-823a-1bb50c0afcf6-kube-api-access-9n26x" pod "0d90f627-6806-4883-823a-1bb50c0afcf6" (UID: "0d90f627-6806-4883-823a-1bb50c0afcf6"). InnerVolumeSpecName "kube-api-access-9n26x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:08:05.165932 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:05.165847    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0d90f627-6806-4883-823a-1bb50c0afcf6-istio-podinfo" pod "0d90f627-6806-4883-823a-1bb50c0afcf6" (UID: "0d90f627-6806-4883-823a-1bb50c0afcf6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:08:05.171143 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:05.166717    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d90f627-6806-4883-823a-1bb50c0afcf6-istio-token" pod "0d90f627-6806-4883-823a-1bb50c0afcf6" (UID: "0d90f627-6806-4883-823a-1bb50c0afcf6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:08:05.171143 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:05.166824    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d90f627-6806-4883-823a-1bb50c0afcf6-istio-envoy" pod "0d90f627-6806-4883-823a-1bb50c0afcf6" (UID: "0d90f627-6806-4883-823a-1bb50c0afcf6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:08:05.171995 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:05.171969    4525 generic.go:356] "Generic (PLEG): container finished" podID="0d90f627-6806-4883-823a-1bb50c0afcf6" containerID="3e0aac395c4b21cc200a5b5506f7ba373b8df4734ba1676f18f6df7daf201d49" exitCode=0
Oct 30 19:08:05.172103 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:05.172088    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-7njf5" event={"ID":"0d90f627-6806-4883-823a-1bb50c0afcf6","Type":"ContainerDied","Data":"3e0aac395c4b21cc200a5b5506f7ba373b8df4734ba1676f18f6df7daf201d49"}
Oct 30 19:08:05.172174 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:05.172137    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-7njf5" event={"ID":"0d90f627-6806-4883-823a-1bb50c0afcf6","Type":"ContainerDied","Data":"17321660de501aba10aec54d2d70ca83bebd1214243df70b2af8b44839c435bb"}
Oct 30 19:08:05.172174 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:05.172164    4525 scope.go:122] "RemoveContainer" containerID="3e0aac395c4b21cc200a5b5506f7ba373b8df4734ba1676f18f6df7daf201d49"
Oct 30 19:08:05.172337 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:05.172325    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-7njf5"
Oct 30 19:08:05.255708 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:05.255510    4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d90f627-6806-4883-823a-1bb50c0afcf6-istio-podinfo\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:08:05.255708 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:05.255574    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9n26x\" (UniqueName: \"kubernetes.io/projected/0d90f627-6806-4883-823a-1bb50c0afcf6-kube-api-access-9n26x\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:08:05.255708 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:05.255596    4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d90f627-6806-4883-823a-1bb50c0afcf6-workload-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:08:05.255708 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:05.255610    4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d90f627-6806-4883-823a-1bb50c0afcf6-istio-data\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:08:05.255708 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:05.255625    4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d90f627-6806-4883-823a-1bb50c0afcf6-credential-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:08:05.255708 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:05.255652    4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d90f627-6806-4883-823a-1bb50c0afcf6-workload-certs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:08:05.255708 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:05.255666    4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d90f627-6806-4883-823a-1bb50c0afcf6-istiod-ca-cert\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:08:05.255708 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:05.255707    4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d90f627-6806-4883-823a-1bb50c0afcf6-istio-token\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:08:05.255708 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:05.255722    4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d90f627-6806-4883-823a-1bb50c0afcf6-istio-envoy\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:08:05.258782 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:05.258761    4525 scope.go:122] "RemoveContainer" containerID="3e0aac395c4b21cc200a5b5506f7ba373b8df4734ba1676f18f6df7daf201d49"
Oct 30 19:08:05.259318 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:08:05.259295    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e0aac395c4b21cc200a5b5506f7ba373b8df4734ba1676f18f6df7daf201d49\": container with ID starting with 3e0aac395c4b21cc200a5b5506f7ba373b8df4734ba1676f18f6df7daf201d49 not found: ID does not exist" containerID="3e0aac395c4b21cc200a5b5506f7ba373b8df4734ba1676f18f6df7daf201d49"
Oct 30 19:08:05.259396 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:05.259328    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e0aac395c4b21cc200a5b5506f7ba373b8df4734ba1676f18f6df7daf201d49"} err="failed to get container status \"3e0aac395c4b21cc200a5b5506f7ba373b8df4734ba1676f18f6df7daf201d49\": rpc error: code = NotFound desc = could not find container \"3e0aac395c4b21cc200a5b5506f7ba373b8df4734ba1676f18f6df7daf201d49\": container with ID starting with 3e0aac395c4b21cc200a5b5506f7ba373b8df4734ba1676f18f6df7daf201d49 not found: ID does not exist"
Oct 30 19:08:05.289321 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:05.288559    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-7njf5"]
Oct 30 19:08:05.318651 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:08:05.318606    4525 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d90f627_6806_4883_823a_1bb50c0afcf6.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d90f627_6806_4883_823a_1bb50c0afcf6.slice/crio-17321660de501aba10aec54d2d70ca83bebd1214243df70b2af8b44839c435bb\": RecentStats: unable to find data in memory cache]"
Oct 30 19:08:05.336966 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:05.336927    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-7njf5"]
Oct 30 19:08:05.562720 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:05.561317    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d90f627-6806-4883-823a-1bb50c0afcf6" path="/var/lib/kubelet/pods/0d90f627-6806-4883-823a-1bb50c0afcf6/volumes"
Oct 30 19:08:09.125632 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:09.125598    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wlrxd"]
Oct 30 19:08:09.126370 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:09.126352    4525 cpu_manager.go:414] "containerMap: removing container" podUID="0d90f627-6806-4883-823a-1bb50c0afcf6" containerName="istio-proxy"
Oct 30 19:08:09.126432 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:09.126374    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d90f627-6806-4883-823a-1bb50c0afcf6" containerName="istio-proxy"
Oct 30 19:08:09.126498 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:09.126478    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d90f627-6806-4883-823a-1bb50c0afcf6" containerName="istio-proxy"
Oct 30 19:08:09.132778 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:09.132754    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wlrxd"
Oct 30 19:08:09.160773 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:09.160745    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wlrxd"]
Oct 30 19:08:09.204969 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:09.204227    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa550fe6-5b53-439d-9c72-391ed15e1fb3-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wlrxd\" (UID: \"aa550fe6-5b53-439d-9c72-391ed15e1fb3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wlrxd"
Oct 30 19:08:09.204969 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:09.204287    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/aa550fe6-5b53-439d-9c72-391ed15e1fb3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wlrxd\" (UID: \"aa550fe6-5b53-439d-9c72-391ed15e1fb3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wlrxd"
Oct 30 19:08:09.204969 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:09.204335    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/aa550fe6-5b53-439d-9c72-391ed15e1fb3-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wlrxd\" (UID: \"aa550fe6-5b53-439d-9c72-391ed15e1fb3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wlrxd"
Oct 30 19:08:09.204969 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:09.204471    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa550fe6-5b53-439d-9c72-391ed15e1fb3-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wlrxd\" (UID: \"aa550fe6-5b53-439d-9c72-391ed15e1fb3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wlrxd"
Oct 30 19:08:09.204969 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:09.204561    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/aa550fe6-5b53-439d-9c72-391ed15e1fb3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wlrxd\" (UID: \"aa550fe6-5b53-439d-9c72-391ed15e1fb3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wlrxd"
Oct 30 19:08:09.204969 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:09.204617    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9qts\" (UniqueName: \"kubernetes.io/projected/aa550fe6-5b53-439d-9c72-391ed15e1fb3-kube-api-access-c9qts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wlrxd\" (UID: \"aa550fe6-5b53-439d-9c72-391ed15e1fb3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wlrxd"
Oct 30 19:08:09.204969 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:09.204745    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/aa550fe6-5b53-439d-9c72-391ed15e1fb3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wlrxd\" (UID: \"aa550fe6-5b53-439d-9c72-391ed15e1fb3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wlrxd"
Oct 30 19:08:09.305852 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:09.305767    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/aa550fe6-5b53-439d-9c72-391ed15e1fb3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wlrxd\" (UID: \"aa550fe6-5b53-439d-9c72-391ed15e1fb3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wlrxd"
Oct 30 19:08:09.305852 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:09.305833    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/aa550fe6-5b53-439d-9c72-391ed15e1fb3-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wlrxd\" (UID: \"aa550fe6-5b53-439d-9c72-391ed15e1fb3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wlrxd"
Oct 30 19:08:09.306110 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:09.306002    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa550fe6-5b53-439d-9c72-391ed15e1fb3-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wlrxd\" (UID: \"aa550fe6-5b53-439d-9c72-391ed15e1fb3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wlrxd"
Oct 30 19:08:09.306181 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:09.306136    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/aa550fe6-5b53-439d-9c72-391ed15e1fb3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wlrxd\" (UID: \"aa550fe6-5b53-439d-9c72-391ed15e1fb3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wlrxd"
Oct 30 19:08:09.306295 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:09.306254    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9qts\" (UniqueName: \"kubernetes.io/projected/aa550fe6-5b53-439d-9c72-391ed15e1fb3-kube-api-access-c9qts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wlrxd\" (UID: \"aa550fe6-5b53-439d-9c72-391ed15e1fb3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wlrxd"
Oct 30 19:08:09.306363 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:09.306353    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/aa550fe6-5b53-439d-9c72-391ed15e1fb3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wlrxd\" (UID: \"aa550fe6-5b53-439d-9c72-391ed15e1fb3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wlrxd"
Oct 30 19:08:09.306428 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:09.306409    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa550fe6-5b53-439d-9c72-391ed15e1fb3-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wlrxd\" (UID: \"aa550fe6-5b53-439d-9c72-391ed15e1fb3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wlrxd"
Oct 30 19:08:09.307358 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:09.307321    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/aa550fe6-5b53-439d-9c72-391ed15e1fb3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wlrxd\" (UID: \"aa550fe6-5b53-439d-9c72-391ed15e1fb3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wlrxd"
Oct 30 19:08:09.308502 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:09.308472    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa550fe6-5b53-439d-9c72-391ed15e1fb3-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wlrxd\" (UID: \"aa550fe6-5b53-439d-9c72-391ed15e1fb3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wlrxd"
Oct 30 19:08:09.308871 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:09.308852    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/aa550fe6-5b53-439d-9c72-391ed15e1fb3-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wlrxd\" (UID: \"aa550fe6-5b53-439d-9c72-391ed15e1fb3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wlrxd"
Oct 30 19:08:09.309168 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:09.309149    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/aa550fe6-5b53-439d-9c72-391ed15e1fb3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wlrxd\" (UID: \"aa550fe6-5b53-439d-9c72-391ed15e1fb3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wlrxd"
Oct 30 19:08:09.309602 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:09.309581    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/aa550fe6-5b53-439d-9c72-391ed15e1fb3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wlrxd\" (UID: \"aa550fe6-5b53-439d-9c72-391ed15e1fb3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wlrxd"
Oct 30 19:08:09.314671 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:09.314649    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9qts\" (UniqueName: \"kubernetes.io/projected/aa550fe6-5b53-439d-9c72-391ed15e1fb3-kube-api-access-c9qts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wlrxd\" (UID: \"aa550fe6-5b53-439d-9c72-391ed15e1fb3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wlrxd"
Oct 30 19:08:09.315405 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:09.315385    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa550fe6-5b53-439d-9c72-391ed15e1fb3-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wlrxd\" (UID: \"aa550fe6-5b53-439d-9c72-391ed15e1fb3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wlrxd"
Oct 30 19:08:09.474021 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:09.473713    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wlrxd"
Oct 30 19:08:09.843763 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:09.794948    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-baa504/auth-test-model-baa504-kserve-64c89767d5-d4pq4" podUID="2300b922-2ef9-4c20-8c7d-df89a055374c" containerName="main" containerID="cri-o://a30a587db8cb8888869d9865080e274038daa33ec39a4edfc514961887edf430" gracePeriod=45
Oct 30 19:08:10.118717 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:10.118543    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:08:10.118717 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:10.118637    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:08:10.142405 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:10.142039    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wlrxd"]
Oct 30 19:08:10.465017 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:10.429427    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wlrxd" event={"ID":"aa550fe6-5b53-439d-9c72-391ed15e1fb3","Type":"ContainerStarted","Data":"7e4a90f3ba5043c9a268580bba5075b51f4f710fbf8f5a8f0fb7723b83ab340b"}
Oct 30 19:08:10.465017 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:10.449870    4525 generic.go:356] "Generic (PLEG): container finished" podID="2300b922-2ef9-4c20-8c7d-df89a055374c" containerID="a30a587db8cb8888869d9865080e274038daa33ec39a4edfc514961887edf430" exitCode=0
Oct 30 19:08:10.465017 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:10.449986    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-baa504/auth-test-model-baa504-kserve-64c89767d5-d4pq4" event={"ID":"2300b922-2ef9-4c20-8c7d-df89a055374c","Type":"ContainerDied","Data":"a30a587db8cb8888869d9865080e274038daa33ec39a4edfc514961887edf430"}
Oct 30 19:08:11.007772 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:11.007402    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-baa504/auth-test-model-baa504-kserve-64c89767d5-d4pq4"
Oct 30 19:08:11.100761 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:11.099845    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2300b922-2ef9-4c20-8c7d-df89a055374c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2300b922-2ef9-4c20-8c7d-df89a055374c-tmp-dir\") pod \"2300b922-2ef9-4c20-8c7d-df89a055374c\" (UID: \"2300b922-2ef9-4c20-8c7d-df89a055374c\") "
Oct 30 19:08:11.100761 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:11.100028    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2300b922-2ef9-4c20-8c7d-df89a055374c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2300b922-2ef9-4c20-8c7d-df89a055374c-dshm\") pod \"2300b922-2ef9-4c20-8c7d-df89a055374c\" (UID: \"2300b922-2ef9-4c20-8c7d-df89a055374c\") "
Oct 30 19:08:11.100761 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:11.100123    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2300b922-2ef9-4c20-8c7d-df89a055374c-kube-api-access-8zcvf\" (UniqueName: \"kubernetes.io/projected/2300b922-2ef9-4c20-8c7d-df89a055374c-kube-api-access-8zcvf\") pod \"2300b922-2ef9-4c20-8c7d-df89a055374c\" (UID: \"2300b922-2ef9-4c20-8c7d-df89a055374c\") "
Oct 30 19:08:11.100761 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:11.100155    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2300b922-2ef9-4c20-8c7d-df89a055374c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2300b922-2ef9-4c20-8c7d-df89a055374c-model-cache\") pod \"2300b922-2ef9-4c20-8c7d-df89a055374c\" (UID: \"2300b922-2ef9-4c20-8c7d-df89a055374c\") "
Oct 30 19:08:11.100761 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:11.100333    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2300b922-2ef9-4c20-8c7d-df89a055374c-tls-certs\" (UniqueName: \"kubernetes.io/secret/2300b922-2ef9-4c20-8c7d-df89a055374c-tls-certs\") pod \"2300b922-2ef9-4c20-8c7d-df89a055374c\" (UID: \"2300b922-2ef9-4c20-8c7d-df89a055374c\") "
Oct 30 19:08:11.100761 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:11.100404    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2300b922-2ef9-4c20-8c7d-df89a055374c-home\" (UniqueName: \"kubernetes.io/empty-dir/2300b922-2ef9-4c20-8c7d-df89a055374c-home\") pod \"2300b922-2ef9-4c20-8c7d-df89a055374c\" (UID: \"2300b922-2ef9-4c20-8c7d-df89a055374c\") "
Oct 30 19:08:11.120702 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:11.120466    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2300b922-2ef9-4c20-8c7d-df89a055374c-model-cache" pod "2300b922-2ef9-4c20-8c7d-df89a055374c" (UID: "2300b922-2ef9-4c20-8c7d-df89a055374c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:08:11.120702 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:11.120655    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2300b922-2ef9-4c20-8c7d-df89a055374c-home" pod "2300b922-2ef9-4c20-8c7d-df89a055374c" (UID: "2300b922-2ef9-4c20-8c7d-df89a055374c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:08:11.122386 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:11.121178    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2300b922-2ef9-4c20-8c7d-df89a055374c-tmp-dir" pod "2300b922-2ef9-4c20-8c7d-df89a055374c" (UID: "2300b922-2ef9-4c20-8c7d-df89a055374c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:08:11.129408 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:11.129359    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2300b922-2ef9-4c20-8c7d-df89a055374c-kube-api-access-8zcvf" pod "2300b922-2ef9-4c20-8c7d-df89a055374c" (UID: "2300b922-2ef9-4c20-8c7d-df89a055374c"). InnerVolumeSpecName "kube-api-access-8zcvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:08:11.131867 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:11.131561    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2300b922-2ef9-4c20-8c7d-df89a055374c-dshm" pod "2300b922-2ef9-4c20-8c7d-df89a055374c" (UID: "2300b922-2ef9-4c20-8c7d-df89a055374c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:08:11.162666 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:11.159058    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2300b922-2ef9-4c20-8c7d-df89a055374c-tls-certs" pod "2300b922-2ef9-4c20-8c7d-df89a055374c" (UID: "2300b922-2ef9-4c20-8c7d-df89a055374c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:08:11.255144 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:11.253755    4525 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2300b922-2ef9-4c20-8c7d-df89a055374c-tmp-dir\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:08:11.255144 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:11.253817    4525 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2300b922-2ef9-4c20-8c7d-df89a055374c-dshm\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:08:11.255144 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:11.253833    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zcvf\" (UniqueName: \"kubernetes.io/projected/2300b922-2ef9-4c20-8c7d-df89a055374c-kube-api-access-8zcvf\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:08:11.255144 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:11.253849    4525 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2300b922-2ef9-4c20-8c7d-df89a055374c-model-cache\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:08:11.255144 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:11.253862    4525 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2300b922-2ef9-4c20-8c7d-df89a055374c-tls-certs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:08:11.255144 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:11.253877    4525 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2300b922-2ef9-4c20-8c7d-df89a055374c-home\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:08:11.750725 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:11.750594    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-baa504/auth-test-model-baa504-kserve-64c89767d5-d4pq4" event={"ID":"2300b922-2ef9-4c20-8c7d-df89a055374c","Type":"ContainerDied","Data":"949d824b2d5271f22ff092e162ecd6c45ba5cc38ba2ef9ddb8034c6e8b9dea9e"}
Oct 30 19:08:11.754207 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:11.750999    4525 scope.go:122] "RemoveContainer" containerID="a30a587db8cb8888869d9865080e274038daa33ec39a4edfc514961887edf430"
Oct 30 19:08:11.754207 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:11.751287    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-baa504/auth-test-model-baa504-kserve-64c89767d5-d4pq4"
Oct 30 19:08:11.758882 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:11.758851    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wlrxd" event={"ID":"aa550fe6-5b53-439d-9c72-391ed15e1fb3","Type":"ContainerStarted","Data":"63e07cf12e4652c483cf5674d2bc5c7c6bd40e194fc52e3e4b60a2a83983c7df"}
Oct 30 19:08:11.760230 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:11.760211    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wlrxd"
Oct 30 19:08:11.771732 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:11.771705    4525 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-wlrxd container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:08:11.771916 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:11.771894    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wlrxd" podUID="aa550fe6-5b53-439d-9c72-391ed15e1fb3" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:08:11.815365 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:11.815271    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wlrxd" podStartSLOduration=2.815248028 podStartE2EDuration="2.815248028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:08:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:08:11.804563307 +0000 UTC m=+1914.430233359" watchObservedRunningTime="2026-10-30 19:08:11.815248028 +0000 UTC m=+1914.440918053"
Oct 30 19:08:11.837408 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:11.837179    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-baa504/auth-test-model-baa504-kserve-64c89767d5-d4pq4"]
Oct 30 19:08:11.845907 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:11.845860    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-baa504/auth-test-model-baa504-kserve-64c89767d5-d4pq4"]
Oct 30 19:08:11.927740 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:11.910785    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-baa504/auth-test-model-baa504-kserve-64c89767d5-d4pq4" podUID="2300b922-2ef9-4c20-8c7d-df89a055374c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.205:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:08:11.959872 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:11.946341    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-68b466/auth-test-model-68b466-kserve-6dd5b87d8f-ddr8r" podUID="1054f7aa-9712-425b-8e66-d78b0f17962a" containerName="main" containerID="cri-o://27c64be629dbbf8354b606dc7787fdb11e3633e329c6ee1fbdd848ab55d62f84" gracePeriod=45
Oct 30 19:08:12.404375 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:12.404340    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-68b466/auth-test-model-68b466-kserve-6dd5b87d8f-ddr8r"
Oct 30 19:08:12.547483 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:12.547440    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1054f7aa-9712-425b-8e66-d78b0f17962a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1054f7aa-9712-425b-8e66-d78b0f17962a-dshm\") pod \"1054f7aa-9712-425b-8e66-d78b0f17962a\" (UID: \"1054f7aa-9712-425b-8e66-d78b0f17962a\") "
Oct 30 19:08:12.547709 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:12.547572    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1054f7aa-9712-425b-8e66-d78b0f17962a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1054f7aa-9712-425b-8e66-d78b0f17962a-model-cache\") pod \"1054f7aa-9712-425b-8e66-d78b0f17962a\" (UID: \"1054f7aa-9712-425b-8e66-d78b0f17962a\") "
Oct 30 19:08:12.547925 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:12.547888    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1054f7aa-9712-425b-8e66-d78b0f17962a-model-cache" pod "1054f7aa-9712-425b-8e66-d78b0f17962a" (UID: "1054f7aa-9712-425b-8e66-d78b0f17962a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:08:12.548167 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:12.548145    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1054f7aa-9712-425b-8e66-d78b0f17962a-home\" (UniqueName: \"kubernetes.io/empty-dir/1054f7aa-9712-425b-8e66-d78b0f17962a-home\") pod \"1054f7aa-9712-425b-8e66-d78b0f17962a\" (UID: \"1054f7aa-9712-425b-8e66-d78b0f17962a\") "
Oct 30 19:08:12.548469 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:12.548450    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1054f7aa-9712-425b-8e66-d78b0f17962a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1054f7aa-9712-425b-8e66-d78b0f17962a-tmp-dir\") pod \"1054f7aa-9712-425b-8e66-d78b0f17962a\" (UID: \"1054f7aa-9712-425b-8e66-d78b0f17962a\") "
Oct 30 19:08:12.548553 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:12.548391    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1054f7aa-9712-425b-8e66-d78b0f17962a-home" pod "1054f7aa-9712-425b-8e66-d78b0f17962a" (UID: "1054f7aa-9712-425b-8e66-d78b0f17962a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:08:12.548699 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:12.548666    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1054f7aa-9712-425b-8e66-d78b0f17962a-tmp-dir" pod "1054f7aa-9712-425b-8e66-d78b0f17962a" (UID: "1054f7aa-9712-425b-8e66-d78b0f17962a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:08:12.548820 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:12.548737    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1054f7aa-9712-425b-8e66-d78b0f17962a-tls-certs\" (UniqueName: \"kubernetes.io/secret/1054f7aa-9712-425b-8e66-d78b0f17962a-tls-certs\") pod \"1054f7aa-9712-425b-8e66-d78b0f17962a\" (UID: \"1054f7aa-9712-425b-8e66-d78b0f17962a\") "
Oct 30 19:08:12.548924 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:12.548894    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1054f7aa-9712-425b-8e66-d78b0f17962a-kube-api-access-59mv5\" (UniqueName: \"kubernetes.io/projected/1054f7aa-9712-425b-8e66-d78b0f17962a-kube-api-access-59mv5\") pod \"1054f7aa-9712-425b-8e66-d78b0f17962a\" (UID: \"1054f7aa-9712-425b-8e66-d78b0f17962a\") "
Oct 30 19:08:12.549759 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:12.549732    4525 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1054f7aa-9712-425b-8e66-d78b0f17962a-model-cache\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:08:12.549866 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:12.549764    4525 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1054f7aa-9712-425b-8e66-d78b0f17962a-home\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:08:12.549866 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:12.549778    4525 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1054f7aa-9712-425b-8e66-d78b0f17962a-tmp-dir\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:08:12.550387 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:12.550355    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1054f7aa-9712-425b-8e66-d78b0f17962a-dshm" pod "1054f7aa-9712-425b-8e66-d78b0f17962a" (UID: "1054f7aa-9712-425b-8e66-d78b0f17962a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:08:12.551299 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:12.551278    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1054f7aa-9712-425b-8e66-d78b0f17962a-tls-certs" pod "1054f7aa-9712-425b-8e66-d78b0f17962a" (UID: "1054f7aa-9712-425b-8e66-d78b0f17962a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:08:12.551859 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:12.551831    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1054f7aa-9712-425b-8e66-d78b0f17962a-kube-api-access-59mv5" pod "1054f7aa-9712-425b-8e66-d78b0f17962a" (UID: "1054f7aa-9712-425b-8e66-d78b0f17962a"). InnerVolumeSpecName "kube-api-access-59mv5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:08:12.669442 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:12.668945    4525 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1054f7aa-9712-425b-8e66-d78b0f17962a-tls-certs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:08:12.669442 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:12.668984    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59mv5\" (UniqueName: \"kubernetes.io/projected/1054f7aa-9712-425b-8e66-d78b0f17962a-kube-api-access-59mv5\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:08:12.669442 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:12.668999    4525 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1054f7aa-9712-425b-8e66-d78b0f17962a-dshm\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:08:12.779338 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:12.779299    4525 generic.go:356] "Generic (PLEG): container finished" podID="1054f7aa-9712-425b-8e66-d78b0f17962a" containerID="27c64be629dbbf8354b606dc7787fdb11e3633e329c6ee1fbdd848ab55d62f84" exitCode=0
Oct 30 19:08:12.779527 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:12.779512    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-68b466/auth-test-model-68b466-kserve-6dd5b87d8f-ddr8r"
Oct 30 19:08:12.780355 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:12.780330    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-68b466/auth-test-model-68b466-kserve-6dd5b87d8f-ddr8r" event={"ID":"1054f7aa-9712-425b-8e66-d78b0f17962a","Type":"ContainerDied","Data":"27c64be629dbbf8354b606dc7787fdb11e3633e329c6ee1fbdd848ab55d62f84"}
Oct 30 19:08:12.780462 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:12.780374    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-68b466/auth-test-model-68b466-kserve-6dd5b87d8f-ddr8r" event={"ID":"1054f7aa-9712-425b-8e66-d78b0f17962a","Type":"ContainerDied","Data":"2b552003c8632db1dcc7532c442d6cdce9079133091a56e81cd7e89ae2c0a6a4"}
Oct 30 19:08:12.780462 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:12.780391    4525 scope.go:122] "RemoveContainer" containerID="27c64be629dbbf8354b606dc7787fdb11e3633e329c6ee1fbdd848ab55d62f84"
Oct 30 19:08:12.786000 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:12.785971    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wlrxd"
Oct 30 19:08:12.795914 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:12.795888    4525 scope.go:122] "RemoveContainer" containerID="27c64be629dbbf8354b606dc7787fdb11e3633e329c6ee1fbdd848ab55d62f84"
Oct 30 19:08:12.796277 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:08:12.796241    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27c64be629dbbf8354b606dc7787fdb11e3633e329c6ee1fbdd848ab55d62f84\": container with ID starting with 27c64be629dbbf8354b606dc7787fdb11e3633e329c6ee1fbdd848ab55d62f84 not found: ID does not exist" containerID="27c64be629dbbf8354b606dc7787fdb11e3633e329c6ee1fbdd848ab55d62f84"
Oct 30 19:08:12.796362 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:12.796277    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27c64be629dbbf8354b606dc7787fdb11e3633e329c6ee1fbdd848ab55d62f84"} err="failed to get container status \"27c64be629dbbf8354b606dc7787fdb11e3633e329c6ee1fbdd848ab55d62f84\": rpc error: code = NotFound desc = could not find container \"27c64be629dbbf8354b606dc7787fdb11e3633e329c6ee1fbdd848ab55d62f84\": container with ID starting with 27c64be629dbbf8354b606dc7787fdb11e3633e329c6ee1fbdd848ab55d62f84 not found: ID does not exist"
Oct 30 19:08:12.834578 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:12.834510    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-68b466/auth-test-model-68b466-kserve-6dd5b87d8f-ddr8r"]
Oct 30 19:08:12.847968 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:12.842460    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-68b466/auth-test-model-68b466-kserve-6dd5b87d8f-ddr8r"]
Oct 30 19:08:12.994897 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:12.994815    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:08:12.994897 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:12.994874    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b" podUID="a8e73ad5-17f6-4205-bc5f-8965afa49f57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:08:13.565862 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:13.565818    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1054f7aa-9712-425b-8e66-d78b0f17962a" path="/var/lib/kubelet/pods/1054f7aa-9712-425b-8e66-d78b0f17962a/volumes"
Oct 30 19:08:13.572017 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:13.571740    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2300b922-2ef9-4c20-8c7d-df89a055374c" path="/var/lib/kubelet/pods/2300b922-2ef9-4c20-8c7d-df89a055374c/volumes"
Oct 30 19:08:18.241789 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:08:18.241745    4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d49l" containerName="extractor"
Oct 30 19:08:21.029257 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:21.029219    4525 scope.go:122] "RemoveContainer" containerID="335e35fb4b1342a9a2103c25ec31815838310b69db68139b24b4183a4191750d"
Oct 30 19:08:21.070482 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:21.070458    4525 scope.go:122] "RemoveContainer" containerID="17141dcbd8999ddd4d7fad1bc31f4f8fe0474c633eeb13c4fcbafce242d926b7"
Oct 30 19:08:21.094112 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:21.094082    4525 scope.go:122] "RemoveContainer" containerID="503c49b2f1e7e8e2d248795849299ca3d0654f870502ace4d3820fd1275799b8"
Oct 30 19:08:21.117545 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:21.117521    4525 scope.go:122] "RemoveContainer" containerID="aff0bc3b44c94c5eaf2f85e12c58c5aa12e11f570e8cad741bc94433e6a5bd86"
Oct 30 19:08:21.152150 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:21.152121    4525 scope.go:122] "RemoveContainer" containerID="3403f2bca61b8b74b244abb2b28473ae39f48739527ca4cc26232755be53e9c9"
Oct 30 19:08:21.196995 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:21.196974    4525 scope.go:122] "RemoveContainer" containerID="88ba3292d9cf75f275029cedd83c48e22836f50c28a7b3b554dc305ab000e4c8"
Oct 30 19:08:21.231015 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:21.230973    4525 scope.go:122] "RemoveContainer" containerID="a1a69b5630659aa7bba6c90b110cbf32204c266d43c744608e47e6e22116c0d6"
Oct 30 19:08:24.387078 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:24.387042    4525 generic.go:356] "Generic (PLEG): container finished" podID="a8e73ad5-17f6-4205-bc5f-8965afa49f57" containerID="1cfe61de8850135e71e9c6cd76641bf7efac9ef158b8006ebc4be79165fb08e3" exitCode=0
Oct 30 19:08:24.387603 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:24.387118    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b" event={"ID":"a8e73ad5-17f6-4205-bc5f-8965afa49f57","Type":"ContainerDied","Data":"1cfe61de8850135e71e9c6cd76641bf7efac9ef158b8006ebc4be79165fb08e3"}
Oct 30 19:08:24.387787 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:24.387768    4525 scope.go:122] "RemoveContainer" containerID="1cfe61de8850135e71e9c6cd76641bf7efac9ef158b8006ebc4be79165fb08e3"
Oct 30 19:08:24.389947 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:24.389917    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:08:24.390083 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:24.390005    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:08:24.390083 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:24.390066    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:08:24.994930 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:24.994836    4525 patch_prober.go:29] interesting pod/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-f5ct2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:08:24.994930 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:24.994919    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-f5ct2" podUID="d9094695-97bc-4924-9526-45b9e8892662" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:08:25.417766 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:25.416959    4525 generic.go:356] "Generic (PLEG): container finished" podID="d9094695-97bc-4924-9526-45b9e8892662" containerID="e2d446f40cf1dd6c12c1badf1f879711f7cb234dd8dca27e13a442572a339ac4" exitCode=0
Oct 30 19:08:25.417766 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:25.417242    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-f5ct2" event={"ID":"d9094695-97bc-4924-9526-45b9e8892662","Type":"ContainerDied","Data":"e2d446f40cf1dd6c12c1badf1f879711f7cb234dd8dca27e13a442572a339ac4"}
Oct 30 19:08:25.418353 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:25.417861    4525 scope.go:122] "RemoveContainer" containerID="e2d446f40cf1dd6c12c1badf1f879711f7cb234dd8dca27e13a442572a339ac4"
Oct 30 19:08:25.423068 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:25.421998    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b" event={"ID":"a8e73ad5-17f6-4205-bc5f-8965afa49f57","Type":"ContainerStarted","Data":"23114b3fc0f0f25c8de13319f8e7cb8a2484deaf68608b286435ca9cb76b69ef"}
Oct 30 19:08:25.423068 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:25.422014    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:08:25.423068 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:25.422123    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:08:25.423068 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:25.422201    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:08:26.001710 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:26.000072    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b"
Oct 30 19:08:26.002385 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:26.002356    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body=
Oct 30 19:08:26.002619 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:26.002592    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b" podUID="a8e73ad5-17f6-4205-bc5f-8965afa49f57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused"
Oct 30 19:08:26.440759 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:26.440729    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-f5ct2" event={"ID":"d9094695-97bc-4924-9526-45b9e8892662","Type":"ContainerStarted","Data":"f9abfb3593f30d2a2f3db76bff3f94bc481e32e4fa4b29f164b78571af541af2"}
Oct 30 19:08:26.952369 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:26.952331    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b"
Oct 30 19:08:26.999095 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:26.999056    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body=
Oct 30 19:08:26.999386 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:26.999359    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b" podUID="a8e73ad5-17f6-4205-bc5f-8965afa49f57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused"
Oct 30 19:08:27.295201 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:27.295104    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-f5ct2"
Oct 30 19:08:27.295529 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:27.295506    4525 patch_prober.go:29] interesting pod/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-f5ct2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body=
Oct 30 19:08:27.295710 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:27.295660    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-f5ct2" podUID="d9094695-97bc-4924-9526-45b9e8892662" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused"
Oct 30 19:08:27.999121 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:27.999079    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body=
Oct 30 19:08:27.999758 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:27.999721    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b" podUID="a8e73ad5-17f6-4205-bc5f-8965afa49f57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused"
Oct 30 19:08:28.310806 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:28.306394    4525 patch_prober.go:29] interesting pod/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-f5ct2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body=
Oct 30 19:08:28.311085 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:28.311049    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-f5ct2" podUID="d9094695-97bc-4924-9526-45b9e8892662" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused"
Oct 30 19:08:29.018708 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:29.017822    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body=
Oct 30 19:08:29.018708 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:29.017892    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b" podUID="a8e73ad5-17f6-4205-bc5f-8965afa49f57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused"
Oct 30 19:08:29.302885 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:29.300040    4525 patch_prober.go:29] interesting pod/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-f5ct2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body=
Oct 30 19:08:29.302885 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:29.300108    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-f5ct2" podUID="d9094695-97bc-4924-9526-45b9e8892662" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused"
Oct 30 19:08:30.000706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:30.000040    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body=
Oct 30 19:08:30.000706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:30.000105    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b" podUID="a8e73ad5-17f6-4205-bc5f-8965afa49f57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused"
Oct 30 19:08:30.304766 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:30.304043    4525 patch_prober.go:29] interesting pod/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-f5ct2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body=
Oct 30 19:08:30.304766 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:30.304110    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-f5ct2" podUID="d9094695-97bc-4924-9526-45b9e8892662" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused"
Oct 30 19:08:31.018392 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:31.018356    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b"
Oct 30 19:08:31.025847 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:31.025606    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b"
Oct 30 19:08:31.302760 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:31.299091    4525 patch_prober.go:29] interesting pod/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-f5ct2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body=
Oct 30 19:08:31.302760 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:31.299156    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-f5ct2" podUID="d9094695-97bc-4924-9526-45b9e8892662" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused"
Oct 30 19:08:32.300043 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:32.300012    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-f5ct2"
Oct 30 19:08:32.300541 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:32.300493    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-f5ct2"
Oct 30 19:08:32.301273 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:08:32.301252    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-f5ct2"
Oct 30 19:08:47.454525 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:08:47.454462    4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d49l" containerName="extractor"
Oct 30 19:09:01.084042 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:01.082290    4525 patch_prober.go:29] interesting pod/e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:09:01.084042 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:01.082368    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4" podUID="0650ae0c-36c1-4b6e-9212-12264a92c920" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:09:01.492876 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:01.492838    4525 patch_prober.go:29] interesting pod/e2e-autores-c164d6-openshift-default-74c7bf49f9-4dd4v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.227:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:09:01.493080 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:01.492926    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4dd4v" podUID="feea5f3c-8630-4e8f-8832-2998030bf2e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.227:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:09:05.088858 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:05.088818    4525 patch_prober.go:29] interesting pod/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.226:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:09:05.089336 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:05.088884    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7" podUID="77adecb1-af53-4dd9-bb52-d8e22ebb03e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.226:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:09:05.160729 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:05.160640    4525 generic.go:356] "Generic (PLEG): container finished" podID="0650ae0c-36c1-4b6e-9212-12264a92c920" containerID="ddde0bd43d503e1026196638a207f5250f5d53e85a6bd1119d79bc22736da3a3" exitCode=0
Oct 30 19:09:05.160883 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:05.160726    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4" event={"ID":"0650ae0c-36c1-4b6e-9212-12264a92c920","Type":"ContainerDied","Data":"ddde0bd43d503e1026196638a207f5250f5d53e85a6bd1119d79bc22736da3a3"}
Oct 30 19:09:05.161321 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:05.161300    4525 scope.go:122] "RemoveContainer" containerID="ddde0bd43d503e1026196638a207f5250f5d53e85a6bd1119d79bc22736da3a3"
Oct 30 19:09:05.163502 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:05.163469    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:09:05.163604 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:05.163548    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:09:05.163604 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:05.163583    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:09:06.172579 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:06.172534    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4" event={"ID":"0650ae0c-36c1-4b6e-9212-12264a92c920","Type":"ContainerStarted","Data":"0bc4d925a818f8e98248567d5c9c1915c6cd0b32a9fb2e98b6959553ed535f14"}
Oct 30 19:09:06.441290 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:06.441202    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4"
Oct 30 19:09:06.441486 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:06.441440    4525 patch_prober.go:29] interesting pod/e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body=
Oct 30 19:09:06.441554 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:06.441490    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4" podUID="0650ae0c-36c1-4b6e-9212-12264a92c920" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused"
Oct 30 19:09:06.870411 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:06.870376    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-7ttp9"]
Oct 30 19:09:07.005708 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:07.000951    4525 cpu_manager.go:414] "containerMap: removing container" podUID="2300b922-2ef9-4c20-8c7d-df89a055374c" containerName="main"
Oct 30 19:09:07.005708 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:07.000988    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2300b922-2ef9-4c20-8c7d-df89a055374c" containerName="main"
Oct 30 19:09:07.005708 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:07.001047    4525 cpu_manager.go:414] "containerMap: removing container" podUID="1054f7aa-9712-425b-8e66-d78b0f17962a" containerName="main"
Oct 30 19:09:07.005708 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:07.001057    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1054f7aa-9712-425b-8e66-d78b0f17962a" containerName="main"
Oct 30 19:09:07.005708 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:07.001312    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="1054f7aa-9712-425b-8e66-d78b0f17962a" containerName="main"
Oct 30 19:09:07.005708 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:07.001338    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="2300b922-2ef9-4c20-8c7d-df89a055374c" containerName="main"
Oct 30 19:09:07.037525 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:07.037495    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-7ttp9"]
Oct 30 19:09:07.037702 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:07.037544    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-t8s45"]
Oct 30 19:09:07.037702 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:07.037628    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-7ttp9"
Oct 30 19:09:07.056066 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:07.056029    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-t8s45"]
Oct 30 19:09:07.056223 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:07.056156    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-t8s45"
Oct 30 19:09:07.122664 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:07.122573    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06b7dec0-c44b-4aea-a753-53ec832dfd24-istio-token\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-t8s45\" (UID: \"06b7dec0-c44b-4aea-a753-53ec832dfd24\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-t8s45"
Oct 30 19:09:07.122664 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:07.122622    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/960cce33-8574-42ab-8d16-c4fb2f0b3202-istiod-ca-cert\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-7ttp9\" (UID: \"960cce33-8574-42ab-8d16-c4fb2f0b3202\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-7ttp9"
Oct 30 19:09:07.122924 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:07.122846    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/960cce33-8574-42ab-8d16-c4fb2f0b3202-istio-podinfo\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-7ttp9\" (UID: \"960cce33-8574-42ab-8d16-c4fb2f0b3202\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-7ttp9"
Oct 30 19:09:07.122983 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:07.122923    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06b7dec0-c44b-4aea-a753-53ec832dfd24-credential-socket\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-t8s45\" (UID: \"06b7dec0-c44b-4aea-a753-53ec832dfd24\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-t8s45"
Oct 30 19:09:07.122983 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:07.122978    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06b7dec0-c44b-4aea-a753-53ec832dfd24-istio-podinfo\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-t8s45\" (UID: \"06b7dec0-c44b-4aea-a753-53ec832dfd24\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-t8s45"
Oct 30 19:09:07.123115 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:07.123096    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/960cce33-8574-42ab-8d16-c4fb2f0b3202-workload-socket\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-7ttp9\" (UID: \"960cce33-8574-42ab-8d16-c4fb2f0b3202\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-7ttp9"
Oct 30 19:09:07.123166 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:07.123127    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06b7dec0-c44b-4aea-a753-53ec832dfd24-istio-data\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-t8s45\" (UID: \"06b7dec0-c44b-4aea-a753-53ec832dfd24\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-t8s45"
Oct 30 19:09:07.123249 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:07.123229    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/960cce33-8574-42ab-8d16-c4fb2f0b3202-istio-data\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-7ttp9\" (UID: \"960cce33-8574-42ab-8d16-c4fb2f0b3202\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-7ttp9"
Oct 30 19:09:07.123304 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:07.123257    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bjhw\" (UniqueName: \"kubernetes.io/projected/960cce33-8574-42ab-8d16-c4fb2f0b3202-kube-api-access-2bjhw\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-7ttp9\" (UID: \"960cce33-8574-42ab-8d16-c4fb2f0b3202\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-7ttp9"
Oct 30 19:09:07.123304 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:07.123275    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv7vb\" (UniqueName: \"kubernetes.io/projected/06b7dec0-c44b-4aea-a753-53ec832dfd24-kube-api-access-gv7vb\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-t8s45\" (UID: \"06b7dec0-c44b-4aea-a753-53ec832dfd24\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-t8s45"
Oct 30 19:09:07.123408 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:07.123312    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/960cce33-8574-42ab-8d16-c4fb2f0b3202-workload-certs\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-7ttp9\" (UID: \"960cce33-8574-42ab-8d16-c4fb2f0b3202\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-7ttp9"
Oct 30 19:09:07.123408 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:07.123336    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06b7dec0-c44b-4aea-a753-53ec832dfd24-workload-certs\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-t8s45\" (UID: \"06b7dec0-c44b-4aea-a753-53ec832dfd24\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-t8s45"
Oct 30 19:09:07.123408 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:07.123361    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/960cce33-8574-42ab-8d16-c4fb2f0b3202-credential-socket\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-7ttp9\" (UID: \"960cce33-8574-42ab-8d16-c4fb2f0b3202\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-7ttp9"
Oct 30 19:09:07.123529 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:07.123448    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06b7dec0-c44b-4aea-a753-53ec832dfd24-workload-socket\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-t8s45\" (UID: \"06b7dec0-c44b-4aea-a753-53ec832dfd24\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-t8s45"
Oct 30 19:09:07.123529 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:07.123504    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/960cce33-8574-42ab-8d16-c4fb2f0b3202-istio-envoy\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-7ttp9\" (UID: \"960cce33-8574-42ab-8d16-c4fb2f0b3202\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-7ttp9"
Oct 30 19:09:07.123599 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:07.123532    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06b7dec0-c44b-4aea-a753-53ec832dfd24-istio-envoy\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-t8s45\" (UID: \"06b7dec0-c44b-4aea-a753-53ec832dfd24\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-t8s45"
Oct 30 19:09:07.123599 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:07.123581    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/960cce33-8574-42ab-8d16-c4fb2f0b3202-istio-token\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-7ttp9\" (UID: \"960cce33-8574-42ab-8d16-c4fb2f0b3202\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-7ttp9"
Oct 30 19:09:07.123726 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:07.123600    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06b7dec0-c44b-4aea-a753-53ec832dfd24-istiod-ca-cert\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-t8s45\" (UID: \"06b7dec0-c44b-4aea-a753-53ec832dfd24\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-t8s45"
Oct 30 19:09:07.225894 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:07.225859    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/960cce33-8574-42ab-8d16-c4fb2f0b3202-istio-token\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-7ttp9\" (UID: \"960cce33-8574-42ab-8d16-c4fb2f0b3202\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-7ttp9"
Oct 30 19:09:07.226405 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:07.225902    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06b7dec0-c44b-4aea-a753-53ec832dfd24-istiod-ca-cert\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-t8s45\" (UID: \"06b7dec0-c44b-4aea-a753-53ec832dfd24\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-t8s45"
Oct 30 19:09:07.226405 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:07.225942    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06b7dec0-c44b-4aea-a753-53ec832dfd24-istio-token\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-t8s45\" (UID: \"06b7dec0-c44b-4aea-a753-53ec832dfd24\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-t8s45"
Oct 30 19:09:07.226405 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:07.225975    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/960cce33-8574-42ab-8d16-c4fb2f0b3202-istiod-ca-cert\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-7ttp9\" (UID: \"960cce33-8574-42ab-8d16-c4fb2f0b3202\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-7ttp9"
Oct 30 19:09:07.226405 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:07.226117    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/960cce33-8574-42ab-8d16-c4fb2f0b3202-istio-podinfo\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-7ttp9\" (UID: \"960cce33-8574-42ab-8d16-c4fb2f0b3202\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-7ttp9"
Oct 30 19:09:07.226405 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:07.226195    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06b7dec0-c44b-4aea-a753-53ec832dfd24-credential-socket\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-t8s45\" (UID: \"06b7dec0-c44b-4aea-a753-53ec832dfd24\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-t8s45"
Oct 30 19:09:07.226405 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:07.226269    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06b7dec0-c44b-4aea-a753-53ec832dfd24-istio-podinfo\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-t8s45\" (UID: \"06b7dec0-c44b-4aea-a753-53ec832dfd24\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-t8s45"
Oct 30 19:09:07.226405 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:07.226355    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/960cce33-8574-42ab-8d16-c4fb2f0b3202-workload-socket\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-7ttp9\" (UID: \"960cce33-8574-42ab-8d16-c4fb2f0b3202\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-7ttp9"
Oct 30 19:09:07.226405 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:07.226373    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06b7dec0-c44b-4aea-a753-53ec832dfd24-istio-data\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-t8s45\" (UID: \"06b7dec0-c44b-4aea-a753-53ec832dfd24\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-t8s45"
Oct 30 19:09:07.227062 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:07.226468    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/960cce33-8574-42ab-8d16-c4fb2f0b3202-istio-data\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-7ttp9\" (UID: \"960cce33-8574-42ab-8d16-c4fb2f0b3202\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-7ttp9"
Oct 30 19:09:07.227062 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:07.226496    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bjhw\" (UniqueName: \"kubernetes.io/projected/960cce33-8574-42ab-8d16-c4fb2f0b3202-kube-api-access-2bjhw\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-7ttp9\" (UID: \"960cce33-8574-42ab-8d16-c4fb2f0b3202\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-7ttp9"
Oct 30 19:09:07.227062 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:07.226521    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv7vb\" (UniqueName: \"kubernetes.io/projected/06b7dec0-c44b-4aea-a753-53ec832dfd24-kube-api-access-gv7vb\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-t8s45\" (UID: \"06b7dec0-c44b-4aea-a753-53ec832dfd24\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-t8s45"
Oct 30 19:09:07.227062 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:07.226562    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/960cce33-8574-42ab-8d16-c4fb2f0b3202-workload-certs\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-7ttp9\" (UID: \"960cce33-8574-42ab-8d16-c4fb2f0b3202\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-7ttp9"
Oct 30 19:09:07.227062 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:07.226601    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06b7dec0-c44b-4aea-a753-53ec832dfd24-workload-certs\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-t8s45\" (UID: \"06b7dec0-c44b-4aea-a753-53ec832dfd24\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-t8s45"
Oct 30 19:09:07.227062 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:07.226637    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06b7dec0-c44b-4aea-a753-53ec832dfd24-credential-socket\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-t8s45\" (UID: \"06b7dec0-c44b-4aea-a753-53ec832dfd24\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-t8s45"
Oct 30 19:09:07.227062 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:07.226647    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/960cce33-8574-42ab-8d16-c4fb2f0b3202-credential-socket\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-7ttp9\" (UID: \"960cce33-8574-42ab-8d16-c4fb2f0b3202\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-7ttp9"
Oct 30 19:09:07.227062 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:07.226748    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06b7dec0-c44b-4aea-a753-53ec832dfd24-workload-socket\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-t8s45\" (UID: \"06b7dec0-c44b-4aea-a753-53ec832dfd24\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-t8s45"
Oct 30 19:09:07.227062 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:07.226766    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06b7dec0-c44b-4aea-a753-53ec832dfd24-istio-data\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-t8s45\" (UID: \"06b7dec0-c44b-4aea-a753-53ec832dfd24\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-t8s45"
Oct 30 19:09:07.227062 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:07.226817    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06b7dec0-c44b-4aea-a753-53ec832dfd24-istiod-ca-cert\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-t8s45\" (UID: \"06b7dec0-c44b-4aea-a753-53ec832dfd24\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-t8s45"
Oct 30 19:09:07.227062 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:07.226824    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/960cce33-8574-42ab-8d16-c4fb2f0b3202-istio-envoy\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-7ttp9\" (UID: \"960cce33-8574-42ab-8d16-c4fb2f0b3202\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-7ttp9"
Oct 30 19:09:07.227062 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:07.226885    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06b7dec0-c44b-4aea-a753-53ec832dfd24-istio-envoy\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-t8s45\" (UID: \"06b7dec0-c44b-4aea-a753-53ec832dfd24\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-t8s45"
Oct 30 19:09:07.227062 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:07.226923    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/960cce33-8574-42ab-8d16-c4fb2f0b3202-istiod-ca-cert\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-7ttp9\" (UID: \"960cce33-8574-42ab-8d16-c4fb2f0b3202\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-7ttp9"
Oct 30 19:09:07.227062 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:07.226993    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/960cce33-8574-42ab-8d16-c4fb2f0b3202-credential-socket\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-7ttp9\" (UID: \"960cce33-8574-42ab-8d16-c4fb2f0b3202\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-7ttp9"
Oct 30 19:09:07.227062 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:07.227016    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06b7dec0-c44b-4aea-a753-53ec832dfd24-workload-socket\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-t8s45\" (UID: \"06b7dec0-c44b-4aea-a753-53ec832dfd24\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-t8s45"
Oct 30 19:09:07.227773 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:07.227138    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/960cce33-8574-42ab-8d16-c4fb2f0b3202-workload-socket\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-7ttp9\" (UID: \"960cce33-8574-42ab-8d16-c4fb2f0b3202\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-7ttp9"
Oct 30 19:09:07.227773 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:07.227267    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/960cce33-8574-42ab-8d16-c4fb2f0b3202-istio-data\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-7ttp9\" (UID: \"960cce33-8574-42ab-8d16-c4fb2f0b3202\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-7ttp9"
Oct 30 19:09:07.227773 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:07.227311    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/960cce33-8574-42ab-8d16-c4fb2f0b3202-workload-certs\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-7ttp9\" (UID: \"960cce33-8574-42ab-8d16-c4fb2f0b3202\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-7ttp9"
Oct 30 19:09:07.227773 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:07.227500    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06b7dec0-c44b-4aea-a753-53ec832dfd24-workload-certs\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-t8s45\" (UID: \"06b7dec0-c44b-4aea-a753-53ec832dfd24\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-t8s45"
Oct 30 19:09:07.353537 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:07.347429    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/960cce33-8574-42ab-8d16-c4fb2f0b3202-istio-token\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-7ttp9\" (UID: \"960cce33-8574-42ab-8d16-c4fb2f0b3202\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-7ttp9"
Oct 30 19:09:07.378818 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:07.378471    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bjhw\" (UniqueName: \"kubernetes.io/projected/960cce33-8574-42ab-8d16-c4fb2f0b3202-kube-api-access-2bjhw\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-7ttp9\" (UID: \"960cce33-8574-42ab-8d16-c4fb2f0b3202\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-7ttp9"
Oct 30 19:09:07.396621 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:07.396539    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06b7dec0-c44b-4aea-a753-53ec832dfd24-istio-podinfo\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-t8s45\" (UID: \"06b7dec0-c44b-4aea-a753-53ec832dfd24\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-t8s45"
Oct 30 19:09:07.397044 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:07.397012    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/960cce33-8574-42ab-8d16-c4fb2f0b3202-istio-envoy\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-7ttp9\" (UID: \"960cce33-8574-42ab-8d16-c4fb2f0b3202\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-7ttp9"
Oct 30 19:09:07.397308 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:07.397286    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06b7dec0-c44b-4aea-a753-53ec832dfd24-istio-envoy\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-t8s45\" (UID: \"06b7dec0-c44b-4aea-a753-53ec832dfd24\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-t8s45"
Oct 30 19:09:07.412160 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:07.409331    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/960cce33-8574-42ab-8d16-c4fb2f0b3202-istio-podinfo\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-7ttp9\" (UID: \"960cce33-8574-42ab-8d16-c4fb2f0b3202\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-7ttp9"
Oct 30 19:09:07.414761 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:07.413451    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv7vb\" (UniqueName: \"kubernetes.io/projected/06b7dec0-c44b-4aea-a753-53ec832dfd24-kube-api-access-gv7vb\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-t8s45\" (UID: \"06b7dec0-c44b-4aea-a753-53ec832dfd24\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-t8s45"
Oct 30 19:09:07.414761 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:07.413853    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06b7dec0-c44b-4aea-a753-53ec832dfd24-istio-token\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-t8s45\" (UID: \"06b7dec0-c44b-4aea-a753-53ec832dfd24\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-t8s45"
Oct 30 19:09:07.442158 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:07.442125    4525 patch_prober.go:29] interesting pod/e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body=
Oct 30 19:09:07.442339 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:07.442181    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4" podUID="0650ae0c-36c1-4b6e-9212-12264a92c920" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused"
Oct 30 19:09:07.665253 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:07.665174    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-7ttp9"
Oct 30 19:09:07.701193 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:07.701048    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-t8s45"
Oct 30 19:09:08.441613 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:08.441570    4525 patch_prober.go:29] interesting pod/e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body=
Oct 30 19:09:08.442276 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:08.441630    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4" podUID="0650ae0c-36c1-4b6e-9212-12264a92c920" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused"
Oct 30 19:09:08.608981 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:08.608941    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-tvbgw"]
Oct 30 19:09:08.673875 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:08.673835    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-tvbgw"]
Oct 30 19:09:08.674087 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:08.673894    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-lkl5j"]
Oct 30 19:09:08.674087 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:08.673948    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-tvbgw"
Oct 30 19:09:08.741841 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:08.741811    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-lkl5j"]
Oct 30 19:09:08.742198 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:08.742126    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-lkl5j"
Oct 30 19:09:08.761812 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:08.761781    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9973f1f-61b0-4d38-a7c2-b9c6fde9da94-istiod-ca-cert\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-tvbgw\" (UID: \"e9973f1f-61b0-4d38-a7c2-b9c6fde9da94\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-tvbgw"
Oct 30 19:09:08.761948 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:08.761838    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9973f1f-61b0-4d38-a7c2-b9c6fde9da94-istio-envoy\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-tvbgw\" (UID: \"e9973f1f-61b0-4d38-a7c2-b9c6fde9da94\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-tvbgw"
Oct 30 19:09:08.761948 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:08.761916    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/865afd2d-c5e2-4ee5-bfdb-303102de6616-istio-podinfo\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-lkl5j\" (UID: \"865afd2d-c5e2-4ee5-bfdb-303102de6616\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-lkl5j"
Oct 30 19:09:08.762132 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:08.761954    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9973f1f-61b0-4d38-a7c2-b9c6fde9da94-istio-data\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-tvbgw\" (UID: \"e9973f1f-61b0-4d38-a7c2-b9c6fde9da94\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-tvbgw"
Oct 30 19:09:08.762132 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:08.761983    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/865afd2d-c5e2-4ee5-bfdb-303102de6616-istio-token\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-lkl5j\" (UID: \"865afd2d-c5e2-4ee5-bfdb-303102de6616\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-lkl5j"
Oct 30 19:09:08.762132 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:08.762043    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/865afd2d-c5e2-4ee5-bfdb-303102de6616-istiod-ca-cert\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-lkl5j\" (UID: \"865afd2d-c5e2-4ee5-bfdb-303102de6616\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-lkl5j"
Oct 30 19:09:08.762132 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:08.762111    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jnx7\" (UniqueName: \"kubernetes.io/projected/e9973f1f-61b0-4d38-a7c2-b9c6fde9da94-kube-api-access-6jnx7\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-tvbgw\" (UID: \"e9973f1f-61b0-4d38-a7c2-b9c6fde9da94\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-tvbgw"
Oct 30 19:09:08.762410 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:08.762170    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/865afd2d-c5e2-4ee5-bfdb-303102de6616-istio-envoy\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-lkl5j\" (UID: \"865afd2d-c5e2-4ee5-bfdb-303102de6616\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-lkl5j"
Oct 30 19:09:08.762410 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:08.762202    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9973f1f-61b0-4d38-a7c2-b9c6fde9da94-credential-socket\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-tvbgw\" (UID: \"e9973f1f-61b0-4d38-a7c2-b9c6fde9da94\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-tvbgw"
Oct 30 19:09:08.762410 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:08.762235    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9973f1f-61b0-4d38-a7c2-b9c6fde9da94-istio-token\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-tvbgw\" (UID: \"e9973f1f-61b0-4d38-a7c2-b9c6fde9da94\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-tvbgw"
Oct 30 19:09:08.762410 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:08.762271    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/865afd2d-c5e2-4ee5-bfdb-303102de6616-workload-socket\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-lkl5j\" (UID: \"865afd2d-c5e2-4ee5-bfdb-303102de6616\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-lkl5j"
Oct 30 19:09:08.762410 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:08.762295    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/865afd2d-c5e2-4ee5-bfdb-303102de6616-workload-certs\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-lkl5j\" (UID: \"865afd2d-c5e2-4ee5-bfdb-303102de6616\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-lkl5j"
Oct 30 19:09:08.762410 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:08.762332    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/865afd2d-c5e2-4ee5-bfdb-303102de6616-credential-socket\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-lkl5j\" (UID: \"865afd2d-c5e2-4ee5-bfdb-303102de6616\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-lkl5j"
Oct 30 19:09:08.762410 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:08.762391    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/865afd2d-c5e2-4ee5-bfdb-303102de6616-istio-data\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-lkl5j\" (UID: \"865afd2d-c5e2-4ee5-bfdb-303102de6616\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-lkl5j"
Oct 30 19:09:08.763067 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:08.762456    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlkhf\" (UniqueName: \"kubernetes.io/projected/865afd2d-c5e2-4ee5-bfdb-303102de6616-kube-api-access-dlkhf\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-lkl5j\" (UID: \"865afd2d-c5e2-4ee5-bfdb-303102de6616\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-lkl5j"
Oct 30 19:09:08.763067 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:08.762545    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9973f1f-61b0-4d38-a7c2-b9c6fde9da94-workload-certs\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-tvbgw\" (UID: \"e9973f1f-61b0-4d38-a7c2-b9c6fde9da94\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-tvbgw"
Oct 30 19:09:08.763067 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:08.762641    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9973f1f-61b0-4d38-a7c2-b9c6fde9da94-istio-podinfo\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-tvbgw\" (UID: \"e9973f1f-61b0-4d38-a7c2-b9c6fde9da94\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-tvbgw"
Oct 30 19:09:08.763067 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:08.762704    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9973f1f-61b0-4d38-a7c2-b9c6fde9da94-workload-socket\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-tvbgw\" (UID: \"e9973f1f-61b0-4d38-a7c2-b9c6fde9da94\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-tvbgw"
Oct 30 19:09:08.864747 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:08.864706    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/865afd2d-c5e2-4ee5-bfdb-303102de6616-istio-podinfo\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-lkl5j\" (UID: \"865afd2d-c5e2-4ee5-bfdb-303102de6616\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-lkl5j"
Oct 30 19:09:08.864920 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:08.864777    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9973f1f-61b0-4d38-a7c2-b9c6fde9da94-istio-data\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-tvbgw\" (UID: \"e9973f1f-61b0-4d38-a7c2-b9c6fde9da94\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-tvbgw"
Oct 30 19:09:08.864920 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:08.864816    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/865afd2d-c5e2-4ee5-bfdb-303102de6616-istio-token\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-lkl5j\" (UID: \"865afd2d-c5e2-4ee5-bfdb-303102de6616\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-lkl5j"
Oct 30 19:09:08.865453 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:08.865424    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/865afd2d-c5e2-4ee5-bfdb-303102de6616-istiod-ca-cert\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-lkl5j\" (UID: \"865afd2d-c5e2-4ee5-bfdb-303102de6616\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-lkl5j"
Oct 30 19:09:08.865558 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:08.865532    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jnx7\" (UniqueName: \"kubernetes.io/projected/e9973f1f-61b0-4d38-a7c2-b9c6fde9da94-kube-api-access-6jnx7\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-tvbgw\" (UID: \"e9973f1f-61b0-4d38-a7c2-b9c6fde9da94\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-tvbgw"
Oct 30 19:09:08.865623 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:08.865609    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/865afd2d-c5e2-4ee5-bfdb-303102de6616-istio-envoy\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-lkl5j\" (UID: \"865afd2d-c5e2-4ee5-bfdb-303102de6616\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-lkl5j"
Oct 30 19:09:08.865703 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:08.865652    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9973f1f-61b0-4d38-a7c2-b9c6fde9da94-credential-socket\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-tvbgw\" (UID: \"e9973f1f-61b0-4d38-a7c2-b9c6fde9da94\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-tvbgw"
Oct 30 19:09:08.865792 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:08.865758    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9973f1f-61b0-4d38-a7c2-b9c6fde9da94-istio-token\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-tvbgw\" (UID: \"e9973f1f-61b0-4d38-a7c2-b9c6fde9da94\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-tvbgw"
Oct 30 19:09:08.865879 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:08.865830    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/865afd2d-c5e2-4ee5-bfdb-303102de6616-workload-socket\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-lkl5j\" (UID: \"865afd2d-c5e2-4ee5-bfdb-303102de6616\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-lkl5j"
Oct 30 19:09:08.865939 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:08.865898    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/865afd2d-c5e2-4ee5-bfdb-303102de6616-workload-certs\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-lkl5j\" (UID: \"865afd2d-c5e2-4ee5-bfdb-303102de6616\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-lkl5j"
Oct 30 19:09:08.865999 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:08.865936    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/865afd2d-c5e2-4ee5-bfdb-303102de6616-credential-socket\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-lkl5j\" (UID: \"865afd2d-c5e2-4ee5-bfdb-303102de6616\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-lkl5j"
Oct 30 19:09:08.866069 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:08.866004    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/865afd2d-c5e2-4ee5-bfdb-303102de6616-istio-data\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-lkl5j\" (UID: \"865afd2d-c5e2-4ee5-bfdb-303102de6616\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-lkl5j"
Oct 30 19:09:08.866135 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:08.866078    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlkhf\" (UniqueName: \"kubernetes.io/projected/865afd2d-c5e2-4ee5-bfdb-303102de6616-kube-api-access-dlkhf\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-lkl5j\" (UID: \"865afd2d-c5e2-4ee5-bfdb-303102de6616\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-lkl5j"
Oct 30 19:09:08.866135 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:08.866117    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9973f1f-61b0-4d38-a7c2-b9c6fde9da94-workload-certs\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-tvbgw\" (UID: \"e9973f1f-61b0-4d38-a7c2-b9c6fde9da94\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-tvbgw"
Oct 30 19:09:08.866248 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:08.866151    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9973f1f-61b0-4d38-a7c2-b9c6fde9da94-istio-podinfo\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-tvbgw\" (UID: \"e9973f1f-61b0-4d38-a7c2-b9c6fde9da94\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-tvbgw"
Oct 30 19:09:08.866248 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:08.866177    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9973f1f-61b0-4d38-a7c2-b9c6fde9da94-workload-socket\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-tvbgw\" (UID: \"e9973f1f-61b0-4d38-a7c2-b9c6fde9da94\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-tvbgw"
Oct 30 19:09:08.866248 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:08.866244    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9973f1f-61b0-4d38-a7c2-b9c6fde9da94-istiod-ca-cert\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-tvbgw\" (UID: \"e9973f1f-61b0-4d38-a7c2-b9c6fde9da94\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-tvbgw"
Oct 30 19:09:08.866430 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:08.866287    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9973f1f-61b0-4d38-a7c2-b9c6fde9da94-istio-envoy\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-tvbgw\" (UID: \"e9973f1f-61b0-4d38-a7c2-b9c6fde9da94\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-tvbgw"
Oct 30 19:09:08.867205 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:08.867179    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/865afd2d-c5e2-4ee5-bfdb-303102de6616-workload-certs\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-lkl5j\" (UID: \"865afd2d-c5e2-4ee5-bfdb-303102de6616\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-lkl5j"
Oct 30 19:09:08.867364 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:08.867338    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9973f1f-61b0-4d38-a7c2-b9c6fde9da94-workload-certs\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-tvbgw\" (UID: \"e9973f1f-61b0-4d38-a7c2-b9c6fde9da94\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-tvbgw"
Oct 30 19:09:08.867655 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:08.867628    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/865afd2d-c5e2-4ee5-bfdb-303102de6616-credential-socket\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-lkl5j\" (UID: \"865afd2d-c5e2-4ee5-bfdb-303102de6616\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-lkl5j"
Oct 30 19:09:08.867860 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:08.867835    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/865afd2d-c5e2-4ee5-bfdb-303102de6616-istiod-ca-cert\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-lkl5j\" (UID: \"865afd2d-c5e2-4ee5-bfdb-303102de6616\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-lkl5j"
Oct 30 19:09:08.868249 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:08.868222    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/865afd2d-c5e2-4ee5-bfdb-303102de6616-istio-data\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-lkl5j\" (UID: \"865afd2d-c5e2-4ee5-bfdb-303102de6616\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-lkl5j"
Oct 30 19:09:08.868992 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:08.868488    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9973f1f-61b0-4d38-a7c2-b9c6fde9da94-credential-socket\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-tvbgw\" (UID: \"e9973f1f-61b0-4d38-a7c2-b9c6fde9da94\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-tvbgw"
Oct 30 19:09:08.868992 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:08.868750    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/865afd2d-c5e2-4ee5-bfdb-303102de6616-workload-socket\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-lkl5j\" (UID: \"865afd2d-c5e2-4ee5-bfdb-303102de6616\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-lkl5j"
Oct 30 19:09:08.868992 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:08.868786    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9973f1f-61b0-4d38-a7c2-b9c6fde9da94-workload-socket\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-tvbgw\" (UID: \"e9973f1f-61b0-4d38-a7c2-b9c6fde9da94\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-tvbgw"
Oct 30 19:09:08.868992 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:08.868849    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9973f1f-61b0-4d38-a7c2-b9c6fde9da94-istio-data\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-tvbgw\" (UID: \"e9973f1f-61b0-4d38-a7c2-b9c6fde9da94\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-tvbgw"
Oct 30 19:09:08.869539 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:08.869513    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9973f1f-61b0-4d38-a7c2-b9c6fde9da94-istiod-ca-cert\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-tvbgw\" (UID: \"e9973f1f-61b0-4d38-a7c2-b9c6fde9da94\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-tvbgw"
Oct 30 19:09:08.871005 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:08.870960    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/865afd2d-c5e2-4ee5-bfdb-303102de6616-istio-podinfo\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-lkl5j\" (UID: \"865afd2d-c5e2-4ee5-bfdb-303102de6616\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-lkl5j"
Oct 30 19:09:08.874413 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:08.874388    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9973f1f-61b0-4d38-a7c2-b9c6fde9da94-istio-podinfo\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-tvbgw\" (UID: \"e9973f1f-61b0-4d38-a7c2-b9c6fde9da94\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-tvbgw"
Oct 30 19:09:08.912560 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:08.912517    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9973f1f-61b0-4d38-a7c2-b9c6fde9da94-istio-token\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-tvbgw\" (UID: \"e9973f1f-61b0-4d38-a7c2-b9c6fde9da94\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-tvbgw"
Oct 30 19:09:08.912819 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:08.912797    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jnx7\" (UniqueName: \"kubernetes.io/projected/e9973f1f-61b0-4d38-a7c2-b9c6fde9da94-kube-api-access-6jnx7\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-tvbgw\" (UID: \"e9973f1f-61b0-4d38-a7c2-b9c6fde9da94\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-tvbgw"
Oct 30 19:09:08.912896 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:08.912814    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/865afd2d-c5e2-4ee5-bfdb-303102de6616-istio-token\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-lkl5j\" (UID: \"865afd2d-c5e2-4ee5-bfdb-303102de6616\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-lkl5j"
Oct 30 19:09:08.915106 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:08.915074    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9973f1f-61b0-4d38-a7c2-b9c6fde9da94-istio-envoy\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-tvbgw\" (UID: \"e9973f1f-61b0-4d38-a7c2-b9c6fde9da94\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-tvbgw"
Oct 30 19:09:08.915380 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:08.915330    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlkhf\" (UniqueName: \"kubernetes.io/projected/865afd2d-c5e2-4ee5-bfdb-303102de6616-kube-api-access-dlkhf\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-lkl5j\" (UID: \"865afd2d-c5e2-4ee5-bfdb-303102de6616\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-lkl5j"
Oct 30 19:09:08.921341 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:08.921283    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/865afd2d-c5e2-4ee5-bfdb-303102de6616-istio-envoy\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-lkl5j\" (UID: \"865afd2d-c5e2-4ee5-bfdb-303102de6616\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-lkl5j"
Oct 30 19:09:09.010396 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:09.010293    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:09:09.010504 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:09.010459    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:09:09.010549 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:09.010532    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:09:09.010591 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:09.010576    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:09:09.011062 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:09.010360    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:09:09.011062 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:09.010702    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:09:09.012257 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:09.012173    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-t8s45"]
Oct 30 19:09:09.017766 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:09.017743    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-7ttp9"]
Oct 30 19:09:09.054864 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:09.054832    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-tvbgw"
Oct 30 19:09:09.137189 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:09.136977    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-lkl5j"
Oct 30 19:09:09.333953 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:09.323417    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-7ttp9" event={"ID":"960cce33-8574-42ab-8d16-c4fb2f0b3202","Type":"ContainerStarted","Data":"751fa8df5468e8cb96d391b1299dffa22a214e78bf156730654cd06fe3c2700a"}
Oct 30 19:09:09.333953 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:09.323486    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-7ttp9" event={"ID":"960cce33-8574-42ab-8d16-c4fb2f0b3202","Type":"ContainerStarted","Data":"8828279bae189514301e4d950df713ca0d1a177cd42dcbbb8e15b469a3049701"}
Oct 30 19:09:09.339539 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:09.338452    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-t8s45" event={"ID":"06b7dec0-c44b-4aea-a753-53ec832dfd24","Type":"ContainerStarted","Data":"a54b4418f6fea83f1a4e0b6cbd3057eeda100829b0edf905d2215e1d3c909fca"}
Oct 30 19:09:09.339539 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:09.338519    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-t8s45" event={"ID":"06b7dec0-c44b-4aea-a753-53ec832dfd24","Type":"ContainerStarted","Data":"5bd2b9f381a44f100acebf19497d7987aa65a38205e463b06c661867519d8e45"}
Oct 30 19:09:09.417768 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:09.384626    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-7ttp9" podStartSLOduration=3.384602631 podStartE2EDuration="3.384602631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:09:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:09:09.365218374 +0000 UTC m=+1971.990888342" watchObservedRunningTime="2026-10-30 19:09:09.384602631 +0000 UTC m=+1972.010272600"
Oct 30 19:09:09.441346 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:09.440916    4525 patch_prober.go:29] interesting pod/e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body=
Oct 30 19:09:09.441346 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:09.440971    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4" podUID="0650ae0c-36c1-4b6e-9212-12264a92c920" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused"
Oct 30 19:09:09.470133 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:09.454642    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-t8s45" podStartSLOduration=3.454622289 podStartE2EDuration="3.454622289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:09:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:09:09.418158695 +0000 UTC m=+1972.043828655" watchObservedRunningTime="2026-10-30 19:09:09.454622289 +0000 UTC m=+1972.080292253"
Oct 30 19:09:09.670319 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:09.669858    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-7ttp9"
Oct 30 19:09:09.727341 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:09.726609    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-t8s45"
Oct 30 19:09:09.822662 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:09:09.822604    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9973f1f_61b0_4d38_a7c2_b9c6fde9da94.slice/crio-6fc7abc9c57392b378b758f895740cd9cdc0d5acbad62fdc2bf22eb0a7e62fb5 WatchSource:0}: Error finding container 6fc7abc9c57392b378b758f895740cd9cdc0d5acbad62fdc2bf22eb0a7e62fb5: Status 404 returned error can't find the container with id 6fc7abc9c57392b378b758f895740cd9cdc0d5acbad62fdc2bf22eb0a7e62fb5
Oct 30 19:09:09.825294 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:09.825247    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-tvbgw"]
Oct 30 19:09:09.854028 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:09.852098    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:09:09.854028 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:09.852191    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:09:09.854028 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:09.852232    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:09:09.939076 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:09.939036    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-lkl5j"]
Oct 30 19:09:10.219716 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:10.219158    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:09:10.219716 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:10.219254    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:09:10.219716 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:10.219306    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:09:10.443976 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:10.443923    4525 patch_prober.go:29] interesting pod/e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body=
Oct 30 19:09:10.444206 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:10.443995    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4" podUID="0650ae0c-36c1-4b6e-9212-12264a92c920" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused"
Oct 30 19:09:10.469352 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:10.469167    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-lkl5j" event={"ID":"865afd2d-c5e2-4ee5-bfdb-303102de6616","Type":"ContainerStarted","Data":"b3343dfa8dd428fec24596c5600c4b9f15907f3f540f62d1bf897cc5514734a0"}
Oct 30 19:09:10.522708 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:10.512571    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-tvbgw" event={"ID":"e9973f1f-61b0-4d38-a7c2-b9c6fde9da94","Type":"ContainerStarted","Data":"6fc7abc9c57392b378b758f895740cd9cdc0d5acbad62fdc2bf22eb0a7e62fb5"}
Oct 30 19:09:10.751078 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:10.750995    4525 patch_prober.go:29] interesting pod/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-t8s45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body=
Oct 30 19:09:10.751352 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:10.751318    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-t8s45" podUID="06b7dec0-c44b-4aea-a753-53ec832dfd24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused"
Oct 30 19:09:10.751499 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:10.751424    4525 patch_prober.go:29] interesting pod/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-7ttp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" start-of-body=
Oct 30 19:09:10.751608 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:10.751593    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-7ttp9" podUID="960cce33-8574-42ab-8d16-c4fb2f0b3202" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused"
Oct 30 19:09:10.760715 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:10.760115    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c164d6-j5b5h"]
Oct 30 19:09:10.861432 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:10.859904    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c164d6-j5b5h"]
Oct 30 19:09:10.861432 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:10.860046    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c164d6-j5b5h"
Oct 30 19:09:10.888130 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:10.887298    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94hcn\""
Oct 30 19:09:10.888130 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:10.888089    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:09:10.950831 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:10.950141    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29e47eb5-9198-4675-8577-215209bdf4f6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-c164d6-j5b5h\" (UID: \"29e47eb5-9198-4675-8577-215209bdf4f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c164d6-j5b5h"
Oct 30 19:09:10.950831 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:10.950200    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29e47eb5-9198-4675-8577-215209bdf4f6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-c164d6-j5b5h\" (UID: \"29e47eb5-9198-4675-8577-215209bdf4f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c164d6-j5b5h"
Oct 30 19:09:10.950831 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:10.950402    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g45sh\" (UniqueName: \"kubernetes.io/projected/29e47eb5-9198-4675-8577-215209bdf4f6-kube-api-access-g45sh\") pod \"maas-api-revoke-keys-e2e-autores-c164d6-j5b5h\" (UID: \"29e47eb5-9198-4675-8577-215209bdf4f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c164d6-j5b5h"
Oct 30 19:09:11.063633 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:11.055024    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29e47eb5-9198-4675-8577-215209bdf4f6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-c164d6-j5b5h\" (UID: \"29e47eb5-9198-4675-8577-215209bdf4f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c164d6-j5b5h"
Oct 30 19:09:11.063633 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:11.055086    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29e47eb5-9198-4675-8577-215209bdf4f6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-c164d6-j5b5h\" (UID: \"29e47eb5-9198-4675-8577-215209bdf4f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c164d6-j5b5h"
Oct 30 19:09:11.063633 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:11.055281    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g45sh\" (UniqueName: \"kubernetes.io/projected/29e47eb5-9198-4675-8577-215209bdf4f6-kube-api-access-g45sh\") pod \"maas-api-revoke-keys-e2e-autores-c164d6-j5b5h\" (UID: \"29e47eb5-9198-4675-8577-215209bdf4f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c164d6-j5b5h"
Oct 30 19:09:11.069189 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:11.067095    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29e47eb5-9198-4675-8577-215209bdf4f6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-c164d6-j5b5h\" (UID: \"29e47eb5-9198-4675-8577-215209bdf4f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c164d6-j5b5h"
Oct 30 19:09:11.216409 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:11.214758    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g45sh\" (UniqueName: \"kubernetes.io/projected/29e47eb5-9198-4675-8577-215209bdf4f6-kube-api-access-g45sh\") pod \"maas-api-revoke-keys-e2e-autores-c164d6-j5b5h\" (UID: \"29e47eb5-9198-4675-8577-215209bdf4f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c164d6-j5b5h"
Oct 30 19:09:11.221230 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:11.221119    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29e47eb5-9198-4675-8577-215209bdf4f6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-c164d6-j5b5h\" (UID: \"29e47eb5-9198-4675-8577-215209bdf4f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c164d6-j5b5h"
Oct 30 19:09:11.477765 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:11.472276    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c164d6-j5b5h"
Oct 30 19:09:11.477765 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:11.473415    4525 patch_prober.go:29] interesting pod/e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body=
Oct 30 19:09:11.477765 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:11.473491    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4" podUID="0650ae0c-36c1-4b6e-9212-12264a92c920" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused"
Oct 30 19:09:11.683773 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:11.666499    4525 patch_prober.go:29] interesting pod/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-7ttp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" start-of-body=
Oct 30 19:09:11.683773 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:11.666556    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-7ttp9" podUID="960cce33-8574-42ab-8d16-c4fb2f0b3202" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused"
Oct 30 19:09:11.723381 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:11.723347    4525 patch_prober.go:29] interesting pod/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-t8s45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body=
Oct 30 19:09:11.751005 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:11.723661    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-t8s45" podUID="06b7dec0-c44b-4aea-a753-53ec832dfd24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused"
Oct 30 19:09:11.919881 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:11.816115    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-tvbgw" podStartSLOduration=3.816093456 podStartE2EDuration="3.816093456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:09:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:09:11.752229261 +0000 UTC m=+1974.377899226" watchObservedRunningTime="2026-10-30 19:09:11.816093456 +0000 UTC m=+1974.441763421"
Oct 30 19:09:12.116087 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:12.116042    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-tvbgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body=
Oct 30 19:09:12.116499 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:12.116450    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-tvbgw" podUID="e9973f1f-61b0-4d38-a7c2-b9c6fde9da94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused"
Oct 30 19:09:12.116865 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:12.116831    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv" podUID="595230e3-55f3-43eb-9a7b-6d89250af9b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:09:12.134859 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:12.120286    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-tvbgw"
Oct 30 19:09:12.137563 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:12.137508    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-tvbgw" event={"ID":"e9973f1f-61b0-4d38-a7c2-b9c6fde9da94","Type":"ContainerStarted","Data":"b0d994c2d6d9338b52f452d8fc13f4c468c9af8dccd52cca1fda8baab62cb723"}
Oct 30 19:09:12.137751 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:12.137578    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-lkl5j" event={"ID":"865afd2d-c5e2-4ee5-bfdb-303102de6616","Type":"ContainerStarted","Data":"94fdd2bda395afacbf6ccac28c4a3256280c56c1e509c8f03220ecbe8e53a885"}
Oct 30 19:09:12.152736 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:12.140747    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-lkl5j"
Oct 30 19:09:12.153163 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:12.152924    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t" podUID="424fca14-6d8a-4663-86ec-0c91165f34ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:09:12.153453 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:12.153335    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-lkl5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" start-of-body=
Oct 30 19:09:12.153453 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:12.153378    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-lkl5j" podUID="865afd2d-c5e2-4ee5-bfdb-303102de6616" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused"
Oct 30 19:09:12.154176 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:12.154118    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-lkl5j" podStartSLOduration=4.154101252 podStartE2EDuration="4.154101252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:09:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:09:12.017727972 +0000 UTC m=+1974.643397929" watchObservedRunningTime="2026-10-30 19:09:12.154101252 +0000 UTC m=+1974.779771217"
Oct 30 19:09:12.242361 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:12.216767    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj" podUID="84c99c49-1d60-4176-bda4-cad29c7d801c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 19:09:12.392888 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:12.392781    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw" podUID="ac6852fd-01be-4d69-8c9b-0c591e50b889" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:09:12.394456 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:12.394423    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb" podUID="12ebc8b9-c794-47b5-b87b-a63c4bd993e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:09:12.442154 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:12.442112    4525 patch_prober.go:29] interesting pod/e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body=
Oct 30 19:09:12.442507 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:12.442475    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4" podUID="0650ae0c-36c1-4b6e-9212-12264a92c920" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused"
Oct 30 19:09:12.666175 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:12.665782    4525 patch_prober.go:29] interesting pod/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-7ttp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" start-of-body=
Oct 30 19:09:12.666175 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:12.665830    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-7ttp9" podUID="960cce33-8574-42ab-8d16-c4fb2f0b3202" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused"
Oct 30 19:09:12.707577 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:12.705133    4525 patch_prober.go:29] interesting pod/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-t8s45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body=
Oct 30 19:09:12.707577 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:12.705205    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-t8s45" podUID="06b7dec0-c44b-4aea-a753-53ec832dfd24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused"
Oct 30 19:09:13.116769 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:13.067142    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-tvbgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body=
Oct 30 19:09:13.117139 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:13.117096    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-tvbgw" podUID="e9973f1f-61b0-4d38-a7c2-b9c6fde9da94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused"
Oct 30 19:09:13.214847 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:13.185396    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-lkl5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" start-of-body=
Oct 30 19:09:13.214847 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:13.185456    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-lkl5j" podUID="865afd2d-c5e2-4ee5-bfdb-303102de6616" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused"
Oct 30 19:09:13.372959 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:13.313769    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-75qr6"]
Oct 30 19:09:13.372959 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:13.332071    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-75qr6"]
Oct 30 19:09:13.372959 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:13.332121    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c164d6-j5b5h"]
Oct 30 19:09:13.372959 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:13.332236    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-75qr6"
Oct 30 19:09:13.473708 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:13.473459    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac875fe9-dc7c-46fe-9843-3e7bc6ad473f-istio-envoy\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-75qr6\" (UID: \"ac875fe9-dc7c-46fe-9843-3e7bc6ad473f\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-75qr6"
Oct 30 19:09:13.473708 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:13.473567    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac875fe9-dc7c-46fe-9843-3e7bc6ad473f-istio-podinfo\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-75qr6\" (UID: \"ac875fe9-dc7c-46fe-9843-3e7bc6ad473f\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-75qr6"
Oct 30 19:09:13.473708 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:13.473605    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2z44\" (UniqueName: \"kubernetes.io/projected/ac875fe9-dc7c-46fe-9843-3e7bc6ad473f-kube-api-access-k2z44\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-75qr6\" (UID: \"ac875fe9-dc7c-46fe-9843-3e7bc6ad473f\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-75qr6"
Oct 30 19:09:13.532710 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:13.477186    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac875fe9-dc7c-46fe-9843-3e7bc6ad473f-workload-certs\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-75qr6\" (UID: \"ac875fe9-dc7c-46fe-9843-3e7bc6ad473f\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-75qr6"
Oct 30 19:09:13.532710 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:13.477350    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac875fe9-dc7c-46fe-9843-3e7bc6ad473f-istiod-ca-cert\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-75qr6\" (UID: \"ac875fe9-dc7c-46fe-9843-3e7bc6ad473f\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-75qr6"
Oct 30 19:09:13.532710 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:13.477404    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac875fe9-dc7c-46fe-9843-3e7bc6ad473f-istio-data\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-75qr6\" (UID: \"ac875fe9-dc7c-46fe-9843-3e7bc6ad473f\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-75qr6"
Oct 30 19:09:13.532710 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:13.477492    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac875fe9-dc7c-46fe-9843-3e7bc6ad473f-workload-socket\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-75qr6\" (UID: \"ac875fe9-dc7c-46fe-9843-3e7bc6ad473f\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-75qr6"
Oct 30 19:09:13.532710 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:13.477606    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac875fe9-dc7c-46fe-9843-3e7bc6ad473f-credential-socket\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-75qr6\" (UID: \"ac875fe9-dc7c-46fe-9843-3e7bc6ad473f\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-75qr6"
Oct 30 19:09:13.622728 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:13.622667    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac875fe9-dc7c-46fe-9843-3e7bc6ad473f-istio-token\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-75qr6\" (UID: \"ac875fe9-dc7c-46fe-9843-3e7bc6ad473f\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-75qr6"
Oct 30 19:09:13.670493 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:13.667716    4525 patch_prober.go:29] interesting pod/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-7ttp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" start-of-body=
Oct 30 19:09:13.670493 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:13.667770    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-7ttp9" podUID="960cce33-8574-42ab-8d16-c4fb2f0b3202" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused"
Oct 30 19:09:13.777710 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:13.752808    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac875fe9-dc7c-46fe-9843-3e7bc6ad473f-istiod-ca-cert\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-75qr6\" (UID: \"ac875fe9-dc7c-46fe-9843-3e7bc6ad473f\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-75qr6"
Oct 30 19:09:13.780390 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:13.779482    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac875fe9-dc7c-46fe-9843-3e7bc6ad473f-istio-data\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-75qr6\" (UID: \"ac875fe9-dc7c-46fe-9843-3e7bc6ad473f\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-75qr6"
Oct 30 19:09:13.780390 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:13.779599    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac875fe9-dc7c-46fe-9843-3e7bc6ad473f-workload-socket\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-75qr6\" (UID: \"ac875fe9-dc7c-46fe-9843-3e7bc6ad473f\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-75qr6"
Oct 30 19:09:13.780390 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:13.779751    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac875fe9-dc7c-46fe-9843-3e7bc6ad473f-credential-socket\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-75qr6\" (UID: \"ac875fe9-dc7c-46fe-9843-3e7bc6ad473f\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-75qr6"
Oct 30 19:09:13.780390 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:13.779852    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac875fe9-dc7c-46fe-9843-3e7bc6ad473f-istio-token\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-75qr6\" (UID: \"ac875fe9-dc7c-46fe-9843-3e7bc6ad473f\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-75qr6"
Oct 30 19:09:13.780390 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:13.779951    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac875fe9-dc7c-46fe-9843-3e7bc6ad473f-istiod-ca-cert\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-75qr6\" (UID: \"ac875fe9-dc7c-46fe-9843-3e7bc6ad473f\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-75qr6"
Oct 30 19:09:13.780390 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:13.779980    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac875fe9-dc7c-46fe-9843-3e7bc6ad473f-istio-envoy\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-75qr6\" (UID: \"ac875fe9-dc7c-46fe-9843-3e7bc6ad473f\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-75qr6"
Oct 30 19:09:13.780390 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:13.780114    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac875fe9-dc7c-46fe-9843-3e7bc6ad473f-istio-podinfo\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-75qr6\" (UID: \"ac875fe9-dc7c-46fe-9843-3e7bc6ad473f\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-75qr6"
Oct 30 19:09:13.780390 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:13.780155    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2z44\" (UniqueName: \"kubernetes.io/projected/ac875fe9-dc7c-46fe-9843-3e7bc6ad473f-kube-api-access-k2z44\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-75qr6\" (UID: \"ac875fe9-dc7c-46fe-9843-3e7bc6ad473f\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-75qr6"
Oct 30 19:09:13.780390 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:13.780255    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac875fe9-dc7c-46fe-9843-3e7bc6ad473f-workload-certs\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-75qr6\" (UID: \"ac875fe9-dc7c-46fe-9843-3e7bc6ad473f\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-75qr6"
Oct 30 19:09:13.805275 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:13.805238    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac875fe9-dc7c-46fe-9843-3e7bc6ad473f-workload-certs\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-75qr6\" (UID: \"ac875fe9-dc7c-46fe-9843-3e7bc6ad473f\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-75qr6"
Oct 30 19:09:13.834400 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:13.830819    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4"
Oct 30 19:09:13.834400 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:13.832649    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4"
Oct 30 19:09:13.834400 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:13.832783    4525 patch_prober.go:29] interesting pod/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-t8s45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body=
Oct 30 19:09:13.834400 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:13.832823    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-t8s45" podUID="06b7dec0-c44b-4aea-a753-53ec832dfd24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused"
Oct 30 19:09:13.834862 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:13.834427    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac875fe9-dc7c-46fe-9843-3e7bc6ad473f-istio-data\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-75qr6\" (UID: \"ac875fe9-dc7c-46fe-9843-3e7bc6ad473f\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-75qr6"
Oct 30 19:09:13.834862 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:13.834714    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac875fe9-dc7c-46fe-9843-3e7bc6ad473f-workload-socket\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-75qr6\" (UID: \"ac875fe9-dc7c-46fe-9843-3e7bc6ad473f\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-75qr6"
Oct 30 19:09:13.834993 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:13.834941    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac875fe9-dc7c-46fe-9843-3e7bc6ad473f-credential-socket\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-75qr6\" (UID: \"ac875fe9-dc7c-46fe-9843-3e7bc6ad473f\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-75qr6"
Oct 30 19:09:13.873366 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:13.858901    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4"
Oct 30 19:09:14.355706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:14.346892    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac875fe9-dc7c-46fe-9843-3e7bc6ad473f-istio-envoy\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-75qr6\" (UID: \"ac875fe9-dc7c-46fe-9843-3e7bc6ad473f\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-75qr6"
Oct 30 19:09:14.372128 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:14.372086    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-tvbgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body=
Oct 30 19:09:14.372397 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:14.372368    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-tvbgw" podUID="e9973f1f-61b0-4d38-a7c2-b9c6fde9da94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused"
Oct 30 19:09:14.539389 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:14.382983    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb" podUID="12ebc8b9-c794-47b5-b87b-a63c4bd993e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 19:09:14.539389 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:14.386623    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw" podUID="ac6852fd-01be-4d69-8c9b-0c591e50b889" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:09:14.539389 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:14.454463    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-lkl5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" start-of-body=
Oct 30 19:09:14.539389 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:14.454560    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-lkl5j" podUID="865afd2d-c5e2-4ee5-bfdb-303102de6616" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused"
Oct 30 19:09:14.831762 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:14.808409    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac875fe9-dc7c-46fe-9843-3e7bc6ad473f-istio-podinfo\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-75qr6\" (UID: \"ac875fe9-dc7c-46fe-9843-3e7bc6ad473f\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-75qr6"
Oct 30 19:09:14.831762 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:14.828182    4525 patch_prober.go:29] interesting pod/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-7ttp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" start-of-body=
Oct 30 19:09:14.831762 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:14.828268    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-7ttp9" podUID="960cce33-8574-42ab-8d16-c4fb2f0b3202" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused"
Oct 30 19:09:15.199235 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:15.166976    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv" podUID="595230e3-55f3-43eb-9a7b-6d89250af9b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 19:09:15.199235 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:15.168007    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-tvbgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body=
Oct 30 19:09:15.199235 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:15.168067    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-tvbgw" podUID="e9973f1f-61b0-4d38-a7c2-b9c6fde9da94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused"
Oct 30 19:09:15.199235 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:15.175506    4525 patch_prober.go:29] interesting pod/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-t8s45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body=
Oct 30 19:09:15.264325 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:15.251895    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-t8s45" podUID="06b7dec0-c44b-4aea-a753-53ec832dfd24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused"
Oct 30 19:09:15.264325 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:15.232162    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2z44\" (UniqueName: \"kubernetes.io/projected/ac875fe9-dc7c-46fe-9843-3e7bc6ad473f-kube-api-access-k2z44\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-75qr6\" (UID: \"ac875fe9-dc7c-46fe-9843-3e7bc6ad473f\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-75qr6"
Oct 30 19:09:15.264325 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:15.263228    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac875fe9-dc7c-46fe-9843-3e7bc6ad473f-istio-token\") pod \"e2e-autores-c164d6-openshift-default-74c7bf49f9-75qr6\" (UID: \"ac875fe9-dc7c-46fe-9843-3e7bc6ad473f\") " pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-75qr6"
Oct 30 19:09:15.352926 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:15.352814    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t" podUID="424fca14-6d8a-4663-86ec-0c91165f34ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 19:09:15.387715 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:15.370389    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj" podUID="84c99c49-1d60-4176-bda4-cad29c7d801c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:09:15.485308 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:15.460069    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw" podUID="ac6852fd-01be-4d69-8c9b-0c591e50b889" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:09:15.517797 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:15.482161    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr" podUID="82bbbf85-ed93-48d2-9f17-49ac1f31c0a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:09:15.517797 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:15.515127    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw"
Oct 30 19:09:15.517797 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:15.482335    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-lkl5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" start-of-body=
Oct 30 19:09:15.517797 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:15.515218    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-lkl5j" podUID="865afd2d-c5e2-4ee5-bfdb-303102de6616" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused"
Oct 30 19:09:15.584738 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:15.527097    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-75qr6"
Oct 30 19:09:15.584738 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:15.550134    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts" podUID="be6d4969-285b-4da0-883e-4847f2cbf3af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:09:15.584738 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:15.554139    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb" podUID="12ebc8b9-c794-47b5-b87b-a63c4bd993e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:09:16.024767 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:15.993946    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zgw99" podUID="72bab614-8d2f-4640-9d16-b54f05759572" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 30 19:09:16.101813 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:16.101775    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-tvbgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body=
Oct 30 19:09:16.102123 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:16.102091    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-tvbgw" podUID="e9973f1f-61b0-4d38-a7c2-b9c6fde9da94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused"
Oct 30 19:09:16.169931 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:16.169881    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv" podUID="595230e3-55f3-43eb-9a7b-6d89250af9b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:09:16.219649 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:16.195467    4525 patch_prober.go:29] interesting pod/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-7ttp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" start-of-body=
Oct 30 19:09:16.219649 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:16.195529    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-7ttp9" podUID="960cce33-8574-42ab-8d16-c4fb2f0b3202" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused"
Oct 30 19:09:16.219649 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:16.215716    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-lkl5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" start-of-body=
Oct 30 19:09:16.219649 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:16.215770    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-lkl5j" podUID="865afd2d-c5e2-4ee5-bfdb-303102de6616" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused"
Oct 30 19:09:16.247856 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:16.247818    4525 patch_prober.go:29] interesting pod/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-t8s45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body=
Oct 30 19:09:16.248122 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:16.248092    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-t8s45" podUID="06b7dec0-c44b-4aea-a753-53ec832dfd24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused"
Oct 30 19:09:16.387825 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:16.386765    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t" podUID="424fca14-6d8a-4663-86ec-0c91165f34ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:09:16.439513 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:16.432211    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb"
Oct 30 19:09:16.439513 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:16.432285    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv"
Oct 30 19:09:16.439513 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:16.432324    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t"
Oct 30 19:09:16.545708 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:16.535005    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw" podUID="ac6852fd-01be-4d69-8c9b-0c591e50b889" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 19:09:16.545708 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:16.535320    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr" podUID="82bbbf85-ed93-48d2-9f17-49ac1f31c0a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 30 19:09:16.545708 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:16.535570    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr"
Oct 30 19:09:16.554787 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:16.551576    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj" podUID="84c99c49-1d60-4176-bda4-cad29c7d801c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:09:16.554787 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:16.554775    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts" podUID="be6d4969-285b-4da0-883e-4847f2cbf3af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Oct 30 19:09:16.597847 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:16.597806    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb" podUID="12ebc8b9-c794-47b5-b87b-a63c4bd993e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:09:16.683947 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:16.683916    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj"
Oct 30 19:09:16.828125 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:16.820329    4525 patch_prober.go:29] interesting pod/e2e-autores-c164d6-openshift-default-74c7bf49f9-4dd4v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.227:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:09:16.908708 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:16.908087    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4dd4v" podUID="feea5f3c-8630-4e8f-8832-2998030bf2e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.227:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:09:16.922141 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:16.922109    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts"
Oct 30 19:09:17.053197 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:17.053103    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c164d6-j5b5h" event={"ID":"29e47eb5-9198-4675-8577-215209bdf4f6","Type":"ContainerStarted","Data":"9c87bd16ba60e6abd5101053d0758c820c95ab8acaa4152e041ca617ec813c36"}
Oct 30 19:09:17.053823 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:17.052219    4525 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb" podUID="12ebc8b9-c794-47b5-b87b-a63c4bd993e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:09:17.054145 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:17.050819    4525 patch_prober.go:29] interesting pod/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-7ttp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" start-of-body=
Oct 30 19:09:17.054300 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:17.054280    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-7ttp9" podUID="960cce33-8574-42ab-8d16-c4fb2f0b3202" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused"
Oct 30 19:09:17.056249 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:17.056226    4525 patch_prober.go:29] interesting pod/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-t8s45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body=
Oct 30 19:09:17.056443 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:17.056422    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-t8s45" podUID="06b7dec0-c44b-4aea-a753-53ec832dfd24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused"
Oct 30 19:09:17.086171 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:17.086137    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-tvbgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body=
Oct 30 19:09:17.089731 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:17.087858    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-tvbgw" podUID="e9973f1f-61b0-4d38-a7c2-b9c6fde9da94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused"
Oct 30 19:09:17.197964 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:17.194013    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-lkl5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" start-of-body=
Oct 30 19:09:17.197964 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:17.194101    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-lkl5j" podUID="865afd2d-c5e2-4ee5-bfdb-303102de6616" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused"
Oct 30 19:09:17.210999 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:17.210939    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv" podUID="595230e3-55f3-43eb-9a7b-6d89250af9b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:09:17.606296 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:17.604639    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t" podUID="424fca14-6d8a-4663-86ec-0c91165f34ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 19:09:17.611828 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:17.609430    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw" podUID="ac6852fd-01be-4d69-8c9b-0c591e50b889" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 19:09:17.611828 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:17.609507    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr" podUID="82bbbf85-ed93-48d2-9f17-49ac1f31c0a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 30 19:09:17.680666 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:17.680619    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj" podUID="84c99c49-1d60-4176-bda4-cad29c7d801c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:09:17.763521 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:17.763457    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts" podUID="be6d4969-285b-4da0-883e-4847f2cbf3af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Oct 30 19:09:17.840861 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:17.766611    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb" podUID="12ebc8b9-c794-47b5-b87b-a63c4bd993e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 19:09:17.944669 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:17.944638    4525 patch_prober.go:29] interesting pod/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-t8s45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body=
Oct 30 19:09:17.944901 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:17.944874    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-t8s45" podUID="06b7dec0-c44b-4aea-a753-53ec832dfd24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused"
Oct 30 19:09:17.989780 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:17.970067    4525 patch_prober.go:29] interesting pod/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-7ttp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" start-of-body=
Oct 30 19:09:17.989780 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:17.970149    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-7ttp9" podUID="960cce33-8574-42ab-8d16-c4fb2f0b3202" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused"
Oct 30 19:09:18.086503 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:18.086454    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-tvbgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body=
Oct 30 19:09:18.087069 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:18.087037    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-tvbgw" podUID="e9973f1f-61b0-4d38-a7c2-b9c6fde9da94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused"
Oct 30 19:09:18.409249 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:18.374517    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-lkl5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" start-of-body=
Oct 30 19:09:18.409249 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:18.374590    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-lkl5j" podUID="865afd2d-c5e2-4ee5-bfdb-303102de6616" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused"
Oct 30 19:09:18.668130 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:18.668026    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv" podUID="595230e3-55f3-43eb-9a7b-6d89250af9b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:09:18.881360 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:18.877055    4525 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj" podUID="84c99c49-1d60-4176-bda4-cad29c7d801c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:09:18.923523 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:09:18.881550    4525 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29e47eb5_9198_4675_8577_215209bdf4f6.slice/crio-151aa2cef83566faae57d2fd168e9f29169251433ebed50ddd24f24c792e7b8f.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29e47eb5_9198_4675_8577_215209bdf4f6.slice/crio-conmon-151aa2cef83566faae57d2fd168e9f29169251433ebed50ddd24f24c792e7b8f.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 19:09:18.923523 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:09:18.884508    4525 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.047s"
Oct 30 19:09:18.923523 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:18.923489    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c164d6-j5b5h" event={"ID":"29e47eb5-9198-4675-8577-215209bdf4f6","Type":"ContainerStarted","Data":"151aa2cef83566faae57d2fd168e9f29169251433ebed50ddd24f24c792e7b8f"}
Oct 30 19:09:18.923851 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:18.923630    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-99fmr"]
Oct 30 19:09:18.940833 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:18.888563    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:09:18.940833 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:18.924424    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:09:18.940833 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:18.924569    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:09:18.940833 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:18.924766    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb" podUID="12ebc8b9-c794-47b5-b87b-a63c4bd993e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:09:18.940833 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:18.898363    4525 patch_prober.go:29] interesting pod/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-t8s45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body=
Oct 30 19:09:18.940833 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:18.925111    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-t8s45" podUID="06b7dec0-c44b-4aea-a753-53ec832dfd24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused"
Oct 30 19:09:18.957436 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:18.957013    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t"
Oct 30 19:09:18.957436 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:18.957061    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-99fmr"]
Oct 30 19:09:18.957436 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:18.957197    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-7ttp9"
Oct 30 19:09:18.959388 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:18.957959    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw" podUID="ac6852fd-01be-4d69-8c9b-0c591e50b889" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:09:18.959388 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:18.958366    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-99fmr"
Oct 30 19:09:18.965435 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:18.963042    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr" podUID="82bbbf85-ed93-48d2-9f17-49ac1f31c0a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:09:18.965435 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:18.963321    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj" podUID="84c99c49-1d60-4176-bda4-cad29c7d801c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:09:18.965435 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:18.963512    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts" podUID="be6d4969-285b-4da0-883e-4847f2cbf3af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:09:18.965435 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:18.963950    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-7ttp9"
Oct 30 19:09:19.014433 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:19.014401    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb"
Oct 30 19:09:19.015197 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:19.015176    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-7ttp9"
Oct 30 19:09:19.033343 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:19.033319    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv"
Oct 30 19:09:19.069188 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:19.066947    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-tvbgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body=
Oct 30 19:09:19.069188 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:19.067013    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-tvbgw" podUID="e9973f1f-61b0-4d38-a7c2-b9c6fde9da94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused"
Oct 30 19:09:19.155702 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:19.155650    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be563a8d-e16a-4b6a-b82d-4d6c4847a527-istiod-ca-cert\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-99fmr\" (UID: \"be563a8d-e16a-4b6a-b82d-4d6c4847a527\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-99fmr"
Oct 30 19:09:19.159647 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:19.159613    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be563a8d-e16a-4b6a-b82d-4d6c4847a527-credential-socket\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-99fmr\" (UID: \"be563a8d-e16a-4b6a-b82d-4d6c4847a527\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-99fmr"
Oct 30 19:09:19.159811 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:19.159672    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwlwk\" (UniqueName: \"kubernetes.io/projected/be563a8d-e16a-4b6a-b82d-4d6c4847a527-kube-api-access-lwlwk\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-99fmr\" (UID: \"be563a8d-e16a-4b6a-b82d-4d6c4847a527\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-99fmr"
Oct 30 19:09:19.159969 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:19.159949    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be563a8d-e16a-4b6a-b82d-4d6c4847a527-istio-podinfo\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-99fmr\" (UID: \"be563a8d-e16a-4b6a-b82d-4d6c4847a527\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-99fmr"
Oct 30 19:09:19.160162 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:19.160143    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be563a8d-e16a-4b6a-b82d-4d6c4847a527-istio-envoy\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-99fmr\" (UID: \"be563a8d-e16a-4b6a-b82d-4d6c4847a527\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-99fmr"
Oct 30 19:09:19.160362 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:19.160345    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be563a8d-e16a-4b6a-b82d-4d6c4847a527-workload-certs\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-99fmr\" (UID: \"be563a8d-e16a-4b6a-b82d-4d6c4847a527\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-99fmr"
Oct 30 19:09:19.160489 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:19.160475    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be563a8d-e16a-4b6a-b82d-4d6c4847a527-istio-token\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-99fmr\" (UID: \"be563a8d-e16a-4b6a-b82d-4d6c4847a527\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-99fmr"
Oct 30 19:09:19.163234 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:19.158805    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-lkl5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" start-of-body=
Oct 30 19:09:19.172373 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:19.172080    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-lkl5j" podUID="865afd2d-c5e2-4ee5-bfdb-303102de6616" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused"
Oct 30 19:09:19.172869 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:19.165918    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw"
Oct 30 19:09:19.173327 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:19.164039    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be563a8d-e16a-4b6a-b82d-4d6c4847a527-istio-data\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-99fmr\" (UID: \"be563a8d-e16a-4b6a-b82d-4d6c4847a527\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-99fmr"
Oct 30 19:09:19.173327 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:19.173144    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be563a8d-e16a-4b6a-b82d-4d6c4847a527-workload-socket\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-99fmr\" (UID: \"be563a8d-e16a-4b6a-b82d-4d6c4847a527\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-99fmr"
Oct 30 19:09:19.173746 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:19.173666    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr"
Oct 30 19:09:19.173912 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:19.173890    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj"
Oct 30 19:09:19.174056 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:19.174043    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts"
Oct 30 19:09:19.237143 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:19.235997    4525 patch_prober.go:29] interesting pod/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.226:15021/healthz/ready\": read tcp 10.133.0.2:56482->10.133.0.226:15021: read: connection reset by peer" start-of-body=
Oct 30 19:09:19.237143 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:19.236061    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7" podUID="77adecb1-af53-4dd9-bb52-d8e22ebb03e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.226:15021/healthz/ready\": read tcp 10.133.0.2:56482->10.133.0.226:15021: read: connection reset by peer"
Oct 30 19:09:19.300778 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:09:19.274095    4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d49l" containerName="extractor"
Oct 30 19:09:19.300778 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:19.274177    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-75qr6"]
Oct 30 19:09:19.326069 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:19.302729    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwlwk\" (UniqueName: \"kubernetes.io/projected/be563a8d-e16a-4b6a-b82d-4d6c4847a527-kube-api-access-lwlwk\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-99fmr\" (UID: \"be563a8d-e16a-4b6a-b82d-4d6c4847a527\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-99fmr"
Oct 30 19:09:19.326069 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:19.303070    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be563a8d-e16a-4b6a-b82d-4d6c4847a527-istio-podinfo\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-99fmr\" (UID: \"be563a8d-e16a-4b6a-b82d-4d6c4847a527\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-99fmr"
Oct 30 19:09:19.326069 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:19.303112    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be563a8d-e16a-4b6a-b82d-4d6c4847a527-istio-envoy\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-99fmr\" (UID: \"be563a8d-e16a-4b6a-b82d-4d6c4847a527\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-99fmr"
Oct 30 19:09:19.326069 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:19.303196    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be563a8d-e16a-4b6a-b82d-4d6c4847a527-workload-certs\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-99fmr\" (UID: \"be563a8d-e16a-4b6a-b82d-4d6c4847a527\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-99fmr"
Oct 30 19:09:19.326069 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:19.303229    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be563a8d-e16a-4b6a-b82d-4d6c4847a527-istio-token\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-99fmr\" (UID: \"be563a8d-e16a-4b6a-b82d-4d6c4847a527\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-99fmr"
Oct 30 19:09:19.326069 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:19.303393    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be563a8d-e16a-4b6a-b82d-4d6c4847a527-istio-data\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-99fmr\" (UID: \"be563a8d-e16a-4b6a-b82d-4d6c4847a527\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-99fmr"
Oct 30 19:09:19.326069 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:19.303456    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be563a8d-e16a-4b6a-b82d-4d6c4847a527-workload-socket\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-99fmr\" (UID: \"be563a8d-e16a-4b6a-b82d-4d6c4847a527\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-99fmr"
Oct 30 19:09:19.326069 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:19.303731    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be563a8d-e16a-4b6a-b82d-4d6c4847a527-istiod-ca-cert\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-99fmr\" (UID: \"be563a8d-e16a-4b6a-b82d-4d6c4847a527\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-99fmr"
Oct 30 19:09:19.326069 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:19.303794    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be563a8d-e16a-4b6a-b82d-4d6c4847a527-credential-socket\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-99fmr\" (UID: \"be563a8d-e16a-4b6a-b82d-4d6c4847a527\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-99fmr"
Oct 30 19:09:19.326069 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:19.304276    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be563a8d-e16a-4b6a-b82d-4d6c4847a527-credential-socket\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-99fmr\" (UID: \"be563a8d-e16a-4b6a-b82d-4d6c4847a527\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-99fmr"
Oct 30 19:09:19.346718 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:19.327376    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be563a8d-e16a-4b6a-b82d-4d6c4847a527-istio-data\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-99fmr\" (UID: \"be563a8d-e16a-4b6a-b82d-4d6c4847a527\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-99fmr"
Oct 30 19:09:19.346718 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:19.332302    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be563a8d-e16a-4b6a-b82d-4d6c4847a527-workload-certs\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-99fmr\" (UID: \"be563a8d-e16a-4b6a-b82d-4d6c4847a527\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-99fmr"
Oct 30 19:09:19.346718 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:19.342729    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c164d6-j5b5h" podStartSLOduration=9.342709222 podStartE2EDuration="9.342709222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:09:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:09:18.92288225 +0000 UTC m=+1981.548552227" watchObservedRunningTime="2026-10-30 19:09:19.342709222 +0000 UTC m=+1981.968379187"
Oct 30 19:09:19.370964 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:19.347590    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be563a8d-e16a-4b6a-b82d-4d6c4847a527-workload-socket\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-99fmr\" (UID: \"be563a8d-e16a-4b6a-b82d-4d6c4847a527\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-99fmr"
Oct 30 19:09:19.370964 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:19.348587    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be563a8d-e16a-4b6a-b82d-4d6c4847a527-istiod-ca-cert\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-99fmr\" (UID: \"be563a8d-e16a-4b6a-b82d-4d6c4847a527\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-99fmr"
Oct 30 19:09:19.436823 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:19.436708    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be563a8d-e16a-4b6a-b82d-4d6c4847a527-istio-podinfo\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-99fmr\" (UID: \"be563a8d-e16a-4b6a-b82d-4d6c4847a527\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-99fmr"
Oct 30 19:09:19.441413 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:19.441078    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwlwk\" (UniqueName: \"kubernetes.io/projected/be563a8d-e16a-4b6a-b82d-4d6c4847a527-kube-api-access-lwlwk\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-99fmr\" (UID: \"be563a8d-e16a-4b6a-b82d-4d6c4847a527\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-99fmr"
Oct 30 19:09:19.464570 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:19.464504    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be563a8d-e16a-4b6a-b82d-4d6c4847a527-istio-envoy\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-99fmr\" (UID: \"be563a8d-e16a-4b6a-b82d-4d6c4847a527\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-99fmr"
Oct 30 19:09:19.626704 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:19.590257    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be563a8d-e16a-4b6a-b82d-4d6c4847a527-istio-token\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-99fmr\" (UID: \"be563a8d-e16a-4b6a-b82d-4d6c4847a527\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-99fmr"
Oct 30 19:09:19.816145 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:19.816049    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-99fmr"
Oct 30 19:09:19.951233 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:19.950110    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-t8s45"
Oct 30 19:09:19.966938 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:19.966906    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-t8s45"
Oct 30 19:09:20.039512 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:20.012553    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-t8s45"
Oct 30 19:09:20.066344 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:20.066183    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-tvbgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body=
Oct 30 19:09:20.066344 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:20.066240    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-tvbgw" podUID="e9973f1f-61b0-4d38-a7c2-b9c6fde9da94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused"
Oct 30 19:09:20.162995 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:20.162958    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-lkl5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" start-of-body=
Oct 30 19:09:20.163584 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:20.163549    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-lkl5j" podUID="865afd2d-c5e2-4ee5-bfdb-303102de6616" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused"
Oct 30 19:09:20.173386 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:20.173356    4525 generic.go:356] "Generic (PLEG): container finished" podID="29e47eb5-9198-4675-8577-215209bdf4f6" containerID="151aa2cef83566faae57d2fd168e9f29169251433ebed50ddd24f24c792e7b8f" exitCode=0
Oct 30 19:09:20.343787 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:20.301759    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c164d6-j5b5h" event={"ID":"29e47eb5-9198-4675-8577-215209bdf4f6","Type":"ContainerDied","Data":"151aa2cef83566faae57d2fd168e9f29169251433ebed50ddd24f24c792e7b8f"}
Oct 30 19:09:20.634705 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:20.608600    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-75qr6" event={"ID":"ac875fe9-dc7c-46fe-9843-3e7bc6ad473f","Type":"ContainerStarted","Data":"7a274493869fc805f0d79df06201ceddfafd8fd4643890696265acd74b423832"}
Oct 30 19:09:20.722739 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:20.656781    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-75qr6" event={"ID":"ac875fe9-dc7c-46fe-9843-3e7bc6ad473f","Type":"ContainerStarted","Data":"f30698e41424c4bdf0e0db72e600329d3b47bca5ca3930272e8f34260c663091"}
Oct 30 19:09:20.922586 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:20.902440    4525 generic.go:356] "Generic (PLEG): container finished" podID="77adecb1-af53-4dd9-bb52-d8e22ebb03e4" containerID="052da6685f672c65b36233b5ccaf20546eb09e6184d1ec4d7da5fc1a377ff3ec" exitCode=0
Oct 30 19:09:20.922586 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:20.903260    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7" event={"ID":"77adecb1-af53-4dd9-bb52-d8e22ebb03e4","Type":"ContainerDied","Data":"052da6685f672c65b36233b5ccaf20546eb09e6184d1ec4d7da5fc1a377ff3ec"}
Oct 30 19:09:20.957018 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:20.956986    4525 scope.go:122] "RemoveContainer" containerID="052da6685f672c65b36233b5ccaf20546eb09e6184d1ec4d7da5fc1a377ff3ec"
Oct 30 19:09:20.967237 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:20.966891    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:09:20.967591 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:20.967562    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:09:20.967902 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:20.967877    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:09:21.213752 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:21.197268    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-75qr6" podStartSLOduration=8.197248274 podStartE2EDuration="8.197248274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:09:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:09:20.750263943 +0000 UTC m=+1983.375933915" watchObservedRunningTime="2026-10-30 19:09:21.197248274 +0000 UTC m=+1983.822918238"
Oct 30 19:09:21.215848 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:09:21.215781    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe563a8d_e16a_4b6a_b82d_4d6c4847a527.slice/crio-01c8aa1cc13d1ab9ad1df6ad4bd4371d7b645d515d04439c43665b57d9bb2eff WatchSource:0}: Error finding container 01c8aa1cc13d1ab9ad1df6ad4bd4371d7b645d515d04439c43665b57d9bb2eff: Status 404 returned error can't find the container with id 01c8aa1cc13d1ab9ad1df6ad4bd4371d7b645d515d04439c43665b57d9bb2eff
Oct 30 19:09:21.230511 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:21.226561    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:09:21.230511 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:21.226664    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:09:21.230511 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:21.226736    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:09:21.258437 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:21.253089    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-lkl5j"
Oct 30 19:09:21.258437 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:21.254826    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-lkl5j"
Oct 30 19:09:21.272741 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:21.272639    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-lkl5j"
Oct 30 19:09:21.300791 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:21.300720    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-tvbgw"
Oct 30 19:09:21.302663 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:21.302629    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-tvbgw"
Oct 30 19:09:21.304587 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:21.304561    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-tvbgw"
Oct 30 19:09:21.458702 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:21.458653    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-99fmr"]
Oct 30 19:09:21.575419 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:21.571625    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-75qr6"
Oct 30 19:09:21.588716 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:21.588068    4525 patch_prober.go:29] interesting pod/e2e-autores-c164d6-openshift-default-74c7bf49f9-75qr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused" start-of-body=
Oct 30 19:09:21.588716 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:21.588134    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-75qr6" podUID="ac875fe9-dc7c-46fe-9843-3e7bc6ad473f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused"
Oct 30 19:09:21.769862 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:21.726589    4525 scope.go:122] "RemoveContainer" containerID="1e6098c8b0278137a665f8f0773c7690ea43ff38814dedb95e00d956f5607b23"
Oct 30 19:09:22.397445 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:22.363502    4525 scope.go:122] "RemoveContainer" containerID="2aa93c230ceed97b517836d94bc90c1a0c08010b7abbf721f6871f85850bbee9"
Oct 30 19:09:22.397445 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:22.370831    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7" event={"ID":"77adecb1-af53-4dd9-bb52-d8e22ebb03e4","Type":"ContainerStarted","Data":"4249fc47d1098fc9d135027761840864157ed12000e934280f9bcd9b3ed2d5be"}
Oct 30 19:09:22.397445 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:22.376163    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-99fmr" event={"ID":"be563a8d-e16a-4b6a-b82d-4d6c4847a527","Type":"ContainerStarted","Data":"01c8aa1cc13d1ab9ad1df6ad4bd4371d7b645d515d04439c43665b57d9bb2eff"}
Oct 30 19:09:22.418807 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:22.418768    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-qlfgx"]
Oct 30 19:09:22.434888 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:22.434856    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-qlfgx"]
Oct 30 19:09:22.435072 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:22.434972    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-qlfgx"
Oct 30 19:09:22.555941 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:22.555886    4525 patch_prober.go:29] interesting pod/e2e-autores-c164d6-openshift-default-74c7bf49f9-75qr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused" start-of-body=
Oct 30 19:09:22.556108 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:22.555947    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-75qr6" podUID="ac875fe9-dc7c-46fe-9843-3e7bc6ad473f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused"
Oct 30 19:09:22.567157 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:22.567123    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a3d6a70-b055-40dc-98a7-d0f5890293c8-istiod-ca-cert\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-qlfgx\" (UID: \"9a3d6a70-b055-40dc-98a7-d0f5890293c8\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-qlfgx"
Oct 30 19:09:22.567305 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:22.567239    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a3d6a70-b055-40dc-98a7-d0f5890293c8-istio-token\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-qlfgx\" (UID: \"9a3d6a70-b055-40dc-98a7-d0f5890293c8\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-qlfgx"
Oct 30 19:09:22.567730 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:22.567491    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a3d6a70-b055-40dc-98a7-d0f5890293c8-workload-socket\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-qlfgx\" (UID: \"9a3d6a70-b055-40dc-98a7-d0f5890293c8\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-qlfgx"
Oct 30 19:09:22.567730 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:22.567594    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a3d6a70-b055-40dc-98a7-d0f5890293c8-workload-certs\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-qlfgx\" (UID: \"9a3d6a70-b055-40dc-98a7-d0f5890293c8\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-qlfgx"
Oct 30 19:09:22.589707 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:22.569017    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a3d6a70-b055-40dc-98a7-d0f5890293c8-istio-podinfo\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-qlfgx\" (UID: \"9a3d6a70-b055-40dc-98a7-d0f5890293c8\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-qlfgx"
Oct 30 19:09:22.589707 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:22.569172    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a3d6a70-b055-40dc-98a7-d0f5890293c8-credential-socket\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-qlfgx\" (UID: \"9a3d6a70-b055-40dc-98a7-d0f5890293c8\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-qlfgx"
Oct 30 19:09:22.589707 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:22.569216    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7dzb\" (UniqueName: \"kubernetes.io/projected/9a3d6a70-b055-40dc-98a7-d0f5890293c8-kube-api-access-v7dzb\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-qlfgx\" (UID: \"9a3d6a70-b055-40dc-98a7-d0f5890293c8\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-qlfgx"
Oct 30 19:09:22.589707 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:22.569421    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a3d6a70-b055-40dc-98a7-d0f5890293c8-istio-data\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-qlfgx\" (UID: \"9a3d6a70-b055-40dc-98a7-d0f5890293c8\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-qlfgx"
Oct 30 19:09:22.589707 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:22.569503    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a3d6a70-b055-40dc-98a7-d0f5890293c8-istio-envoy\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-qlfgx\" (UID: \"9a3d6a70-b055-40dc-98a7-d0f5890293c8\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-qlfgx"
Oct 30 19:09:22.767892 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:22.733643    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a3d6a70-b055-40dc-98a7-d0f5890293c8-istio-podinfo\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-qlfgx\" (UID: \"9a3d6a70-b055-40dc-98a7-d0f5890293c8\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-qlfgx"
Oct 30 19:09:22.767892 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:22.733816    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a3d6a70-b055-40dc-98a7-d0f5890293c8-credential-socket\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-qlfgx\" (UID: \"9a3d6a70-b055-40dc-98a7-d0f5890293c8\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-qlfgx"
Oct 30 19:09:22.767892 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:22.733881    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7dzb\" (UniqueName: \"kubernetes.io/projected/9a3d6a70-b055-40dc-98a7-d0f5890293c8-kube-api-access-v7dzb\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-qlfgx\" (UID: \"9a3d6a70-b055-40dc-98a7-d0f5890293c8\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-qlfgx"
Oct 30 19:09:22.767892 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:22.734004    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a3d6a70-b055-40dc-98a7-d0f5890293c8-istio-data\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-qlfgx\" (UID: \"9a3d6a70-b055-40dc-98a7-d0f5890293c8\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-qlfgx"
Oct 30 19:09:22.767892 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:22.734069    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a3d6a70-b055-40dc-98a7-d0f5890293c8-istio-envoy\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-qlfgx\" (UID: \"9a3d6a70-b055-40dc-98a7-d0f5890293c8\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-qlfgx"
Oct 30 19:09:22.767892 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:22.734142    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a3d6a70-b055-40dc-98a7-d0f5890293c8-istiod-ca-cert\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-qlfgx\" (UID: \"9a3d6a70-b055-40dc-98a7-d0f5890293c8\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-qlfgx"
Oct 30 19:09:22.767892 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:22.734208    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a3d6a70-b055-40dc-98a7-d0f5890293c8-istio-token\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-qlfgx\" (UID: \"9a3d6a70-b055-40dc-98a7-d0f5890293c8\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-qlfgx"
Oct 30 19:09:22.767892 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:22.734334    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a3d6a70-b055-40dc-98a7-d0f5890293c8-workload-socket\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-qlfgx\" (UID: \"9a3d6a70-b055-40dc-98a7-d0f5890293c8\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-qlfgx"
Oct 30 19:09:22.767892 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:22.734413    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a3d6a70-b055-40dc-98a7-d0f5890293c8-workload-certs\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-qlfgx\" (UID: \"9a3d6a70-b055-40dc-98a7-d0f5890293c8\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-qlfgx"
Oct 30 19:09:22.767892 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:22.744073    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a3d6a70-b055-40dc-98a7-d0f5890293c8-workload-socket\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-qlfgx\" (UID: \"9a3d6a70-b055-40dc-98a7-d0f5890293c8\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-qlfgx"
Oct 30 19:09:22.767892 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:22.752487    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a3d6a70-b055-40dc-98a7-d0f5890293c8-workload-certs\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-qlfgx\" (UID: \"9a3d6a70-b055-40dc-98a7-d0f5890293c8\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-qlfgx"
Oct 30 19:09:22.768941 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:22.768053    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a3d6a70-b055-40dc-98a7-d0f5890293c8-istiod-ca-cert\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-qlfgx\" (UID: \"9a3d6a70-b055-40dc-98a7-d0f5890293c8\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-qlfgx"
Oct 30 19:09:22.769881 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:22.769853    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a3d6a70-b055-40dc-98a7-d0f5890293c8-istio-data\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-qlfgx\" (UID: \"9a3d6a70-b055-40dc-98a7-d0f5890293c8\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-qlfgx"
Oct 30 19:09:22.832169 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:22.832113    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a3d6a70-b055-40dc-98a7-d0f5890293c8-credential-socket\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-qlfgx\" (UID: \"9a3d6a70-b055-40dc-98a7-d0f5890293c8\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-qlfgx"
Oct 30 19:09:22.872770 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:22.872695    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a3d6a70-b055-40dc-98a7-d0f5890293c8-istio-token\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-qlfgx\" (UID: \"9a3d6a70-b055-40dc-98a7-d0f5890293c8\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-qlfgx"
Oct 30 19:09:22.874630 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:22.874561    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7dzb\" (UniqueName: \"kubernetes.io/projected/9a3d6a70-b055-40dc-98a7-d0f5890293c8-kube-api-access-v7dzb\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-qlfgx\" (UID: \"9a3d6a70-b055-40dc-98a7-d0f5890293c8\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-qlfgx"
Oct 30 19:09:22.921196 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:22.912516    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a3d6a70-b055-40dc-98a7-d0f5890293c8-istio-envoy\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-qlfgx\" (UID: \"9a3d6a70-b055-40dc-98a7-d0f5890293c8\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-qlfgx"
Oct 30 19:09:22.963476 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:22.963440    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a3d6a70-b055-40dc-98a7-d0f5890293c8-istio-podinfo\") pod \"e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-qlfgx\" (UID: \"9a3d6a70-b055-40dc-98a7-d0f5890293c8\") " pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-qlfgx"
Oct 30 19:09:23.064296 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:23.064028    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-qlfgx"
Oct 30 19:09:23.148728 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:23.143479    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7"
Oct 30 19:09:23.148728 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:23.143630    4525 patch_prober.go:29] interesting pod/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" start-of-body=
Oct 30 19:09:23.148728 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:23.143695    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7" podUID="77adecb1-af53-4dd9-bb52-d8e22ebb03e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused"
Oct 30 19:09:23.158561 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:23.158190    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c164d6-j5b5h"
Oct 30 19:09:23.455972 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:23.432866    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29e47eb5-9198-4675-8577-215209bdf4f6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29e47eb5-9198-4675-8577-215209bdf4f6-maas-api-xks-ca\") pod \"29e47eb5-9198-4675-8577-215209bdf4f6\" (UID: \"29e47eb5-9198-4675-8577-215209bdf4f6\") "
Oct 30 19:09:23.455972 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:23.432957    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29e47eb5-9198-4675-8577-215209bdf4f6-kube-api-access-g45sh\" (UniqueName: \"kubernetes.io/projected/29e47eb5-9198-4675-8577-215209bdf4f6-kube-api-access-g45sh\") pod \"29e47eb5-9198-4675-8577-215209bdf4f6\" (UID: \"29e47eb5-9198-4675-8577-215209bdf4f6\") "
Oct 30 19:09:23.455972 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:23.433231    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29e47eb5-9198-4675-8577-215209bdf4f6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29e47eb5-9198-4675-8577-215209bdf4f6-maas-api-openshift-ca\") pod \"29e47eb5-9198-4675-8577-215209bdf4f6\" (UID: \"29e47eb5-9198-4675-8577-215209bdf4f6\") "
Oct 30 19:09:23.528265 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:23.528220    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29e47eb5-9198-4675-8577-215209bdf4f6-maas-api-openshift-ca" pod "29e47eb5-9198-4675-8577-215209bdf4f6" (UID: "29e47eb5-9198-4675-8577-215209bdf4f6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:09:23.565738 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:23.565153    4525 patch_prober.go:29] interesting pod/e2e-autores-c164d6-openshift-default-74c7bf49f9-75qr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused" start-of-body=
Oct 30 19:09:23.565738 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:23.565229    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-75qr6" podUID="ac875fe9-dc7c-46fe-9843-3e7bc6ad473f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused"
Oct 30 19:09:23.568980 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:23.568952    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29e47eb5-9198-4675-8577-215209bdf4f6-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:24.017709 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:24.012234    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c164d6-j5b5h"
Oct 30 19:09:24.041726 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:24.040278    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29e47eb5-9198-4675-8577-215209bdf4f6-maas-api-xks-ca" pod "29e47eb5-9198-4675-8577-215209bdf4f6" (UID: "29e47eb5-9198-4675-8577-215209bdf4f6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:09:24.499009 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:24.489618    4525 patch_prober.go:29] interesting pod/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-99fmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body=
Oct 30 19:09:24.521581 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:24.504622    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-99fmr" podUID="be563a8d-e16a-4b6a-b82d-4d6c4847a527" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused"
Oct 30 19:09:24.577157 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:24.577102    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29e47eb5-9198-4675-8577-215209bdf4f6-kube-api-access-g45sh" pod "29e47eb5-9198-4675-8577-215209bdf4f6" (UID: "29e47eb5-9198-4675-8577-215209bdf4f6"). InnerVolumeSpecName "kube-api-access-g45sh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:09:24.599069 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:24.599018    4525 patch_prober.go:29] interesting pod/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" start-of-body=
Oct 30 19:09:24.873800 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:24.855532    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7" podUID="77adecb1-af53-4dd9-bb52-d8e22ebb03e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused"
Oct 30 19:09:24.873800 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:24.624108    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr" podUID="82bbbf85-ed93-48d2-9f17-49ac1f31c0a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:09:24.873800 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:24.623543    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t" podUID="424fca14-6d8a-4663-86ec-0c91165f34ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:09:25.005714 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:25.005660    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29e47eb5-9198-4675-8577-215209bdf4f6-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:25.089344 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:25.008554    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb" podUID="12ebc8b9-c794-47b5-b87b-a63c4bd993e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 19:09:25.266539 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:09:25.266499    4525 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.716s"
Oct 30 19:09:25.267409 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:25.034548    4525 patch_prober.go:29] interesting pod/e2e-autores-c164d6-openshift-default-74c7bf49f9-75qr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused" start-of-body=
Oct 30 19:09:25.268137 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:25.268107    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-99fmr" event={"ID":"be563a8d-e16a-4b6a-b82d-4d6c4847a527","Type":"ContainerStarted","Data":"a50a4388e06b290cfb03ef75805f28936ed1f88439497a667234524251e1ac2e"}
Oct 30 19:09:25.268333 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:25.268315    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c164d6-j5b5h" event={"ID":"29e47eb5-9198-4675-8577-215209bdf4f6","Type":"ContainerDied","Data":"9c87bd16ba60e6abd5101053d0758c820c95ab8acaa4152e041ca617ec813c36"}
Oct 30 19:09:25.268823 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:25.268799    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c87bd16ba60e6abd5101053d0758c820c95ab8acaa4152e041ca617ec813c36"
Oct 30 19:09:25.268966 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:25.268954    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-99fmr"
Oct 30 19:09:25.269200 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:25.269185    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-8wnxx"]
Oct 30 19:09:25.336006 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:25.092393    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts" podUID="be6d4969-285b-4da0-883e-4847f2cbf3af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Oct 30 19:09:25.360704 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:25.336824    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:09:25.360704 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:25.359043    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:09:25.360704 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:25.359205    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:09:25.360704 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:25.267250    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-75qr6" podUID="ac875fe9-dc7c-46fe-9843-3e7bc6ad473f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused"
Oct 30 19:09:25.360704 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:25.080247    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g45sh\" (UniqueName: \"kubernetes.io/projected/29e47eb5-9198-4675-8577-215209bdf4f6-kube-api-access-g45sh\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:25.360704 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:25.347520    4525 cpu_manager.go:414] "containerMap: removing container" podUID="29e47eb5-9198-4675-8577-215209bdf4f6" containerName="revoke-keys"
Oct 30 19:09:25.360704 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:25.360589    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29e47eb5-9198-4675-8577-215209bdf4f6" containerName="revoke-keys"
Oct 30 19:09:25.369708 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:25.369489    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="29e47eb5-9198-4675-8577-215209bdf4f6" containerName="revoke-keys"
Oct 30 19:09:25.370656 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:25.100341    4525 patch_prober.go:29] interesting pod/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" start-of-body=
Oct 30 19:09:25.371136 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:25.371109    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7" podUID="77adecb1-af53-4dd9-bb52-d8e22ebb03e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused"
Oct 30 19:09:25.496061 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:25.495911    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv" podUID="595230e3-55f3-43eb-9a7b-6d89250af9b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:09:25.496843 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:25.244361    4525 patch_prober.go:29] interesting pod/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-99fmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body=
Oct 30 19:09:25.497103 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:25.497058    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-99fmr" podUID="be563a8d-e16a-4b6a-b82d-4d6c4847a527" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused"
Oct 30 19:09:25.526591 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:25.518972    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-8wnxx"]
Oct 30 19:09:25.674092 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:25.092342    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj" podUID="84c99c49-1d60-4176-bda4-cad29c7d801c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 19:09:25.674092 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:25.552369    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-99fmr" podStartSLOduration=10.552343184 podStartE2EDuration="10.552343184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:09:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:09:24.584325594 +0000 UTC m=+1987.209995560" watchObservedRunningTime="2026-10-30 19:09:25.552343184 +0000 UTC m=+1988.178013149"
Oct 30 19:09:25.769209 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:25.769125    4525 patch_prober.go:29] interesting pod/e2e-autores-c164d6-openshift-default-74c7bf49f9-75qr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused" start-of-body=
Oct 30 19:09:25.769388 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:25.769248    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-75qr6" podUID="ac875fe9-dc7c-46fe-9843-3e7bc6ad473f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused"
Oct 30 19:09:25.789717 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:25.788524    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-8wnxx"
Oct 30 19:09:25.870721 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:25.870640    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr" podUID="82bbbf85-ed93-48d2-9f17-49ac1f31c0a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 30 19:09:25.873541 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:25.873503    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t" podUID="424fca14-6d8a-4663-86ec-0c91165f34ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 19:09:25.944705 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:25.939146    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nc69\" (UniqueName: \"kubernetes.io/projected/72e67919-78bc-4e8f-b8f1-13e34b8f54ce-kube-api-access-4nc69\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-8wnxx\" (UID: \"72e67919-78bc-4e8f-b8f1-13e34b8f54ce\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-8wnxx"
Oct 30 19:09:25.944705 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:25.939235    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72e67919-78bc-4e8f-b8f1-13e34b8f54ce-istio-token\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-8wnxx\" (UID: \"72e67919-78bc-4e8f-b8f1-13e34b8f54ce\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-8wnxx"
Oct 30 19:09:25.979723 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:25.977087    4525 patch_prober.go:29] interesting pod/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-99fmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body=
Oct 30 19:09:25.979723 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:25.977163    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-99fmr" podUID="be563a8d-e16a-4b6a-b82d-4d6c4847a527" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused"
Oct 30 19:09:26.009739 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:25.998936    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr"
Oct 30 19:09:26.009739 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:25.999029    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t"
Oct 30 19:09:26.034996 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:26.016799    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72e67919-78bc-4e8f-b8f1-13e34b8f54ce-workload-socket\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-8wnxx\" (UID: \"72e67919-78bc-4e8f-b8f1-13e34b8f54ce\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-8wnxx"
Oct 30 19:09:26.034996 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:26.017734    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72e67919-78bc-4e8f-b8f1-13e34b8f54ce-istio-data\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-8wnxx\" (UID: \"72e67919-78bc-4e8f-b8f1-13e34b8f54ce\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-8wnxx"
Oct 30 19:09:26.034996 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:26.017907    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72e67919-78bc-4e8f-b8f1-13e34b8f54ce-istio-envoy\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-8wnxx\" (UID: \"72e67919-78bc-4e8f-b8f1-13e34b8f54ce\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-8wnxx"
Oct 30 19:09:26.034996 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:26.017955    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72e67919-78bc-4e8f-b8f1-13e34b8f54ce-istiod-ca-cert\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-8wnxx\" (UID: \"72e67919-78bc-4e8f-b8f1-13e34b8f54ce\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-8wnxx"
Oct 30 19:09:26.034996 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:26.017995    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72e67919-78bc-4e8f-b8f1-13e34b8f54ce-istio-podinfo\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-8wnxx\" (UID: \"72e67919-78bc-4e8f-b8f1-13e34b8f54ce\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-8wnxx"
Oct 30 19:09:26.034996 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:26.018272    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72e67919-78bc-4e8f-b8f1-13e34b8f54ce-credential-socket\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-8wnxx\" (UID: \"72e67919-78bc-4e8f-b8f1-13e34b8f54ce\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-8wnxx"
Oct 30 19:09:26.051979 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:26.047765    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72e67919-78bc-4e8f-b8f1-13e34b8f54ce-workload-certs\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-8wnxx\" (UID: \"72e67919-78bc-4e8f-b8f1-13e34b8f54ce\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-8wnxx"
Oct 30 19:09:26.167925 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:26.167850    4525 patch_prober.go:29] interesting pod/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" start-of-body=
Oct 30 19:09:26.168114 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:26.167966    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7" podUID="77adecb1-af53-4dd9-bb52-d8e22ebb03e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused"
Oct 30 19:09:26.300596 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:26.192800    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72e67919-78bc-4e8f-b8f1-13e34b8f54ce-credential-socket\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-8wnxx\" (UID: \"72e67919-78bc-4e8f-b8f1-13e34b8f54ce\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-8wnxx"
Oct 30 19:09:26.300596 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:26.239322    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72e67919-78bc-4e8f-b8f1-13e34b8f54ce-credential-socket\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-8wnxx\" (UID: \"72e67919-78bc-4e8f-b8f1-13e34b8f54ce\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-8wnxx"
Oct 30 19:09:26.384706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:26.371456    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72e67919-78bc-4e8f-b8f1-13e34b8f54ce-workload-certs\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-8wnxx\" (UID: \"72e67919-78bc-4e8f-b8f1-13e34b8f54ce\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-8wnxx"
Oct 30 19:09:26.384706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:26.371558    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nc69\" (UniqueName: \"kubernetes.io/projected/72e67919-78bc-4e8f-b8f1-13e34b8f54ce-kube-api-access-4nc69\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-8wnxx\" (UID: \"72e67919-78bc-4e8f-b8f1-13e34b8f54ce\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-8wnxx"
Oct 30 19:09:26.384706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:26.371625    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72e67919-78bc-4e8f-b8f1-13e34b8f54ce-istio-token\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-8wnxx\" (UID: \"72e67919-78bc-4e8f-b8f1-13e34b8f54ce\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-8wnxx"
Oct 30 19:09:26.384706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:26.371735    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72e67919-78bc-4e8f-b8f1-13e34b8f54ce-workload-socket\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-8wnxx\" (UID: \"72e67919-78bc-4e8f-b8f1-13e34b8f54ce\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-8wnxx"
Oct 30 19:09:26.384706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:26.371837    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72e67919-78bc-4e8f-b8f1-13e34b8f54ce-istio-data\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-8wnxx\" (UID: \"72e67919-78bc-4e8f-b8f1-13e34b8f54ce\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-8wnxx"
Oct 30 19:09:26.384706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:26.372036    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72e67919-78bc-4e8f-b8f1-13e34b8f54ce-istio-envoy\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-8wnxx\" (UID: \"72e67919-78bc-4e8f-b8f1-13e34b8f54ce\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-8wnxx"
Oct 30 19:09:26.384706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:26.372099    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72e67919-78bc-4e8f-b8f1-13e34b8f54ce-istiod-ca-cert\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-8wnxx\" (UID: \"72e67919-78bc-4e8f-b8f1-13e34b8f54ce\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-8wnxx"
Oct 30 19:09:26.384706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:26.372144    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72e67919-78bc-4e8f-b8f1-13e34b8f54ce-istio-podinfo\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-8wnxx\" (UID: \"72e67919-78bc-4e8f-b8f1-13e34b8f54ce\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-8wnxx"
Oct 30 19:09:26.384706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:26.373285    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72e67919-78bc-4e8f-b8f1-13e34b8f54ce-workload-certs\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-8wnxx\" (UID: \"72e67919-78bc-4e8f-b8f1-13e34b8f54ce\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-8wnxx"
Oct 30 19:09:26.430459 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:26.430418    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb" podUID="12ebc8b9-c794-47b5-b87b-a63c4bd993e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 19:09:26.747866 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:26.697807    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72e67919-78bc-4e8f-b8f1-13e34b8f54ce-istiod-ca-cert\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-8wnxx\" (UID: \"72e67919-78bc-4e8f-b8f1-13e34b8f54ce\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-8wnxx"
Oct 30 19:09:26.747866 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:26.698233    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb"
Oct 30 19:09:26.747866 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:26.746836    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72e67919-78bc-4e8f-b8f1-13e34b8f54ce-istio-podinfo\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-8wnxx\" (UID: \"72e67919-78bc-4e8f-b8f1-13e34b8f54ce\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-8wnxx"
Oct 30 19:09:26.747866 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:26.747654    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72e67919-78bc-4e8f-b8f1-13e34b8f54ce-istio-data\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-8wnxx\" (UID: \"72e67919-78bc-4e8f-b8f1-13e34b8f54ce\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-8wnxx"
Oct 30 19:09:26.754232 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:26.748402    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72e67919-78bc-4e8f-b8f1-13e34b8f54ce-istio-envoy\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-8wnxx\" (UID: \"72e67919-78bc-4e8f-b8f1-13e34b8f54ce\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-8wnxx"
Oct 30 19:09:26.754232 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:26.751578    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv" podUID="595230e3-55f3-43eb-9a7b-6d89250af9b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 19:09:26.754232 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:26.752617    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv"
Oct 30 19:09:26.929717 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:26.902822    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw" podUID="ac6852fd-01be-4d69-8c9b-0c591e50b889" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:09:26.929717 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:26.903162    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr" podUID="82bbbf85-ed93-48d2-9f17-49ac1f31c0a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:09:26.929717 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:26.903728    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72e67919-78bc-4e8f-b8f1-13e34b8f54ce-workload-socket\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-8wnxx\" (UID: \"72e67919-78bc-4e8f-b8f1-13e34b8f54ce\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-8wnxx"
Oct 30 19:09:27.071985 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:27.040972    4525 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb" podUID="12ebc8b9-c794-47b5-b87b-a63c4bd993e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": context deadline exceeded"
Oct 30 19:09:27.071985 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:27.041118    4525 patch_prober.go:29] interesting pod/e2e-autores-c164d6-openshift-default-74c7bf49f9-75qr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused" start-of-body=
Oct 30 19:09:27.071985 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:27.041153    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-75qr6" podUID="ac875fe9-dc7c-46fe-9843-3e7bc6ad473f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused"
Oct 30 19:09:27.131753 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:27.127004    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nc69\" (UniqueName: \"kubernetes.io/projected/72e67919-78bc-4e8f-b8f1-13e34b8f54ce-kube-api-access-4nc69\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-8wnxx\" (UID: \"72e67919-78bc-4e8f-b8f1-13e34b8f54ce\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-8wnxx"
Oct 30 19:09:27.163851 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:27.163220    4525 patch_prober.go:29] interesting pod/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-99fmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body=
Oct 30 19:09:27.163851 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:27.163285    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-99fmr" podUID="be563a8d-e16a-4b6a-b82d-4d6c4847a527" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused"
Oct 30 19:09:27.244719 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:27.224067    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-qlfgx"]
Oct 30 19:09:27.316403 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:27.315999    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj" podUID="84c99c49-1d60-4176-bda4-cad29c7d801c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:09:27.354842 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:27.352733    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72e67919-78bc-4e8f-b8f1-13e34b8f54ce-istio-token\") pod \"e2e-extmod-a-e51c10-openshift-default-54b5584fb5-8wnxx\" (UID: \"72e67919-78bc-4e8f-b8f1-13e34b8f54ce\") " pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-8wnxx"
Oct 30 19:09:27.517708 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:27.497274    4525 patch_prober.go:29] interesting pod/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" start-of-body=
Oct 30 19:09:27.517708 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:27.497349    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7" podUID="77adecb1-af53-4dd9-bb52-d8e22ebb03e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused"
Oct 30 19:09:27.637907 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:27.637807    4525 patch_prober.go:29] interesting pod/e2e-autores-c164d6-openshift-default-74c7bf49f9-75qr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused" start-of-body=
Oct 30 19:09:27.731059 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:27.731007    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-75qr6" podUID="ac875fe9-dc7c-46fe-9843-3e7bc6ad473f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused"
Oct 30 19:09:27.731339 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:27.712790    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb"
Oct 30 19:09:27.765057 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:27.765015    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv" podUID="595230e3-55f3-43eb-9a7b-6d89250af9b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:09:27.842846 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:27.842803    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t" podUID="424fca14-6d8a-4663-86ec-0c91165f34ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:09:27.862642 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:27.862603    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-8wnxx"
Oct 30 19:09:27.977732 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:27.968213    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr" podUID="82bbbf85-ed93-48d2-9f17-49ac1f31c0a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 30 19:09:27.977732 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:27.970108    4525 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr" podUID="82bbbf85-ed93-48d2-9f17-49ac1f31c0a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": context deadline exceeded"
Oct 30 19:09:27.977732 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:27.970170    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts" podUID="be6d4969-285b-4da0-883e-4847f2cbf3af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Oct 30 19:09:27.977732 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:27.971201    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw" podUID="ac6852fd-01be-4d69-8c9b-0c591e50b889" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 19:09:28.000872 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:27.987545    4525 patch_prober.go:29] interesting pod/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-99fmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body=
Oct 30 19:09:28.000872 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:27.987603    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-99fmr" podUID="be563a8d-e16a-4b6a-b82d-4d6c4847a527" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused"
Oct 30 19:09:28.152181 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:28.151961    4525 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj" podUID="84c99c49-1d60-4176-bda4-cad29c7d801c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:09:28.153056 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:28.153027    4525 patch_prober.go:29] interesting pod/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" start-of-body=
Oct 30 19:09:28.257810 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:28.153251    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7" podUID="77adecb1-af53-4dd9-bb52-d8e22ebb03e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused"
Oct 30 19:09:28.257810 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:28.252423    4525 patch_prober.go:29] interesting pod/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-qlfgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" start-of-body=
Oct 30 19:09:28.257810 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:28.252475    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-qlfgx" podUID="9a3d6a70-b055-40dc-98a7-d0f5890293c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused"
Oct 30 19:09:28.257810 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:28.254901    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-qlfgx"
Oct 30 19:09:28.257810 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:28.255026    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw"
Oct 30 19:09:28.257810 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:28.255061    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw"
Oct 30 19:09:28.257810 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:28.255101    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-qlfgx" event={"ID":"9a3d6a70-b055-40dc-98a7-d0f5890293c8","Type":"ContainerStarted","Data":"043728e1c4c8db9a53c02c0fdf9069f6c450cc50fef4da5da344e8ccf891eb85"}
Oct 30 19:09:28.257810 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:28.255134    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-qlfgx" event={"ID":"9a3d6a70-b055-40dc-98a7-d0f5890293c8","Type":"ContainerStarted","Data":"7ba1f869f50ef1fea758fee34fba182f12d57cd14196da00eda76aa266e6fe59"}
Oct 30 19:09:28.344931 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:28.327803    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj" podUID="84c99c49-1d60-4176-bda4-cad29c7d801c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:09:28.349458 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:28.348512    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj"
Oct 30 19:09:28.365362 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:28.364971    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t"
Oct 30 19:09:28.501761 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:28.440978    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-qlfgx" podStartSLOduration=7.440957526 podStartE2EDuration="7.440957526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:09:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:09:28.011820615 +0000 UTC m=+1990.637490579" watchObservedRunningTime="2026-10-30 19:09:28.440957526 +0000 UTC m=+1991.066627491"
Oct 30 19:09:28.739016 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:28.737717    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb" podUID="12ebc8b9-c794-47b5-b87b-a63c4bd993e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 19:09:28.766607 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:28.766504    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv" podUID="595230e3-55f3-43eb-9a7b-6d89250af9b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:09:28.767601 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:28.506977    4525 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw" podUID="ac6852fd-01be-4d69-8c9b-0c591e50b889" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:09:28.906724 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:28.877814    4525 patch_prober.go:29] interesting pod/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-99fmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body=
Oct 30 19:09:28.906724 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:28.877893    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-99fmr" podUID="be563a8d-e16a-4b6a-b82d-4d6c4847a527" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused"
Oct 30 19:09:29.059836 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:28.989795    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr" podUID="82bbbf85-ed93-48d2-9f17-49ac1f31c0a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 30 19:09:29.106328 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:29.067896    4525 patch_prober.go:29] interesting pod/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-qlfgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" start-of-body=
Oct 30 19:09:29.106328 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:29.067961    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-qlfgx" podUID="9a3d6a70-b055-40dc-98a7-d0f5890293c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused"
Oct 30 19:09:29.232697 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:29.231602    4525 patch_prober.go:29] interesting pod/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" start-of-body=
Oct 30 19:09:29.232697 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:29.231666    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7" podUID="77adecb1-af53-4dd9-bb52-d8e22ebb03e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused"
Oct 30 19:09:29.346071 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:29.345869    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj" podUID="84c99c49-1d60-4176-bda4-cad29c7d801c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 19:09:29.663713 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:29.661713    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:09:29.663713 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:29.661823    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:09:29.663713 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:29.661881    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:09:29.779859 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:29.779814    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb" podUID="12ebc8b9-c794-47b5-b87b-a63c4bd993e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 19:09:29.825866 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:29.814787    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv" podUID="595230e3-55f3-43eb-9a7b-6d89250af9b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:09:30.026593 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:30.026500    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr" podUID="82bbbf85-ed93-48d2-9f17-49ac1f31c0a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 30 19:09:30.157831 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:30.117015    4525 patch_prober.go:29] interesting pod/e2e-autores-c164d6-openshift-default-74c7bf49f9-75qr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.234:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:09:30.157831 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:30.117127    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-75qr6" podUID="ac875fe9-dc7c-46fe-9843-3e7bc6ad473f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.234:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:09:30.239167 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:30.239105    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb"
Oct 30 19:09:30.239767 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:30.239735    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj"
Oct 30 19:09:30.329227 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:30.329117    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv"
Oct 30 19:09:30.339730 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:30.336275    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr"
Oct 30 19:09:30.344593 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:30.344511    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-75qr6"
Oct 30 19:09:30.362448 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:30.354412    4525 patch_prober.go:29] interesting pod/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-99fmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body=
Oct 30 19:09:30.362448 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:30.354466    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-99fmr" podUID="be563a8d-e16a-4b6a-b82d-4d6c4847a527" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused"
Oct 30 19:09:30.362448 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:30.354550    4525 patch_prober.go:29] interesting pod/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-qlfgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" start-of-body=
Oct 30 19:09:30.362448 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:30.354575    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-qlfgx" podUID="9a3d6a70-b055-40dc-98a7-d0f5890293c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused"
Oct 30 19:09:30.364153 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:30.364131    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-75qr6"
Oct 30 19:09:30.364460 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:30.364443    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb"
Oct 30 19:09:30.472863 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:30.472838    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-75qr6"
Oct 30 19:09:30.475401 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:30.475359    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-8wnxx" event={"ID":"72e67919-78bc-4e8f-b8f1-13e34b8f54ce","Type":"ContainerStarted","Data":"99a2002736016d93fa48c438db3f1b8c38ddc6f611881105c4ce4f026fa63b1f"}
Oct 30 19:09:30.497731 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:30.488570    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-8wnxx"]
Oct 30 19:09:30.758218 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:30.573847    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7"
Oct 30 19:09:30.758218 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:30.578581    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7"
Oct 30 19:09:30.758218 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:30.608558    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7"
Oct 30 19:09:30.916434 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:30.883833    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-99fmr"
Oct 30 19:09:30.916434 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:30.885981    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-99fmr"
Oct 30 19:09:30.916434 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:30.887078    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-99fmr"
Oct 30 19:09:31.164185 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:31.157395    4525 patch_prober.go:29] interesting pod/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-qlfgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" start-of-body=
Oct 30 19:09:31.164185 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:31.157458    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-qlfgx" podUID="9a3d6a70-b055-40dc-98a7-d0f5890293c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused"
Oct 30 19:09:31.559540 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:31.559452    4525 patch_prober.go:29] interesting pod/e2e-autores-c164d6-openshift-default-74c7bf49f9-4dd4v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.227:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:09:31.559988 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:31.559946    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4dd4v" podUID="feea5f3c-8630-4e8f-8832-2998030bf2e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.227:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:09:31.583582 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:31.583545    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-8wnxx" event={"ID":"72e67919-78bc-4e8f-b8f1-13e34b8f54ce","Type":"ContainerStarted","Data":"e1029d0a9968cfcbd673cd6acfed8b749433a60f7dfa3558245729d65976c53f"}
Oct 30 19:09:31.619523 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:31.619432    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-8wnxx" podStartSLOduration=8.619410898 podStartE2EDuration="8.619410898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:09:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:09:31.61484725 +0000 UTC m=+1994.240517216" watchObservedRunningTime="2026-10-30 19:09:31.619410898 +0000 UTC m=+1994.245080863"
Oct 30 19:09:31.946806 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:31.946765    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-8wnxx"
Oct 30 19:09:31.948127 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:31.948089    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-8wnxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body=
Oct 30 19:09:31.948262 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:31.948157    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-8wnxx" podUID="72e67919-78bc-4e8f-b8f1-13e34b8f54ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused"
Oct 30 19:09:32.065301 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:32.065260    4525 patch_prober.go:29] interesting pod/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-qlfgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" start-of-body=
Oct 30 19:09:32.065472 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:32.065332    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-qlfgx" podUID="9a3d6a70-b055-40dc-98a7-d0f5890293c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused"
Oct 30 19:09:32.946376 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:32.946339    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-8wnxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body=
Oct 30 19:09:32.946582 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:32.946403    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-8wnxx" podUID="72e67919-78bc-4e8f-b8f1-13e34b8f54ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused"
Oct 30 19:09:33.065929 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:33.065883    4525 patch_prober.go:29] interesting pod/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-qlfgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" start-of-body=
Oct 30 19:09:33.066346 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:33.065962    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-qlfgx" podUID="9a3d6a70-b055-40dc-98a7-d0f5890293c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused"
Oct 30 19:09:33.946413 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:33.946369    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-8wnxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body=
Oct 30 19:09:33.946613 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:33.946431    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-8wnxx" podUID="72e67919-78bc-4e8f-b8f1-13e34b8f54ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused"
Oct 30 19:09:34.065224 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:34.065184    4525 patch_prober.go:29] interesting pod/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-qlfgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" start-of-body=
Oct 30 19:09:34.065427 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:34.065247    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-qlfgx" podUID="9a3d6a70-b055-40dc-98a7-d0f5890293c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused"
Oct 30 19:09:35.080705 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:35.070499    4525 patch_prober.go:29] interesting pod/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-qlfgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" start-of-body=
Oct 30 19:09:35.080705 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:35.070577    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-qlfgx" podUID="9a3d6a70-b055-40dc-98a7-d0f5890293c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused"
Oct 30 19:09:35.161247 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:35.153815    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-8wnxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body=
Oct 30 19:09:35.161247 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:35.153887    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-8wnxx" podUID="72e67919-78bc-4e8f-b8f1-13e34b8f54ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused"
Oct 30 19:09:35.948228 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:35.948185    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-8wnxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body=
Oct 30 19:09:35.948547 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:35.948252    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-8wnxx" podUID="72e67919-78bc-4e8f-b8f1-13e34b8f54ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused"
Oct 30 19:09:35.959056 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:35.959017    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw" podUID="ac6852fd-01be-4d69-8c9b-0c591e50b889" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:09:35.968476 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:35.968434    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c164d6-j5b5h"]
Oct 30 19:09:35.968757 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:35.968714    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c164d6-j5b5h"]
Oct 30 19:09:36.244172 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:36.244076    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-qlfgx"
Oct 30 19:09:36.244674 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:36.244644    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-qlfgx"
Oct 30 19:09:36.245474 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:36.245454    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-qlfgx"
Oct 30 19:09:36.974704 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:36.965978    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-8wnxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body=
Oct 30 19:09:36.974704 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:36.966088    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-8wnxx" podUID="72e67919-78bc-4e8f-b8f1-13e34b8f54ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused"
Oct 30 19:09:37.546990 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:37.546954    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29e47eb5-9198-4675-8577-215209bdf4f6" path="/var/lib/kubelet/pods/29e47eb5-9198-4675-8577-215209bdf4f6/volumes"
Oct 30 19:09:37.946333 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:37.946297    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-8wnxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body=
Oct 30 19:09:37.946523 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:37.946350    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-8wnxx" podUID="72e67919-78bc-4e8f-b8f1-13e34b8f54ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused"
Oct 30 19:09:38.946325 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:38.946291    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-8wnxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body=
Oct 30 19:09:38.946736 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:38.946360    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-8wnxx" podUID="72e67919-78bc-4e8f-b8f1-13e34b8f54ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused"
Oct 30 19:09:39.951449 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:39.951393    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-8wnxx"
Oct 30 19:09:39.951996 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:39.951789    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-8wnxx"
Oct 30 19:09:39.952751 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:39.952731    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-8wnxx"
Oct 30 19:09:42.165582 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:42.165551    4525 generic.go:356] "Generic (PLEG): container finished" podID="feea5f3c-8630-4e8f-8832-2998030bf2e1" containerID="5355dfafbeb18323c6c9b14ebabc1c273fab1826acfb5c40e6382e56248a21da" exitCode=0
Oct 30 19:09:42.166096 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:42.165627    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4dd4v" event={"ID":"feea5f3c-8630-4e8f-8832-2998030bf2e1","Type":"ContainerDied","Data":"5355dfafbeb18323c6c9b14ebabc1c273fab1826acfb5c40e6382e56248a21da"}
Oct 30 19:09:42.166204 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:42.166189    4525 scope.go:122] "RemoveContainer" containerID="5355dfafbeb18323c6c9b14ebabc1c273fab1826acfb5c40e6382e56248a21da"
Oct 30 19:09:42.168211 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:42.168184    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:09:42.168286 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:42.168259    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:09:42.168323 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:42.168291    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:09:43.198707 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:43.198230    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4dd4v" event={"ID":"feea5f3c-8630-4e8f-8832-2998030bf2e1","Type":"ContainerStarted","Data":"bbea266235cedcd0979ab26e186323b1c2d0d5bc0b80da7e38b57e47b3bd066a"}
Oct 30 19:09:43.267720 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:43.267673    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4dd4v"
Oct 30 19:09:43.268629 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:43.268607    4525 patch_prober.go:29] interesting pod/e2e-autores-c164d6-openshift-default-74c7bf49f9-4dd4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" start-of-body=
Oct 30 19:09:43.268734 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:43.268663    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4dd4v" podUID="feea5f3c-8630-4e8f-8832-2998030bf2e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused"
Oct 30 19:09:43.921555 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:43.921513    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4dd4v"]
Oct 30 19:09:43.928090 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:43.928061    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-75qr6"]
Oct 30 19:09:43.928754 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:43.928399    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-75qr6" podUID="ac875fe9-dc7c-46fe-9843-3e7bc6ad473f" containerName="istio-proxy" containerID="cri-o://7a274493869fc805f0d79df06201ceddfafd8fd4643890696265acd74b423832" gracePeriod=30
Oct 30 19:09:44.270777 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:44.269176    4525 patch_prober.go:29] interesting pod/e2e-autores-c164d6-openshift-default-74c7bf49f9-4dd4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" start-of-body=
Oct 30 19:09:44.270777 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:44.269241    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4dd4v" podUID="feea5f3c-8630-4e8f-8832-2998030bf2e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused"
Oct 30 19:09:44.283326 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:44.283286    4525 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4dd4v" secret="" err="secret \"e2e-autores-c164d6-openshift-default-dockercfg-c42sr\" not found"
Oct 30 19:09:45.267303 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:45.267261    4525 patch_prober.go:29] interesting pod/e2e-autores-c164d6-openshift-default-74c7bf49f9-4dd4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" start-of-body=
Oct 30 19:09:45.267492 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:45.267319    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4dd4v" podUID="feea5f3c-8630-4e8f-8832-2998030bf2e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused"
Oct 30 19:09:45.307382 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:45.307316    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4dd4v" podUID="feea5f3c-8630-4e8f-8832-2998030bf2e1" containerName="istio-proxy" containerID="cri-o://bbea266235cedcd0979ab26e186323b1c2d0d5bc0b80da7e38b57e47b3bd066a" gracePeriod=30
Oct 30 19:09:45.387711 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:45.382266    4525 patch_prober.go:29] interesting pod/e2e-autores-c164d6-openshift-default-74c7bf49f9-75qr6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:09:45.389800 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:45.382331    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-75qr6" podUID="ac875fe9-dc7c-46fe-9843-3e7bc6ad473f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:09:45.455820 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:45.455786    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4dd4v"
Oct 30 19:09:47.430847 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:09:47.430766    4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d49l" containerName="extractor"
Oct 30 19:09:49.305883 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:49.305848    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-75qr6"
Oct 30 19:09:49.338752 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:49.338672    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac875fe9-dc7c-46fe-9843-3e7bc6ad473f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac875fe9-dc7c-46fe-9843-3e7bc6ad473f-workload-certs\") pod \"ac875fe9-dc7c-46fe-9843-3e7bc6ad473f\" (UID: \"ac875fe9-dc7c-46fe-9843-3e7bc6ad473f\") "
Oct 30 19:09:49.339031 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:49.339010    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac875fe9-dc7c-46fe-9843-3e7bc6ad473f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac875fe9-dc7c-46fe-9843-3e7bc6ad473f-istio-envoy\") pod \"ac875fe9-dc7c-46fe-9843-3e7bc6ad473f\" (UID: \"ac875fe9-dc7c-46fe-9843-3e7bc6ad473f\") "
Oct 30 19:09:49.339253 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:49.339234    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac875fe9-dc7c-46fe-9843-3e7bc6ad473f-kube-api-access-k2z44\" (UniqueName: \"kubernetes.io/projected/ac875fe9-dc7c-46fe-9843-3e7bc6ad473f-kube-api-access-k2z44\") pod \"ac875fe9-dc7c-46fe-9843-3e7bc6ad473f\" (UID: \"ac875fe9-dc7c-46fe-9843-3e7bc6ad473f\") "
Oct 30 19:09:49.339513 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:49.339492    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ac875fe9-dc7c-46fe-9843-3e7bc6ad473f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac875fe9-dc7c-46fe-9843-3e7bc6ad473f-istio-podinfo\") pod \"ac875fe9-dc7c-46fe-9843-3e7bc6ad473f\" (UID: \"ac875fe9-dc7c-46fe-9843-3e7bc6ad473f\") "
Oct 30 19:09:49.339798 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:49.339780    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac875fe9-dc7c-46fe-9843-3e7bc6ad473f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac875fe9-dc7c-46fe-9843-3e7bc6ad473f-istio-data\") pod \"ac875fe9-dc7c-46fe-9843-3e7bc6ad473f\" (UID: \"ac875fe9-dc7c-46fe-9843-3e7bc6ad473f\") "
Oct 30 19:09:49.340015 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:49.339998    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac875fe9-dc7c-46fe-9843-3e7bc6ad473f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac875fe9-dc7c-46fe-9843-3e7bc6ad473f-credential-socket\") pod \"ac875fe9-dc7c-46fe-9843-3e7bc6ad473f\" (UID: \"ac875fe9-dc7c-46fe-9843-3e7bc6ad473f\") "
Oct 30 19:09:49.340160 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:49.340144    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac875fe9-dc7c-46fe-9843-3e7bc6ad473f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac875fe9-dc7c-46fe-9843-3e7bc6ad473f-istiod-ca-cert\") pod \"ac875fe9-dc7c-46fe-9843-3e7bc6ad473f\" (UID: \"ac875fe9-dc7c-46fe-9843-3e7bc6ad473f\") "
Oct 30 19:09:49.340287 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:49.340273    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac875fe9-dc7c-46fe-9843-3e7bc6ad473f-istio-token\" (UniqueName: \"kubernetes.io/projected/ac875fe9-dc7c-46fe-9843-3e7bc6ad473f-istio-token\") pod \"ac875fe9-dc7c-46fe-9843-3e7bc6ad473f\" (UID: \"ac875fe9-dc7c-46fe-9843-3e7bc6ad473f\") "
Oct 30 19:09:49.340448 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:49.340432    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac875fe9-dc7c-46fe-9843-3e7bc6ad473f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac875fe9-dc7c-46fe-9843-3e7bc6ad473f-workload-socket\") pod \"ac875fe9-dc7c-46fe-9843-3e7bc6ad473f\" (UID: \"ac875fe9-dc7c-46fe-9843-3e7bc6ad473f\") "
Oct 30 19:09:49.347733 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:49.343804    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac875fe9-dc7c-46fe-9843-3e7bc6ad473f-workload-socket" pod "ac875fe9-dc7c-46fe-9843-3e7bc6ad473f" (UID: "ac875fe9-dc7c-46fe-9843-3e7bc6ad473f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:49.347733 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:49.344401    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac875fe9-dc7c-46fe-9843-3e7bc6ad473f-istiod-ca-cert" pod "ac875fe9-dc7c-46fe-9843-3e7bc6ad473f" (UID: "ac875fe9-dc7c-46fe-9843-3e7bc6ad473f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:09:49.347733 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:49.344863    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac875fe9-dc7c-46fe-9843-3e7bc6ad473f-istio-data" pod "ac875fe9-dc7c-46fe-9843-3e7bc6ad473f" (UID: "ac875fe9-dc7c-46fe-9843-3e7bc6ad473f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:49.347733 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:49.345217    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac875fe9-dc7c-46fe-9843-3e7bc6ad473f-workload-certs" pod "ac875fe9-dc7c-46fe-9843-3e7bc6ad473f" (UID: "ac875fe9-dc7c-46fe-9843-3e7bc6ad473f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:49.347733 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:49.345969    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac875fe9-dc7c-46fe-9843-3e7bc6ad473f-credential-socket" pod "ac875fe9-dc7c-46fe-9843-3e7bc6ad473f" (UID: "ac875fe9-dc7c-46fe-9843-3e7bc6ad473f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:49.353578 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:49.353549    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac875fe9-dc7c-46fe-9843-3e7bc6ad473f-istio-envoy" pod "ac875fe9-dc7c-46fe-9843-3e7bc6ad473f" (UID: "ac875fe9-dc7c-46fe-9843-3e7bc6ad473f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:49.353578 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:49.353549    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ac875fe9-dc7c-46fe-9843-3e7bc6ad473f-istio-podinfo" pod "ac875fe9-dc7c-46fe-9843-3e7bc6ad473f" (UID: "ac875fe9-dc7c-46fe-9843-3e7bc6ad473f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:09:49.353773 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:49.353578    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac875fe9-dc7c-46fe-9843-3e7bc6ad473f-kube-api-access-k2z44" pod "ac875fe9-dc7c-46fe-9843-3e7bc6ad473f" (UID: "ac875fe9-dc7c-46fe-9843-3e7bc6ad473f"). InnerVolumeSpecName "kube-api-access-k2z44". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:09:49.353843 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:49.353825    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac875fe9-dc7c-46fe-9843-3e7bc6ad473f-istio-token" pod "ac875fe9-dc7c-46fe-9843-3e7bc6ad473f" (UID: "ac875fe9-dc7c-46fe-9843-3e7bc6ad473f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:09:49.379105 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:49.379071    4525 generic.go:356] "Generic (PLEG): container finished" podID="ac875fe9-dc7c-46fe-9843-3e7bc6ad473f" containerID="7a274493869fc805f0d79df06201ceddfafd8fd4643890696265acd74b423832" exitCode=0
Oct 30 19:09:49.379324 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:49.379300    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-75qr6" event={"ID":"ac875fe9-dc7c-46fe-9843-3e7bc6ad473f","Type":"ContainerDied","Data":"7a274493869fc805f0d79df06201ceddfafd8fd4643890696265acd74b423832"}
Oct 30 19:09:49.379411 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:49.379395    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-75qr6" event={"ID":"ac875fe9-dc7c-46fe-9843-3e7bc6ad473f","Type":"ContainerDied","Data":"f30698e41424c4bdf0e0db72e600329d3b47bca5ca3930272e8f34260c663091"}
Oct 30 19:09:49.380982 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:49.379432    4525 scope.go:122] "RemoveContainer" containerID="7a274493869fc805f0d79df06201ceddfafd8fd4643890696265acd74b423832"
Oct 30 19:09:49.380982 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:49.379597    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-75qr6"
Oct 30 19:09:49.395824 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:49.395792    4525 scope.go:122] "RemoveContainer" containerID="7a274493869fc805f0d79df06201ceddfafd8fd4643890696265acd74b423832"
Oct 30 19:09:49.396132 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:09:49.396114    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a274493869fc805f0d79df06201ceddfafd8fd4643890696265acd74b423832\": container with ID starting with 7a274493869fc805f0d79df06201ceddfafd8fd4643890696265acd74b423832 not found: ID does not exist" containerID="7a274493869fc805f0d79df06201ceddfafd8fd4643890696265acd74b423832"
Oct 30 19:09:49.396201 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:49.396140    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a274493869fc805f0d79df06201ceddfafd8fd4643890696265acd74b423832"} err="failed to get container status \"7a274493869fc805f0d79df06201ceddfafd8fd4643890696265acd74b423832\": rpc error: code = NotFound desc = could not find container \"7a274493869fc805f0d79df06201ceddfafd8fd4643890696265acd74b423832\": container with ID starting with 7a274493869fc805f0d79df06201ceddfafd8fd4643890696265acd74b423832 not found: ID does not exist"
Oct 30 19:09:49.412301 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:49.412178    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-75qr6"]
Oct 30 19:09:49.422168 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:49.422141    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-75qr6"]
Oct 30 19:09:49.443733 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:49.443700    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2z44\" (UniqueName: \"kubernetes.io/projected/ac875fe9-dc7c-46fe-9843-3e7bc6ad473f-kube-api-access-k2z44\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:49.443733 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:49.443732    4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac875fe9-dc7c-46fe-9843-3e7bc6ad473f-istio-podinfo\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:49.443915 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:49.443741    4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac875fe9-dc7c-46fe-9843-3e7bc6ad473f-istio-data\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:49.443915 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:49.443750    4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac875fe9-dc7c-46fe-9843-3e7bc6ad473f-credential-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:49.443915 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:49.443758    4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac875fe9-dc7c-46fe-9843-3e7bc6ad473f-istiod-ca-cert\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:49.443915 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:49.443768    4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac875fe9-dc7c-46fe-9843-3e7bc6ad473f-istio-token\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:49.443915 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:49.443776    4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac875fe9-dc7c-46fe-9843-3e7bc6ad473f-workload-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:49.443915 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:49.443783    4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac875fe9-dc7c-46fe-9843-3e7bc6ad473f-workload-certs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:49.443915 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:49.443791    4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac875fe9-dc7c-46fe-9843-3e7bc6ad473f-istio-envoy\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:49.578433 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:49.578362    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac875fe9-dc7c-46fe-9843-3e7bc6ad473f" path="/var/lib/kubelet/pods/ac875fe9-dc7c-46fe-9843-3e7bc6ad473f/volumes"
Oct 30 19:09:55.205501 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:55.205473    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4dd4v"
Oct 30 19:09:55.346142 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:55.346094    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/feea5f3c-8630-4e8f-8832-2998030bf2e1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/feea5f3c-8630-4e8f-8832-2998030bf2e1-istio-podinfo\") pod \"feea5f3c-8630-4e8f-8832-2998030bf2e1\" (UID: \"feea5f3c-8630-4e8f-8832-2998030bf2e1\") "
Oct 30 19:09:55.347282 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:55.346167    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/feea5f3c-8630-4e8f-8832-2998030bf2e1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/feea5f3c-8630-4e8f-8832-2998030bf2e1-istio-data\") pod \"feea5f3c-8630-4e8f-8832-2998030bf2e1\" (UID: \"feea5f3c-8630-4e8f-8832-2998030bf2e1\") "
Oct 30 19:09:55.347282 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:55.346231    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/feea5f3c-8630-4e8f-8832-2998030bf2e1-istio-token\" (UniqueName: \"kubernetes.io/projected/feea5f3c-8630-4e8f-8832-2998030bf2e1-istio-token\") pod \"feea5f3c-8630-4e8f-8832-2998030bf2e1\" (UID: \"feea5f3c-8630-4e8f-8832-2998030bf2e1\") "
Oct 30 19:09:55.347282 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:55.346261    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/feea5f3c-8630-4e8f-8832-2998030bf2e1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/feea5f3c-8630-4e8f-8832-2998030bf2e1-credential-socket\") pod \"feea5f3c-8630-4e8f-8832-2998030bf2e1\" (UID: \"feea5f3c-8630-4e8f-8832-2998030bf2e1\") "
Oct 30 19:09:55.347282 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:55.346304    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/feea5f3c-8630-4e8f-8832-2998030bf2e1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/feea5f3c-8630-4e8f-8832-2998030bf2e1-istio-envoy\") pod \"feea5f3c-8630-4e8f-8832-2998030bf2e1\" (UID: \"feea5f3c-8630-4e8f-8832-2998030bf2e1\") "
Oct 30 19:09:55.347282 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:55.346445    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/feea5f3c-8630-4e8f-8832-2998030bf2e1-kube-api-access-7vm9r\" (UniqueName: \"kubernetes.io/projected/feea5f3c-8630-4e8f-8832-2998030bf2e1-kube-api-access-7vm9r\") pod \"feea5f3c-8630-4e8f-8832-2998030bf2e1\" (UID: \"feea5f3c-8630-4e8f-8832-2998030bf2e1\") "
Oct 30 19:09:55.347282 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:55.346505    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/feea5f3c-8630-4e8f-8832-2998030bf2e1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/feea5f3c-8630-4e8f-8832-2998030bf2e1-workload-socket\") pod \"feea5f3c-8630-4e8f-8832-2998030bf2e1\" (UID: \"feea5f3c-8630-4e8f-8832-2998030bf2e1\") "
Oct 30 19:09:55.347282 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:55.346515    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feea5f3c-8630-4e8f-8832-2998030bf2e1-istio-data" pod "feea5f3c-8630-4e8f-8832-2998030bf2e1" (UID: "feea5f3c-8630-4e8f-8832-2998030bf2e1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:55.347282 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:55.346581    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/feea5f3c-8630-4e8f-8832-2998030bf2e1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/feea5f3c-8630-4e8f-8832-2998030bf2e1-workload-certs\") pod \"feea5f3c-8630-4e8f-8832-2998030bf2e1\" (UID: \"feea5f3c-8630-4e8f-8832-2998030bf2e1\") "
Oct 30 19:09:55.347282 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:55.346590    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feea5f3c-8630-4e8f-8832-2998030bf2e1-credential-socket" pod "feea5f3c-8630-4e8f-8832-2998030bf2e1" (UID: "feea5f3c-8630-4e8f-8832-2998030bf2e1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:55.347282 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:55.346724    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/feea5f3c-8630-4e8f-8832-2998030bf2e1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/feea5f3c-8630-4e8f-8832-2998030bf2e1-istiod-ca-cert\") pod \"feea5f3c-8630-4e8f-8832-2998030bf2e1\" (UID: \"feea5f3c-8630-4e8f-8832-2998030bf2e1\") "
Oct 30 19:09:55.347282 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:55.346794    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feea5f3c-8630-4e8f-8832-2998030bf2e1-workload-socket" pod "feea5f3c-8630-4e8f-8832-2998030bf2e1" (UID: "feea5f3c-8630-4e8f-8832-2998030bf2e1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:55.347282 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:55.346890    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feea5f3c-8630-4e8f-8832-2998030bf2e1-workload-certs" pod "feea5f3c-8630-4e8f-8832-2998030bf2e1" (UID: "feea5f3c-8630-4e8f-8832-2998030bf2e1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:55.349387 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:55.347311    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/feea5f3c-8630-4e8f-8832-2998030bf2e1-istiod-ca-cert" pod "feea5f3c-8630-4e8f-8832-2998030bf2e1" (UID: "feea5f3c-8630-4e8f-8832-2998030bf2e1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:09:55.349387 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:55.347469    4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/feea5f3c-8630-4e8f-8832-2998030bf2e1-istio-data\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:55.349387 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:55.347490    4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/feea5f3c-8630-4e8f-8832-2998030bf2e1-credential-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:55.349387 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:55.347500    4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/feea5f3c-8630-4e8f-8832-2998030bf2e1-workload-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:55.349387 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:55.347508    4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/feea5f3c-8630-4e8f-8832-2998030bf2e1-workload-certs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:55.349387 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:55.347516    4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/feea5f3c-8630-4e8f-8832-2998030bf2e1-istiod-ca-cert\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:55.349387 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:55.348803    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feea5f3c-8630-4e8f-8832-2998030bf2e1-istio-envoy" pod "feea5f3c-8630-4e8f-8832-2998030bf2e1" (UID: "feea5f3c-8630-4e8f-8832-2998030bf2e1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:55.349387 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:55.348899    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/feea5f3c-8630-4e8f-8832-2998030bf2e1-kube-api-access-7vm9r" pod "feea5f3c-8630-4e8f-8832-2998030bf2e1" (UID: "feea5f3c-8630-4e8f-8832-2998030bf2e1"). InnerVolumeSpecName "kube-api-access-7vm9r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:09:55.349387 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:55.348913    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/feea5f3c-8630-4e8f-8832-2998030bf2e1-istio-podinfo" pod "feea5f3c-8630-4e8f-8832-2998030bf2e1" (UID: "feea5f3c-8630-4e8f-8832-2998030bf2e1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:09:55.350479 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:55.349459    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/feea5f3c-8630-4e8f-8832-2998030bf2e1-istio-token" pod "feea5f3c-8630-4e8f-8832-2998030bf2e1" (UID: "feea5f3c-8630-4e8f-8832-2998030bf2e1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:09:55.449449 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:55.449352    4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/feea5f3c-8630-4e8f-8832-2998030bf2e1-istio-podinfo\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:55.449449 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:55.449397    4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/feea5f3c-8630-4e8f-8832-2998030bf2e1-istio-token\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:55.449449 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:55.449408    4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/feea5f3c-8630-4e8f-8832-2998030bf2e1-istio-envoy\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:55.449449 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:55.449417    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vm9r\" (UniqueName: \"kubernetes.io/projected/feea5f3c-8630-4e8f-8832-2998030bf2e1-kube-api-access-7vm9r\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:55.672610 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:55.672555    4525 generic.go:356] "Generic (PLEG): container finished" podID="feea5f3c-8630-4e8f-8832-2998030bf2e1" containerID="bbea266235cedcd0979ab26e186323b1c2d0d5bc0b80da7e38b57e47b3bd066a" exitCode=0
Oct 30 19:09:55.672901 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:55.672612    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4dd4v" event={"ID":"feea5f3c-8630-4e8f-8832-2998030bf2e1","Type":"ContainerDied","Data":"bbea266235cedcd0979ab26e186323b1c2d0d5bc0b80da7e38b57e47b3bd066a"}
Oct 30 19:09:55.672901 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:55.672635    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4dd4v"
Oct 30 19:09:55.672901 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:55.672706    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4dd4v" event={"ID":"feea5f3c-8630-4e8f-8832-2998030bf2e1","Type":"ContainerDied","Data":"b37993812cb1e45807a5ec4a43fb1c1776a5cb0f71d9996273cce1b4a61da62d"}
Oct 30 19:09:55.672901 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:55.672739    4525 scope.go:122] "RemoveContainer" containerID="bbea266235cedcd0979ab26e186323b1c2d0d5bc0b80da7e38b57e47b3bd066a"
Oct 30 19:09:55.690035 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:55.689593    4525 scope.go:122] "RemoveContainer" containerID="5355dfafbeb18323c6c9b14ebabc1c273fab1826acfb5c40e6382e56248a21da"
Oct 30 19:09:55.699774 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:55.699713    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4dd4v"]
Oct 30 19:09:55.702151 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:55.702132    4525 scope.go:122] "RemoveContainer" containerID="bbea266235cedcd0979ab26e186323b1c2d0d5bc0b80da7e38b57e47b3bd066a"
Oct 30 19:09:55.702442 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:09:55.702421    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbea266235cedcd0979ab26e186323b1c2d0d5bc0b80da7e38b57e47b3bd066a\": container with ID starting with bbea266235cedcd0979ab26e186323b1c2d0d5bc0b80da7e38b57e47b3bd066a not found: ID does not exist" containerID="bbea266235cedcd0979ab26e186323b1c2d0d5bc0b80da7e38b57e47b3bd066a"
Oct 30 19:09:55.702532 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:55.702452    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbea266235cedcd0979ab26e186323b1c2d0d5bc0b80da7e38b57e47b3bd066a"} err="failed to get container status \"bbea266235cedcd0979ab26e186323b1c2d0d5bc0b80da7e38b57e47b3bd066a\": rpc error: code = NotFound desc = could not find container \"bbea266235cedcd0979ab26e186323b1c2d0d5bc0b80da7e38b57e47b3bd066a\": container with ID starting with bbea266235cedcd0979ab26e186323b1c2d0d5bc0b80da7e38b57e47b3bd066a not found: ID does not exist"
Oct 30 19:09:55.702532 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:55.702470    4525 scope.go:122] "RemoveContainer" containerID="5355dfafbeb18323c6c9b14ebabc1c273fab1826acfb5c40e6382e56248a21da"
Oct 30 19:09:55.703429 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:09:55.702767    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5355dfafbeb18323c6c9b14ebabc1c273fab1826acfb5c40e6382e56248a21da\": container with ID starting with 5355dfafbeb18323c6c9b14ebabc1c273fab1826acfb5c40e6382e56248a21da not found: ID does not exist" containerID="5355dfafbeb18323c6c9b14ebabc1c273fab1826acfb5c40e6382e56248a21da"
Oct 30 19:09:55.703429 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:55.702785    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5355dfafbeb18323c6c9b14ebabc1c273fab1826acfb5c40e6382e56248a21da"} err="failed to get container status \"5355dfafbeb18323c6c9b14ebabc1c273fab1826acfb5c40e6382e56248a21da\": rpc error: code = NotFound desc = could not find container \"5355dfafbeb18323c6c9b14ebabc1c273fab1826acfb5c40e6382e56248a21da\": container with ID starting with 5355dfafbeb18323c6c9b14ebabc1c273fab1826acfb5c40e6382e56248a21da not found: ID does not exist"
Oct 30 19:09:55.707276 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:55.707256    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-c164d6-openshift-default-74c7bf49f9-4dd4v"]
Oct 30 19:09:57.547317 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:09:57.547282    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="feea5f3c-8630-4e8f-8832-2998030bf2e1" path="/var/lib/kubelet/pods/feea5f3c-8630-4e8f-8832-2998030bf2e1/volumes"
Oct 30 19:10:00.742780 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:00.742125    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-mfdt7"]
Oct 30 19:10:00.743627 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:00.743587    4525 cpu_manager.go:414] "containerMap: removing container" podUID="feea5f3c-8630-4e8f-8832-2998030bf2e1" containerName="istio-proxy"
Oct 30 19:10:00.743627 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:00.743616    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="feea5f3c-8630-4e8f-8832-2998030bf2e1" containerName="istio-proxy"
Oct 30 19:10:00.743896 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:00.743645    4525 cpu_manager.go:414] "containerMap: removing container" podUID="ac875fe9-dc7c-46fe-9843-3e7bc6ad473f" containerName="istio-proxy"
Oct 30 19:10:00.743896 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:00.743653    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac875fe9-dc7c-46fe-9843-3e7bc6ad473f" containerName="istio-proxy"
Oct 30 19:10:00.743896 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:00.743848    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="feea5f3c-8630-4e8f-8832-2998030bf2e1" containerName="istio-proxy"
Oct 30 19:10:00.743896 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:00.743869    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac875fe9-dc7c-46fe-9843-3e7bc6ad473f" containerName="istio-proxy"
Oct 30 19:10:00.743896 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:00.743883    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="feea5f3c-8630-4e8f-8832-2998030bf2e1" containerName="istio-proxy"
Oct 30 19:10:00.747883 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:00.747848    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-mfdt7"
Oct 30 19:10:00.777765 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:00.777733    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-mfdt7"]
Oct 30 19:10:00.944422 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:00.944363    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be16530b-c26d-4da7-b25e-f5246eff6c3d-credential-socket\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-mfdt7\" (UID: \"be16530b-c26d-4da7-b25e-f5246eff6c3d\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-mfdt7"
Oct 30 19:10:00.944422 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:00.944420    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be16530b-c26d-4da7-b25e-f5246eff6c3d-istiod-ca-cert\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-mfdt7\" (UID: \"be16530b-c26d-4da7-b25e-f5246eff6c3d\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-mfdt7"
Oct 30 19:10:00.944817 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:00.944520    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be16530b-c26d-4da7-b25e-f5246eff6c3d-istio-envoy\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-mfdt7\" (UID: \"be16530b-c26d-4da7-b25e-f5246eff6c3d\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-mfdt7"
Oct 30 19:10:00.944817 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:00.944564    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be16530b-c26d-4da7-b25e-f5246eff6c3d-istio-data\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-mfdt7\" (UID: \"be16530b-c26d-4da7-b25e-f5246eff6c3d\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-mfdt7"
Oct 30 19:10:00.944817 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:00.944614    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5dqn\" (UniqueName: \"kubernetes.io/projected/be16530b-c26d-4da7-b25e-f5246eff6c3d-kube-api-access-m5dqn\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-mfdt7\" (UID: \"be16530b-c26d-4da7-b25e-f5246eff6c3d\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-mfdt7"
Oct 30 19:10:00.944817 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:00.944736    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be16530b-c26d-4da7-b25e-f5246eff6c3d-workload-certs\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-mfdt7\" (UID: \"be16530b-c26d-4da7-b25e-f5246eff6c3d\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-mfdt7"
Oct 30 19:10:00.944817 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:00.944813    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be16530b-c26d-4da7-b25e-f5246eff6c3d-workload-socket\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-mfdt7\" (UID: \"be16530b-c26d-4da7-b25e-f5246eff6c3d\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-mfdt7"
Oct 30 19:10:00.945262 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:00.944856    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be16530b-c26d-4da7-b25e-f5246eff6c3d-istio-podinfo\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-mfdt7\" (UID: \"be16530b-c26d-4da7-b25e-f5246eff6c3d\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-mfdt7"
Oct 30 19:10:00.945262 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:00.944961    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be16530b-c26d-4da7-b25e-f5246eff6c3d-istio-token\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-mfdt7\" (UID: \"be16530b-c26d-4da7-b25e-f5246eff6c3d\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-mfdt7"
Oct 30 19:10:01.049190 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:01.047221    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be16530b-c26d-4da7-b25e-f5246eff6c3d-workload-certs\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-mfdt7\" (UID: \"be16530b-c26d-4da7-b25e-f5246eff6c3d\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-mfdt7"
Oct 30 19:10:01.049190 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:01.047334    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be16530b-c26d-4da7-b25e-f5246eff6c3d-workload-socket\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-mfdt7\" (UID: \"be16530b-c26d-4da7-b25e-f5246eff6c3d\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-mfdt7"
Oct 30 19:10:01.049190 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:01.047388    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be16530b-c26d-4da7-b25e-f5246eff6c3d-istio-podinfo\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-mfdt7\" (UID: \"be16530b-c26d-4da7-b25e-f5246eff6c3d\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-mfdt7"
Oct 30 19:10:01.049190 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:01.047434    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be16530b-c26d-4da7-b25e-f5246eff6c3d-istio-token\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-mfdt7\" (UID: \"be16530b-c26d-4da7-b25e-f5246eff6c3d\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-mfdt7"
Oct 30 19:10:01.049190 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:01.047669    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be16530b-c26d-4da7-b25e-f5246eff6c3d-credential-socket\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-mfdt7\" (UID: \"be16530b-c26d-4da7-b25e-f5246eff6c3d\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-mfdt7"
Oct 30 19:10:01.049190 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:01.047728    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be16530b-c26d-4da7-b25e-f5246eff6c3d-istiod-ca-cert\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-mfdt7\" (UID: \"be16530b-c26d-4da7-b25e-f5246eff6c3d\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-mfdt7"
Oct 30 19:10:01.049190 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:01.047836    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be16530b-c26d-4da7-b25e-f5246eff6c3d-istio-envoy\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-mfdt7\" (UID: \"be16530b-c26d-4da7-b25e-f5246eff6c3d\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-mfdt7"
Oct 30 19:10:01.049190 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:01.047889    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be16530b-c26d-4da7-b25e-f5246eff6c3d-istio-data\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-mfdt7\" (UID: \"be16530b-c26d-4da7-b25e-f5246eff6c3d\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-mfdt7"
Oct 30 19:10:01.049190 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:01.047951    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5dqn\" (UniqueName: \"kubernetes.io/projected/be16530b-c26d-4da7-b25e-f5246eff6c3d-kube-api-access-m5dqn\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-mfdt7\" (UID: \"be16530b-c26d-4da7-b25e-f5246eff6c3d\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-mfdt7"
Oct 30 19:10:01.049190 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:01.048940    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be16530b-c26d-4da7-b25e-f5246eff6c3d-workload-certs\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-mfdt7\" (UID: \"be16530b-c26d-4da7-b25e-f5246eff6c3d\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-mfdt7"
Oct 30 19:10:01.050559 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:01.050042    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be16530b-c26d-4da7-b25e-f5246eff6c3d-workload-socket\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-mfdt7\" (UID: \"be16530b-c26d-4da7-b25e-f5246eff6c3d\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-mfdt7"
Oct 30 19:10:01.055199 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:01.055162    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be16530b-c26d-4da7-b25e-f5246eff6c3d-istiod-ca-cert\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-mfdt7\" (UID: \"be16530b-c26d-4da7-b25e-f5246eff6c3d\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-mfdt7"
Oct 30 19:10:01.055761 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:01.055736    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be16530b-c26d-4da7-b25e-f5246eff6c3d-credential-socket\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-mfdt7\" (UID: \"be16530b-c26d-4da7-b25e-f5246eff6c3d\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-mfdt7"
Oct 30 19:10:01.057431 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:01.057407    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be16530b-c26d-4da7-b25e-f5246eff6c3d-istio-data\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-mfdt7\" (UID: \"be16530b-c26d-4da7-b25e-f5246eff6c3d\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-mfdt7"
Oct 30 19:10:01.058637 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:01.058605    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be16530b-c26d-4da7-b25e-f5246eff6c3d-istio-envoy\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-mfdt7\" (UID: \"be16530b-c26d-4da7-b25e-f5246eff6c3d\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-mfdt7"
Oct 30 19:10:01.059956 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:01.059932    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5dqn\" (UniqueName: \"kubernetes.io/projected/be16530b-c26d-4da7-b25e-f5246eff6c3d-kube-api-access-m5dqn\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-mfdt7\" (UID: \"be16530b-c26d-4da7-b25e-f5246eff6c3d\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-mfdt7"
Oct 30 19:10:01.061569 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:01.061523    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be16530b-c26d-4da7-b25e-f5246eff6c3d-istio-podinfo\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-mfdt7\" (UID: \"be16530b-c26d-4da7-b25e-f5246eff6c3d\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-mfdt7"
Oct 30 19:10:01.063302 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:01.063279    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be16530b-c26d-4da7-b25e-f5246eff6c3d-istio-token\") pod \"e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-mfdt7\" (UID: \"be16530b-c26d-4da7-b25e-f5246eff6c3d\") " pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-mfdt7"
Oct 30 19:10:01.077573 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:01.077549    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-mfdt7"
Oct 30 19:10:01.543705 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:01.537211    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-mfdt7"]
Oct 30 19:10:01.630480 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:01.630426    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:10:01.630921 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:01.630885    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:10:01.631098 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:01.631080    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:10:01.796651 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:01.796552    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-mfdt7" event={"ID":"be16530b-c26d-4da7-b25e-f5246eff6c3d","Type":"ContainerStarted","Data":"fb6a77bb5fce8a7da75eda45f47dcf56bd0ec1ca4ad69532fc887d503568a238"}
Oct 30 19:10:02.811424 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:02.811382    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-mfdt7" event={"ID":"be16530b-c26d-4da7-b25e-f5246eff6c3d","Type":"ContainerStarted","Data":"09f5a51a5134a8acef4c236f5cd80462bf04576891b2f0856f510489f51e1a9e"}
Oct 30 19:10:02.841979 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:02.841915    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-mfdt7" podStartSLOduration=2.841897999 podStartE2EDuration="2.841897999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:10:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:10:02.835674556 +0000 UTC m=+2025.461344521" watchObservedRunningTime="2026-10-30 19:10:02.841897999 +0000 UTC m=+2025.467567968"
Oct 30 19:10:03.095760 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:03.087989    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-mfdt7"
Oct 30 19:10:03.095760 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:03.088192    4525 patch_prober.go:29] interesting pod/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-mfdt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body=
Oct 30 19:10:03.095760 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:03.088256    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-mfdt7" podUID="be16530b-c26d-4da7-b25e-f5246eff6c3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused"
Oct 30 19:10:04.079709 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:04.079093    4525 patch_prober.go:29] interesting pod/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-mfdt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body=
Oct 30 19:10:04.079709 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:04.079172    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-mfdt7" podUID="be16530b-c26d-4da7-b25e-f5246eff6c3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused"
Oct 30 19:10:05.122709 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:05.122058    4525 patch_prober.go:29] interesting pod/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-mfdt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body=
Oct 30 19:10:05.122709 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:05.122122    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-mfdt7" podUID="be16530b-c26d-4da7-b25e-f5246eff6c3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused"
Oct 30 19:10:06.094706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:06.090129    4525 patch_prober.go:29] interesting pod/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-mfdt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body=
Oct 30 19:10:06.094706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:06.090198    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-mfdt7" podUID="be16530b-c26d-4da7-b25e-f5246eff6c3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused"
Oct 30 19:10:07.100262 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:07.100226    4525 patch_prober.go:29] interesting pod/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-mfdt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body=
Oct 30 19:10:07.101068 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:07.101026    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-mfdt7" podUID="be16530b-c26d-4da7-b25e-f5246eff6c3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused"
Oct 30 19:10:08.078134 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:08.078093    4525 patch_prober.go:29] interesting pod/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-mfdt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body=
Oct 30 19:10:08.078456 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:08.078422    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-mfdt7" podUID="be16530b-c26d-4da7-b25e-f5246eff6c3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused"
Oct 30 19:10:09.078922 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:09.078883    4525 patch_prober.go:29] interesting pod/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-mfdt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body=
Oct 30 19:10:09.079337 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:09.078942    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-mfdt7" podUID="be16530b-c26d-4da7-b25e-f5246eff6c3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused"
Oct 30 19:10:10.084237 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:10.084209    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-mfdt7"
Oct 30 19:10:10.084653 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:10.084572    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-mfdt7"
Oct 30 19:10:10.085347 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:10.085330    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-mfdt7"
Oct 30 19:10:13.871851 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:13.871809    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-37f2b8-546dfdf475-q9nfs"]
Oct 30 19:10:13.873578 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:13.873549    4525 cpu_manager.go:414] "containerMap: removing container" podUID="feea5f3c-8630-4e8f-8832-2998030bf2e1" containerName="istio-proxy"
Oct 30 19:10:13.873706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:13.873581    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="feea5f3c-8630-4e8f-8832-2998030bf2e1" containerName="istio-proxy"
Oct 30 19:10:13.877916 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:13.877894    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-37f2b8-546dfdf475-q9nfs"]
Oct 30 19:10:13.878204 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:13.878122    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-37f2b8-546dfdf475-q9nfs"
Oct 30 19:10:13.882637 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:13.882610    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 19:10:13.883324 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:13.883306    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-37f2b8-serving-cert\""
Oct 30 19:10:13.883471 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:13.883449    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-pd8db\""
Oct 30 19:10:14.047708 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:14.030511    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81d638fc-4ff9-4cd3-9b13-cb71154b4e9a-metrics-certs\") pod \"maas-api-e2e-extmod-a-37f2b8-546dfdf475-q9nfs\" (UID: \"81d638fc-4ff9-4cd3-9b13-cb71154b4e9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-37f2b8-546dfdf475-q9nfs"
Oct 30 19:10:14.047708 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:14.030671    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81d638fc-4ff9-4cd3-9b13-cb71154b4e9a-maas-api-tls\") pod \"maas-api-e2e-extmod-a-37f2b8-546dfdf475-q9nfs\" (UID: \"81d638fc-4ff9-4cd3-9b13-cb71154b4e9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-37f2b8-546dfdf475-q9nfs"
Oct 30 19:10:14.047708 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:14.030789    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfcj9\" (UniqueName: \"kubernetes.io/projected/81d638fc-4ff9-4cd3-9b13-cb71154b4e9a-kube-api-access-vfcj9\") pod \"maas-api-e2e-extmod-a-37f2b8-546dfdf475-q9nfs\" (UID: \"81d638fc-4ff9-4cd3-9b13-cb71154b4e9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-37f2b8-546dfdf475-q9nfs"
Oct 30 19:10:14.047708 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:14.030972    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81d638fc-4ff9-4cd3-9b13-cb71154b4e9a-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-37f2b8-546dfdf475-q9nfs\" (UID: \"81d638fc-4ff9-4cd3-9b13-cb71154b4e9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-37f2b8-546dfdf475-q9nfs"
Oct 30 19:10:14.132095 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:14.132003    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81d638fc-4ff9-4cd3-9b13-cb71154b4e9a-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-37f2b8-546dfdf475-q9nfs\" (UID: \"81d638fc-4ff9-4cd3-9b13-cb71154b4e9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-37f2b8-546dfdf475-q9nfs"
Oct 30 19:10:14.132910 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:14.132138    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81d638fc-4ff9-4cd3-9b13-cb71154b4e9a-metrics-certs\") pod \"maas-api-e2e-extmod-a-37f2b8-546dfdf475-q9nfs\" (UID: \"81d638fc-4ff9-4cd3-9b13-cb71154b4e9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-37f2b8-546dfdf475-q9nfs"
Oct 30 19:10:14.132910 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:14.132287    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81d638fc-4ff9-4cd3-9b13-cb71154b4e9a-maas-api-tls\") pod \"maas-api-e2e-extmod-a-37f2b8-546dfdf475-q9nfs\" (UID: \"81d638fc-4ff9-4cd3-9b13-cb71154b4e9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-37f2b8-546dfdf475-q9nfs"
Oct 30 19:10:14.132910 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:14.132376    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfcj9\" (UniqueName: \"kubernetes.io/projected/81d638fc-4ff9-4cd3-9b13-cb71154b4e9a-kube-api-access-vfcj9\") pod \"maas-api-e2e-extmod-a-37f2b8-546dfdf475-q9nfs\" (UID: \"81d638fc-4ff9-4cd3-9b13-cb71154b4e9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-37f2b8-546dfdf475-q9nfs"
Oct 30 19:10:14.132910 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:10:14.132431    4525 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-37f2b8-serving-cert: secret "maas-api-e2e-extmod-a-37f2b8-serving-cert" not found
Oct 30 19:10:14.132910 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:10:14.132506    4525 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81d638fc-4ff9-4cd3-9b13-cb71154b4e9a-maas-api-tls podName:81d638fc-4ff9-4cd3-9b13-cb71154b4e9a nodeName:}" failed. No retries permitted until 2026-10-30 19:10:14.632487834 +0000 UTC m=+2037.258157791 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/81d638fc-4ff9-4cd3-9b13-cb71154b4e9a-maas-api-tls") pod "maas-api-e2e-extmod-a-37f2b8-546dfdf475-q9nfs" (UID: "81d638fc-4ff9-4cd3-9b13-cb71154b4e9a") : secret "maas-api-e2e-extmod-a-37f2b8-serving-cert" not found
Oct 30 19:10:14.132910 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:14.132774    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81d638fc-4ff9-4cd3-9b13-cb71154b4e9a-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-37f2b8-546dfdf475-q9nfs\" (UID: \"81d638fc-4ff9-4cd3-9b13-cb71154b4e9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-37f2b8-546dfdf475-q9nfs"
Oct 30 19:10:14.135145 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:14.135110    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81d638fc-4ff9-4cd3-9b13-cb71154b4e9a-metrics-certs\") pod \"maas-api-e2e-extmod-a-37f2b8-546dfdf475-q9nfs\" (UID: \"81d638fc-4ff9-4cd3-9b13-cb71154b4e9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-37f2b8-546dfdf475-q9nfs"
Oct 30 19:10:14.144172 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:14.144149    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfcj9\" (UniqueName: \"kubernetes.io/projected/81d638fc-4ff9-4cd3-9b13-cb71154b4e9a-kube-api-access-vfcj9\") pod \"maas-api-e2e-extmod-a-37f2b8-546dfdf475-q9nfs\" (UID: \"81d638fc-4ff9-4cd3-9b13-cb71154b4e9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-37f2b8-546dfdf475-q9nfs"
Oct 30 19:10:14.668161 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:14.668120    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81d638fc-4ff9-4cd3-9b13-cb71154b4e9a-maas-api-tls\") pod \"maas-api-e2e-extmod-a-37f2b8-546dfdf475-q9nfs\" (UID: \"81d638fc-4ff9-4cd3-9b13-cb71154b4e9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-37f2b8-546dfdf475-q9nfs"
Oct 30 19:10:14.672748 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:14.672709    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81d638fc-4ff9-4cd3-9b13-cb71154b4e9a-maas-api-tls\") pod \"maas-api-e2e-extmod-a-37f2b8-546dfdf475-q9nfs\" (UID: \"81d638fc-4ff9-4cd3-9b13-cb71154b4e9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-37f2b8-546dfdf475-q9nfs"
Oct 30 19:10:14.797900 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:14.797859    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-37f2b8-546dfdf475-q9nfs"
Oct 30 19:10:15.100753 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:10:15.100715    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81d638fc_4ff9_4cd3_9b13_cb71154b4e9a.slice/crio-4ee2f567d1a16e71378ca6d5d55224a30221f30b1966f0ce600409d8ae7b7e0d WatchSource:0}: Error finding container 4ee2f567d1a16e71378ca6d5d55224a30221f30b1966f0ce600409d8ae7b7e0d: Status 404 returned error can't find the container with id 4ee2f567d1a16e71378ca6d5d55224a30221f30b1966f0ce600409d8ae7b7e0d
Oct 30 19:10:15.102075 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:15.102051    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-37f2b8-546dfdf475-q9nfs"]
Oct 30 19:10:15.102384 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:15.102364    4525 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:10:16.084908 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:16.084823    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-37f2b8-546dfdf475-q9nfs" event={"ID":"81d638fc-4ff9-4cd3-9b13-cb71154b4e9a","Type":"ContainerStarted","Data":"4ee2f567d1a16e71378ca6d5d55224a30221f30b1966f0ce600409d8ae7b7e0d"}
Oct 30 19:10:17.405612 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:10:17.405570    4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d49l" containerName="extractor"
Oct 30 19:10:19.222332 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:19.220642    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-37f2b8-546dfdf475-q9nfs" event={"ID":"81d638fc-4ff9-4cd3-9b13-cb71154b4e9a","Type":"ContainerStarted","Data":"cf6e6ddcc3db9c349c46bf3e241ee24fff7502865e20516c7b17f6ae9363d022"}
Oct 30 19:10:19.223025 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:19.222999    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-37f2b8-546dfdf475-q9nfs"
Oct 30 19:10:19.243551 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:19.243503    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-37f2b8-546dfdf475-q9nfs" podStartSLOduration=3.147148476 podStartE2EDuration="6.243490812s" totalImagesPullingTime="3.096342336s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:10:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:10:19.241205263 +0000 UTC m=+2041.866875226" watchObservedRunningTime="2026-10-30 19:10:19.243490812 +0000 UTC m=+2041.869160776"
Oct 30 19:10:22.583703 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:22.579019    4525 scope.go:122] "RemoveContainer" containerID="b7901d56252fc83784ba3e6cb5095a20432b97256969e1f369d0af5a0015849d"
Oct 30 19:10:25.235081 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:25.235052    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-37f2b8-546dfdf475-q9nfs"
Oct 30 19:10:35.952110 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:35.952054    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr" podUID="82bbbf85-ed93-48d2-9f17-49ac1f31c0a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:10:35.990834 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:35.990781    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw" podUID="ac6852fd-01be-4d69-8c9b-0c591e50b889" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:10:40.541282 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:40.541247    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-ecc266-68f9ddc76c-2whjr"]
Oct 30 19:10:40.547914 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:40.547886    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ecc266-68f9ddc76c-2whjr"
Oct 30 19:10:40.551694 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:40.551650    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-ecc266\""
Oct 30 19:10:40.552438 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:40.552417    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-ecc266-dockercfg-4zksm\""
Oct 30 19:10:40.552749 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:40.552730    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-ecc266-68f9ddc76c-2whjr"]
Oct 30 19:10:40.743742 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:40.728465    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1eb4c854-2d00-4016-a851-33b9908bc34d-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-ecc266-68f9ddc76c-2whjr\" (UID: \"1eb4c854-2d00-4016-a851-33b9908bc34d\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ecc266-68f9ddc76c-2whjr"
Oct 30 19:10:40.744434 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:40.744398    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgcpj\" (UniqueName: \"kubernetes.io/projected/1eb4c854-2d00-4016-a851-33b9908bc34d-kube-api-access-qgcpj\") pod \"payload-pre-processing-e2e-extmod-b-ecc266-68f9ddc76c-2whjr\" (UID: \"1eb4c854-2d00-4016-a851-33b9908bc34d\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ecc266-68f9ddc76c-2whjr"
Oct 30 19:10:40.914561 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:40.914523    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1eb4c854-2d00-4016-a851-33b9908bc34d-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-ecc266-68f9ddc76c-2whjr\" (UID: \"1eb4c854-2d00-4016-a851-33b9908bc34d\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ecc266-68f9ddc76c-2whjr"
Oct 30 19:10:40.915175 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:40.915150    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgcpj\" (UniqueName: \"kubernetes.io/projected/1eb4c854-2d00-4016-a851-33b9908bc34d-kube-api-access-qgcpj\") pod \"payload-pre-processing-e2e-extmod-b-ecc266-68f9ddc76c-2whjr\" (UID: \"1eb4c854-2d00-4016-a851-33b9908bc34d\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ecc266-68f9ddc76c-2whjr"
Oct 30 19:10:40.947715 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:40.922258    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1eb4c854-2d00-4016-a851-33b9908bc34d-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-ecc266-68f9ddc76c-2whjr\" (UID: \"1eb4c854-2d00-4016-a851-33b9908bc34d\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ecc266-68f9ddc76c-2whjr"
Oct 30 19:10:41.093642 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:41.093603    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgcpj\" (UniqueName: \"kubernetes.io/projected/1eb4c854-2d00-4016-a851-33b9908bc34d-kube-api-access-qgcpj\") pod \"payload-pre-processing-e2e-extmod-b-ecc266-68f9ddc76c-2whjr\" (UID: \"1eb4c854-2d00-4016-a851-33b9908bc34d\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ecc266-68f9ddc76c-2whjr"
Oct 30 19:10:41.167099 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:41.166573    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ecc266-68f9ddc76c-2whjr"
Oct 30 19:10:41.500948 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:41.500851    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-ecc266-68f9ddc76c-2whjr"]
Oct 30 19:10:41.508216 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:10:41.508179    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1eb4c854_2d00_4016_a851_33b9908bc34d.slice/crio-3eafd822163d8e767937032d8262a930baaaed7ef6646c320aed07d4da731517 WatchSource:0}: Error finding container 3eafd822163d8e767937032d8262a930baaaed7ef6646c320aed07d4da731517: Status 404 returned error can't find the container with id 3eafd822163d8e767937032d8262a930baaaed7ef6646c320aed07d4da731517
Oct 30 19:10:41.587749 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:41.587716    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ecc266-68f9ddc76c-2whjr" event={"ID":"1eb4c854-2d00-4016-a851-33b9908bc34d","Type":"ContainerStarted","Data":"3eafd822163d8e767937032d8262a930baaaed7ef6646c320aed07d4da731517"}
Oct 30 19:10:42.598046 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:42.597990    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ecc266-68f9ddc76c-2whjr" event={"ID":"1eb4c854-2d00-4016-a851-33b9908bc34d","Type":"ContainerStarted","Data":"2e0795455b95992a78b03ce1175876ce8eaeb573ca5133c6a04440604cd09345"}
Oct 30 19:10:42.598534 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:42.598067    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ecc266-68f9ddc76c-2whjr"
Oct 30 19:10:42.623721 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:42.623649    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ecc266-68f9ddc76c-2whjr" podStartSLOduration=2.623631739 podStartE2EDuration="2.623631739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:10:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:10:42.622626659 +0000 UTC m=+2065.248296623" watchObservedRunningTime="2026-10-30 19:10:42.623631739 +0000 UTC m=+2065.249301703"
Oct 30 19:10:47.400555 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:10:47.400518    4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d49l" containerName="extractor"
Oct 30 19:10:53.834857 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:10:53.834827    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ecc266-68f9ddc76c-2whjr"
Oct 30 19:11:17.401117 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:11:17.401072    4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d49l" containerName="extractor"
Oct 30 19:11:25.029432 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:11:25.029395    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50605e61-5dwzf"]
Oct 30 19:11:25.034564 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:11:25.034538    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50605e61-5dwzf"]
Oct 30 19:11:25.547024 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:11:25.546987    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e87cddb0-df80-4259-a332-3f3ea69f431a" path="/var/lib/kubelet/pods/e87cddb0-df80-4259-a332-3f3ea69f431a/volumes"
Oct 30 19:11:26.024942 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:11:26.024905    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e92d4a69-pqvbh"]
Oct 30 19:11:26.029826 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:11:26.029803    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e92d4a69-pqvbh"]
Oct 30 19:11:27.547466 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:11:27.547435    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d5589b4-0028-4529-91d8-5ef4b4684476" path="/var/lib/kubelet/pods/6d5589b4-0028-4529-91d8-5ef4b4684476/volumes"
Oct 30 19:11:35.158757 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:11:35.157866    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aff7c585-wtlks"]
Oct 30 19:11:35.258323 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:11:35.248737    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aff7c585-wtlks"]
Oct 30 19:11:35.617725 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:11:35.608555    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30f43a69-f88c-42fe-a3bc-2d89f22e590c" path="/var/lib/kubelet/pods/30f43a69-f88c-42fe-a3bc-2d89f22e590c/volumes"
Oct 30 19:11:35.776943 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:11:35.775785    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv" podUID="595230e3-55f3-43eb-9a7b-6d89250af9b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:11:35.846870 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:11:35.844967    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t" podUID="424fca14-6d8a-4663-86ec-0c91165f34ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:11:35.846870 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:11:35.846157    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts" podUID="be6d4969-285b-4da0-883e-4847f2cbf3af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:11:35.913762 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:11:35.907822    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr" podUID="82bbbf85-ed93-48d2-9f17-49ac1f31c0a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:11:35.992870 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:11:35.992787    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw" podUID="ac6852fd-01be-4d69-8c9b-0c591e50b889" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:11:36.083827 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:11:36.083768    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3f6d48f-vqc5d"]
Oct 30 19:11:36.143709 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:11:36.140550    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3f6d48f-vqc5d"]
Oct 30 19:11:36.258929 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:11:36.258828    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj" podUID="84c99c49-1d60-4176-bda4-cad29c7d801c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:11:36.744834 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:11:36.744776    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb" podUID="12ebc8b9-c794-47b5-b87b-a63c4bd993e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:11:36.848900 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:11:36.848838    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t" podUID="424fca14-6d8a-4663-86ec-0c91165f34ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:11:36.849822 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:11:36.849794    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t"
Oct 30 19:11:36.942854 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:11:36.942803    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr" podUID="82bbbf85-ed93-48d2-9f17-49ac1f31c0a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:11:36.943747 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:11:36.943722    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr"
Oct 30 19:11:37.032952 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:11:37.013858    4525 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb" podUID="12ebc8b9-c794-47b5-b87b-a63c4bd993e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": context deadline exceeded"
Oct 30 19:11:37.032952 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:11:37.018173    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw" podUID="ac6852fd-01be-4d69-8c9b-0c591e50b889" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:11:37.032952 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:11:37.020063    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw"
Oct 30 19:11:37.261369 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:11:37.261316    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj" podUID="84c99c49-1d60-4176-bda4-cad29c7d801c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:11:37.262617 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:11:37.262593    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj"
Oct 30 19:11:37.391484 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:11:37.391449    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw"
Oct 30 19:11:37.391826 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:11:37.391515    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr"
Oct 30 19:11:37.392030 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:11:37.391942    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj"
Oct 30 19:11:37.392919 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:11:37.392894    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t"
Oct 30 19:11:37.558784 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:11:37.558745    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc3415ec-21bd-40f3-baec-cb80ebe2468f" path="/var/lib/kubelet/pods/fc3415ec-21bd-40f3-baec-cb80ebe2468f/volumes"
Oct 30 19:11:39.814795 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:11:39.814735    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv" podUID="595230e3-55f3-43eb-9a7b-6d89250af9b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:11:39.865411 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:11:39.815429    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb" podUID="12ebc8b9-c794-47b5-b87b-a63c4bd993e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:11:39.877618 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:11:39.853509    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts" podUID="be6d4969-285b-4da0-883e-4847f2cbf3af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:11:40.401164 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:11:40.401113    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw" podUID="ac6852fd-01be-4d69-8c9b-0c591e50b889" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 19:11:40.402379 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:11:40.401164    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj" podUID="84c99c49-1d60-4176-bda4-cad29c7d801c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:11:40.402663 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:11:40.402636    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr" podUID="82bbbf85-ed93-48d2-9f17-49ac1f31c0a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:11:40.420163 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:11:40.414813    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t" podUID="424fca14-6d8a-4663-86ec-0c91165f34ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:11:47.410167 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:11:47.410076    4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d49l" containerName="extractor"
Oct 30 19:11:52.839865 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:11:52.837222    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.219:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:11:52.839865 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:11:52.837303    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx" podUID="b2099828-c41e-43c1-b174-63e8eb826c05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.219:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:11:59.153707 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:11:59.147819    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e2a74ee-8rn8t"]
Oct 30 19:11:59.167760 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:11:59.167720    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e2a74ee-8rn8t"]
Oct 30 19:11:59.549567 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:11:59.549486    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7feb7ae1-63f7-499a-ab49-cc561b2a2ca5" path="/var/lib/kubelet/pods/7feb7ae1-63f7-499a-ab49-cc561b2a2ca5/volumes"
Oct 30 19:12:00.034358 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:12:00.034324    4525 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/system.slice/system-systemd\\\\x2dcoredump.slice/systemd-coredump@33-38137-0.service\": RecentStats: unable to find data in memory cache]"
Oct 30 19:12:00.968406 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:00.968372    4525 generic.go:356] "Generic (PLEG): container finished" podID="b2099828-c41e-43c1-b174-63e8eb826c05" containerID="4e4ca10e76fb3b83340b3f05a3a5934faae758882759cd4acbf5258f4c00f1a7" exitCode=0
Oct 30 19:12:00.968890 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:00.968446    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx" event={"ID":"b2099828-c41e-43c1-b174-63e8eb826c05","Type":"ContainerDied","Data":"4e4ca10e76fb3b83340b3f05a3a5934faae758882759cd4acbf5258f4c00f1a7"}
Oct 30 19:12:00.969665 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:00.969001    4525 scope.go:122] "RemoveContainer" containerID="4e4ca10e76fb3b83340b3f05a3a5934faae758882759cd4acbf5258f4c00f1a7"
Oct 30 19:12:00.971384 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:00.971350    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:12:00.971519 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:00.971445    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:12:00.971519 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:00.971497    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:12:02.007155 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:02.007112    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx" event={"ID":"b2099828-c41e-43c1-b174-63e8eb826c05","Type":"ContainerStarted","Data":"d2361ac39ee640bb402fb081951a8bce097a4f38e46382ffd99fc6093953ce05"}
Oct 30 19:12:02.439529 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:02.439502    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx"
Oct 30 19:12:02.440448 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:02.440422    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" start-of-body=
Oct 30 19:12:02.440569 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:02.440479    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx" podUID="b2099828-c41e-43c1-b174-63e8eb826c05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused"
Oct 30 19:12:03.439113 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:03.439072    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" start-of-body=
Oct 30 19:12:03.439547 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:03.439149    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx" podUID="b2099828-c41e-43c1-b174-63e8eb826c05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused"
Oct 30 19:12:04.443702 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:04.442778    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" start-of-body=
Oct 30 19:12:04.443702 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:04.442838    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx" podUID="b2099828-c41e-43c1-b174-63e8eb826c05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused"
Oct 30 19:12:05.439083 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:05.439047    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" start-of-body=
Oct 30 19:12:05.439296 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:05.439109    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx" podUID="b2099828-c41e-43c1-b174-63e8eb826c05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused"
Oct 30 19:12:06.331441 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:06.331404    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e93a3449-tnzzz"]
Oct 30 19:12:06.336449 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:06.336428    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e93a3449-tnzzz"
Oct 30 19:12:06.349395 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:06.349370    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e93a3449-tnzzz"]
Oct 30 19:12:06.352001 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:06.351974    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc053426-d0cc-4c1e-928a-5b7624ae02bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e93a3449-tnzzz\" (UID: \"bc053426-d0cc-4c1e-928a-5b7624ae02bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e93a3449-tnzzz"
Oct 30 19:12:06.352393 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:06.352366    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc053426-d0cc-4c1e-928a-5b7624ae02bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e93a3449-tnzzz\" (UID: \"bc053426-d0cc-4c1e-928a-5b7624ae02bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e93a3449-tnzzz"
Oct 30 19:12:06.352548 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:06.352525    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pw8pv\" (UniqueName: \"kubernetes.io/projected/bc053426-d0cc-4c1e-928a-5b7624ae02bb-kube-api-access-pw8pv\") pod \"maas-api-revoke-sub-e93a3449-tnzzz\" (UID: \"bc053426-d0cc-4c1e-928a-5b7624ae02bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e93a3449-tnzzz"
Oct 30 19:12:06.401983 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:06.401941    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:12:06.402208 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:06.402104    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94hcn\""
Oct 30 19:12:06.439339 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:06.439305    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" start-of-body=
Oct 30 19:12:06.440451 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:06.439838    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx" podUID="b2099828-c41e-43c1-b174-63e8eb826c05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused"
Oct 30 19:12:06.453978 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:06.453950    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc053426-d0cc-4c1e-928a-5b7624ae02bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e93a3449-tnzzz\" (UID: \"bc053426-d0cc-4c1e-928a-5b7624ae02bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e93a3449-tnzzz"
Oct 30 19:12:06.454149 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:06.454029    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw8pv\" (UniqueName: \"kubernetes.io/projected/bc053426-d0cc-4c1e-928a-5b7624ae02bb-kube-api-access-pw8pv\") pod \"maas-api-revoke-sub-e93a3449-tnzzz\" (UID: \"bc053426-d0cc-4c1e-928a-5b7624ae02bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e93a3449-tnzzz"
Oct 30 19:12:06.454149 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:06.454111    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc053426-d0cc-4c1e-928a-5b7624ae02bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e93a3449-tnzzz\" (UID: \"bc053426-d0cc-4c1e-928a-5b7624ae02bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e93a3449-tnzzz"
Oct 30 19:12:06.454627 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:06.454607    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc053426-d0cc-4c1e-928a-5b7624ae02bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e93a3449-tnzzz\" (UID: \"bc053426-d0cc-4c1e-928a-5b7624ae02bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e93a3449-tnzzz"
Oct 30 19:12:06.456974 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:06.456949    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc053426-d0cc-4c1e-928a-5b7624ae02bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e93a3449-tnzzz\" (UID: \"bc053426-d0cc-4c1e-928a-5b7624ae02bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e93a3449-tnzzz"
Oct 30 19:12:06.464458 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:06.464433    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pw8pv\" (UniqueName: \"kubernetes.io/projected/bc053426-d0cc-4c1e-928a-5b7624ae02bb-kube-api-access-pw8pv\") pod \"maas-api-revoke-sub-e93a3449-tnzzz\" (UID: \"bc053426-d0cc-4c1e-928a-5b7624ae02bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e93a3449-tnzzz"
Oct 30 19:12:06.653205 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:06.653172    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e93a3449-tnzzz"
Oct 30 19:12:06.807215 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:06.804909    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx"
Oct 30 19:12:07.013719 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:07.013654    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e93a3449-tnzzz"]
Oct 30 19:12:07.134803 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:07.134768    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e93a3449-tnzzz" event={"ID":"bc053426-d0cc-4c1e-928a-5b7624ae02bb","Type":"ContainerStarted","Data":"f7d4561ca78e57a2c66e0abf11a8dc0f7f7a52f03ee0a0ceb27dfa884633995c"}
Oct 30 19:12:07.438955 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:07.438925    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" start-of-body=
Oct 30 19:12:07.439463 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:07.438980    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx" podUID="b2099828-c41e-43c1-b174-63e8eb826c05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused"
Oct 30 19:12:08.144793 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:08.144742    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e93a3449-tnzzz" event={"ID":"bc053426-d0cc-4c1e-928a-5b7624ae02bb","Type":"ContainerStarted","Data":"5ba9ca05a407e818624c09d442b6e49af8bacb1de7ec962516e79e121990b961"}
Oct 30 19:12:08.165712 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:08.165643    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e93a3449-tnzzz" podStartSLOduration=2.165629131 podStartE2EDuration="2.165629131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:12:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:12:08.162927422 +0000 UTC m=+2150.788597386" watchObservedRunningTime="2026-10-30 19:12:08.165629131 +0000 UTC m=+2150.791299095"
Oct 30 19:12:08.444075 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:08.443997    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx"
Oct 30 19:12:08.445390 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:08.445370    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx"
Oct 30 19:12:09.028365 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:09.028328    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1af7d73c-njhbh"]
Oct 30 19:12:09.033414 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:09.033387    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1af7d73c-njhbh"]
Oct 30 19:12:09.155106 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:09.155073    4525 generic.go:356] "Generic (PLEG): container finished" podID="bc053426-d0cc-4c1e-928a-5b7624ae02bb" containerID="5ba9ca05a407e818624c09d442b6e49af8bacb1de7ec962516e79e121990b961" exitCode=0
Oct 30 19:12:09.155295 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:09.155157    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e93a3449-tnzzz" event={"ID":"bc053426-d0cc-4c1e-928a-5b7624ae02bb","Type":"ContainerDied","Data":"5ba9ca05a407e818624c09d442b6e49af8bacb1de7ec962516e79e121990b961"}
Oct 30 19:12:09.623720 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:09.613896    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8df5475-2c04-4c22-b53e-f01d9c4e6e32" path="/var/lib/kubelet/pods/e8df5475-2c04-4c22-b53e-f01d9c4e6e32/volumes"
Oct 30 19:12:10.368374 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:10.368340    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e93a3449-tnzzz"
Oct 30 19:12:10.420340 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:10.420242    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc053426-d0cc-4c1e-928a-5b7624ae02bb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc053426-d0cc-4c1e-928a-5b7624ae02bb-maas-api-xks-ca\") pod \"bc053426-d0cc-4c1e-928a-5b7624ae02bb\" (UID: \"bc053426-d0cc-4c1e-928a-5b7624ae02bb\") "
Oct 30 19:12:10.420340 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:10.420329    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc053426-d0cc-4c1e-928a-5b7624ae02bb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc053426-d0cc-4c1e-928a-5b7624ae02bb-maas-api-openshift-ca\") pod \"bc053426-d0cc-4c1e-928a-5b7624ae02bb\" (UID: \"bc053426-d0cc-4c1e-928a-5b7624ae02bb\") "
Oct 30 19:12:10.420594 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:10.420520    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc053426-d0cc-4c1e-928a-5b7624ae02bb-kube-api-access-pw8pv\" (UniqueName: \"kubernetes.io/projected/bc053426-d0cc-4c1e-928a-5b7624ae02bb-kube-api-access-pw8pv\") pod \"bc053426-d0cc-4c1e-928a-5b7624ae02bb\" (UID: \"bc053426-d0cc-4c1e-928a-5b7624ae02bb\") "
Oct 30 19:12:10.421255 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:10.421225    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc053426-d0cc-4c1e-928a-5b7624ae02bb-maas-api-openshift-ca" pod "bc053426-d0cc-4c1e-928a-5b7624ae02bb" (UID: "bc053426-d0cc-4c1e-928a-5b7624ae02bb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:12:10.421756 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:10.421730    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc053426-d0cc-4c1e-928a-5b7624ae02bb-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:10.423068 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:10.423040    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc053426-d0cc-4c1e-928a-5b7624ae02bb-maas-api-xks-ca" pod "bc053426-d0cc-4c1e-928a-5b7624ae02bb" (UID: "bc053426-d0cc-4c1e-928a-5b7624ae02bb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:12:10.423274 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:10.423255    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc053426-d0cc-4c1e-928a-5b7624ae02bb-kube-api-access-pw8pv" pod "bc053426-d0cc-4c1e-928a-5b7624ae02bb" (UID: "bc053426-d0cc-4c1e-928a-5b7624ae02bb"). InnerVolumeSpecName "kube-api-access-pw8pv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:12:10.566714 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:10.561216    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc053426-d0cc-4c1e-928a-5b7624ae02bb-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:10.566714 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:10.561253    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pw8pv\" (UniqueName: \"kubernetes.io/projected/bc053426-d0cc-4c1e-928a-5b7624ae02bb-kube-api-access-pw8pv\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:11.174362 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:11.174334    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e93a3449-tnzzz"
Oct 30 19:12:11.174822 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:11.174334    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e93a3449-tnzzz" event={"ID":"bc053426-d0cc-4c1e-928a-5b7624ae02bb","Type":"ContainerDied","Data":"f7d4561ca78e57a2c66e0abf11a8dc0f7f7a52f03ee0a0ceb27dfa884633995c"}
Oct 30 19:12:11.174822 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:11.174471    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f7d4561ca78e57a2c66e0abf11a8dc0f7f7a52f03ee0a0ceb27dfa884633995c"
Oct 30 19:12:14.031448 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:14.031407    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6708ccfa-txvfl"]
Oct 30 19:12:14.032591 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:14.032556    4525 cpu_manager.go:414] "containerMap: removing container" podUID="bc053426-d0cc-4c1e-928a-5b7624ae02bb" containerName="revoke-keys"
Oct 30 19:12:14.032591 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:14.032589    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc053426-d0cc-4c1e-928a-5b7624ae02bb" containerName="revoke-keys"
Oct 30 19:12:14.032829 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:14.032814    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc053426-d0cc-4c1e-928a-5b7624ae02bb" containerName="revoke-keys"
Oct 30 19:12:14.036941 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:14.036922    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6708ccfa-txvfl"
Oct 30 19:12:14.042983 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:14.042948    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:12:14.043121 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:14.043064    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94hcn\""
Oct 30 19:12:14.047462 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:14.047437    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6708ccfa-txvfl"]
Oct 30 19:12:14.132575 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:14.132540    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0fe7254-bbf9-44ed-93b3-52df9ca26921-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6708ccfa-txvfl\" (UID: \"f0fe7254-bbf9-44ed-93b3-52df9ca26921\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6708ccfa-txvfl"
Oct 30 19:12:14.132802 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:14.132766    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vh64\" (UniqueName: \"kubernetes.io/projected/f0fe7254-bbf9-44ed-93b3-52df9ca26921-kube-api-access-7vh64\") pod \"maas-api-revoke-sub-6708ccfa-txvfl\" (UID: \"f0fe7254-bbf9-44ed-93b3-52df9ca26921\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6708ccfa-txvfl"
Oct 30 19:12:14.132871 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:14.132803    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0fe7254-bbf9-44ed-93b3-52df9ca26921-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6708ccfa-txvfl\" (UID: \"f0fe7254-bbf9-44ed-93b3-52df9ca26921\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6708ccfa-txvfl"
Oct 30 19:12:14.234282 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:14.234237    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0fe7254-bbf9-44ed-93b3-52df9ca26921-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6708ccfa-txvfl\" (UID: \"f0fe7254-bbf9-44ed-93b3-52df9ca26921\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6708ccfa-txvfl"
Oct 30 19:12:14.234488 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:14.234445    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vh64\" (UniqueName: \"kubernetes.io/projected/f0fe7254-bbf9-44ed-93b3-52df9ca26921-kube-api-access-7vh64\") pod \"maas-api-revoke-sub-6708ccfa-txvfl\" (UID: \"f0fe7254-bbf9-44ed-93b3-52df9ca26921\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6708ccfa-txvfl"
Oct 30 19:12:14.234561 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:14.234489    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0fe7254-bbf9-44ed-93b3-52df9ca26921-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6708ccfa-txvfl\" (UID: \"f0fe7254-bbf9-44ed-93b3-52df9ca26921\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6708ccfa-txvfl"
Oct 30 19:12:14.235251 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:14.235221    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0fe7254-bbf9-44ed-93b3-52df9ca26921-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6708ccfa-txvfl\" (UID: \"f0fe7254-bbf9-44ed-93b3-52df9ca26921\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6708ccfa-txvfl"
Oct 30 19:12:14.237161 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:14.237133    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0fe7254-bbf9-44ed-93b3-52df9ca26921-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6708ccfa-txvfl\" (UID: \"f0fe7254-bbf9-44ed-93b3-52df9ca26921\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6708ccfa-txvfl"
Oct 30 19:12:14.243815 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:14.243787    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vh64\" (UniqueName: \"kubernetes.io/projected/f0fe7254-bbf9-44ed-93b3-52df9ca26921-kube-api-access-7vh64\") pod \"maas-api-revoke-sub-6708ccfa-txvfl\" (UID: \"f0fe7254-bbf9-44ed-93b3-52df9ca26921\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6708ccfa-txvfl"
Oct 30 19:12:14.368292 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:14.368255    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6708ccfa-txvfl"
Oct 30 19:12:14.689153 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:12:14.689120    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0fe7254_bbf9_44ed_93b3_52df9ca26921.slice/crio-4d5a9afafb1b2994723a1bd869e3b5b1e3c666c484d696b24680adaedda10e6c WatchSource:0}: Error finding container 4d5a9afafb1b2994723a1bd869e3b5b1e3c666c484d696b24680adaedda10e6c: Status 404 returned error can't find the container with id 4d5a9afafb1b2994723a1bd869e3b5b1e3c666c484d696b24680adaedda10e6c
Oct 30 19:12:14.691854 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:14.691818    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6708ccfa-txvfl"]
Oct 30 19:12:15.215547 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:15.215516    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6708ccfa-txvfl" event={"ID":"f0fe7254-bbf9-44ed-93b3-52df9ca26921","Type":"ContainerStarted","Data":"05ed619cb3a472a3e5421f05c6e998770f19035f900d1ca5c48e89d8ace4ce66"}
Oct 30 19:12:15.216372 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:15.215576    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6708ccfa-txvfl" event={"ID":"f0fe7254-bbf9-44ed-93b3-52df9ca26921","Type":"ContainerStarted","Data":"4d5a9afafb1b2994723a1bd869e3b5b1e3c666c484d696b24680adaedda10e6c"}
Oct 30 19:12:15.234923 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:15.234874    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6708ccfa-txvfl" podStartSLOduration=1.234858815 podStartE2EDuration="1.234858815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:12:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:12:15.23311989 +0000 UTC m=+2157.858789855" watchObservedRunningTime="2026-10-30 19:12:15.234858815 +0000 UTC m=+2157.860528778"
Oct 30 19:12:16.226153 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:16.226114    4525 generic.go:356] "Generic (PLEG): container finished" podID="f0fe7254-bbf9-44ed-93b3-52df9ca26921" containerID="05ed619cb3a472a3e5421f05c6e998770f19035f900d1ca5c48e89d8ace4ce66" exitCode=0
Oct 30 19:12:16.226596 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:16.226202    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6708ccfa-txvfl" event={"ID":"f0fe7254-bbf9-44ed-93b3-52df9ca26921","Type":"ContainerDied","Data":"05ed619cb3a472a3e5421f05c6e998770f19035f900d1ca5c48e89d8ace4ce66"}
Oct 30 19:12:17.495990 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:12:17.495946    4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d49l" containerName="extractor"
Oct 30 19:12:17.824213 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:17.824176    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6708ccfa-txvfl"
Oct 30 19:12:17.976037 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:17.974187    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0fe7254-bbf9-44ed-93b3-52df9ca26921-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0fe7254-bbf9-44ed-93b3-52df9ca26921-maas-api-openshift-ca\") pod \"f0fe7254-bbf9-44ed-93b3-52df9ca26921\" (UID: \"f0fe7254-bbf9-44ed-93b3-52df9ca26921\") "
Oct 30 19:12:17.976037 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:17.974459    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0fe7254-bbf9-44ed-93b3-52df9ca26921-kube-api-access-7vh64\" (UniqueName: \"kubernetes.io/projected/f0fe7254-bbf9-44ed-93b3-52df9ca26921-kube-api-access-7vh64\") pod \"f0fe7254-bbf9-44ed-93b3-52df9ca26921\" (UID: \"f0fe7254-bbf9-44ed-93b3-52df9ca26921\") "
Oct 30 19:12:17.976037 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:17.974785    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0fe7254-bbf9-44ed-93b3-52df9ca26921-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0fe7254-bbf9-44ed-93b3-52df9ca26921-maas-api-xks-ca\") pod \"f0fe7254-bbf9-44ed-93b3-52df9ca26921\" (UID: \"f0fe7254-bbf9-44ed-93b3-52df9ca26921\") "
Oct 30 19:12:18.236804 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:18.236758    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0fe7254-bbf9-44ed-93b3-52df9ca26921-maas-api-openshift-ca" pod "f0fe7254-bbf9-44ed-93b3-52df9ca26921" (UID: "f0fe7254-bbf9-44ed-93b3-52df9ca26921"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:12:18.243916 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:18.243876    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0fe7254-bbf9-44ed-93b3-52df9ca26921-kube-api-access-7vh64" pod "f0fe7254-bbf9-44ed-93b3-52df9ca26921" (UID: "f0fe7254-bbf9-44ed-93b3-52df9ca26921"). InnerVolumeSpecName "kube-api-access-7vh64". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:12:18.337813 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:18.244148    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0fe7254-bbf9-44ed-93b3-52df9ca26921-maas-api-xks-ca" pod "f0fe7254-bbf9-44ed-93b3-52df9ca26921" (UID: "f0fe7254-bbf9-44ed-93b3-52df9ca26921"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:12:18.356753 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:18.347392    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0fe7254-bbf9-44ed-93b3-52df9ca26921-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:18.359602 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:18.357017    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0fe7254-bbf9-44ed-93b3-52df9ca26921-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:18.360029 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:18.360006    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vh64\" (UniqueName: \"kubernetes.io/projected/f0fe7254-bbf9-44ed-93b3-52df9ca26921-kube-api-access-7vh64\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:18.366697 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:18.366653    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6708ccfa-txvfl"
Oct 30 19:12:18.368886 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:18.368851    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28c95bf3-7b55d"]
Oct 30 19:12:18.370330 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:18.370309    4525 cpu_manager.go:414] "containerMap: removing container" podUID="f0fe7254-bbf9-44ed-93b3-52df9ca26921" containerName="revoke-keys"
Oct 30 19:12:18.370474 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:18.370458    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0fe7254-bbf9-44ed-93b3-52df9ca26921" containerName="revoke-keys"
Oct 30 19:12:18.370834 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:18.370818    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0fe7254-bbf9-44ed-93b3-52df9ca26921" containerName="revoke-keys"
Oct 30 19:12:18.375421 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:18.375397    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6708ccfa-txvfl" event={"ID":"f0fe7254-bbf9-44ed-93b3-52df9ca26921","Type":"ContainerDied","Data":"4d5a9afafb1b2994723a1bd869e3b5b1e3c666c484d696b24680adaedda10e6c"}
Oct 30 19:12:18.375596 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:18.375580    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d5a9afafb1b2994723a1bd869e3b5b1e3c666c484d696b24680adaedda10e6c"
Oct 30 19:12:18.375743 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:18.375730    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28c95bf3-7b55d"]
Oct 30 19:12:18.376477 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:18.375500    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28c95bf3-7b55d"
Oct 30 19:12:18.379664 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:18.379642    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:12:18.380047 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:18.380030    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94hcn\""
Oct 30 19:12:18.461333 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:18.461296    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/783874cd-5bd5-40f4-829a-842dd6d9cafd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28c95bf3-7b55d\" (UID: \"783874cd-5bd5-40f4-829a-842dd6d9cafd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28c95bf3-7b55d"
Oct 30 19:12:18.461603 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:18.461579    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/783874cd-5bd5-40f4-829a-842dd6d9cafd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28c95bf3-7b55d\" (UID: \"783874cd-5bd5-40f4-829a-842dd6d9cafd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28c95bf3-7b55d"
Oct 30 19:12:18.461843 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:18.461824    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlxct\" (UniqueName: \"kubernetes.io/projected/783874cd-5bd5-40f4-829a-842dd6d9cafd-kube-api-access-hlxct\") pod \"maas-api-revoke-sub-28c95bf3-7b55d\" (UID: \"783874cd-5bd5-40f4-829a-842dd6d9cafd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28c95bf3-7b55d"
Oct 30 19:12:18.564313 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:18.564274    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/783874cd-5bd5-40f4-829a-842dd6d9cafd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28c95bf3-7b55d\" (UID: \"783874cd-5bd5-40f4-829a-842dd6d9cafd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28c95bf3-7b55d"
Oct 30 19:12:18.565178 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:18.565149    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlxct\" (UniqueName: \"kubernetes.io/projected/783874cd-5bd5-40f4-829a-842dd6d9cafd-kube-api-access-hlxct\") pod \"maas-api-revoke-sub-28c95bf3-7b55d\" (UID: \"783874cd-5bd5-40f4-829a-842dd6d9cafd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28c95bf3-7b55d"
Oct 30 19:12:18.570621 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:18.570593    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/783874cd-5bd5-40f4-829a-842dd6d9cafd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28c95bf3-7b55d\" (UID: \"783874cd-5bd5-40f4-829a-842dd6d9cafd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28c95bf3-7b55d"
Oct 30 19:12:18.573495 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:18.573349    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/783874cd-5bd5-40f4-829a-842dd6d9cafd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28c95bf3-7b55d\" (UID: \"783874cd-5bd5-40f4-829a-842dd6d9cafd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28c95bf3-7b55d"
Oct 30 19:12:18.594331 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:18.592453    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/783874cd-5bd5-40f4-829a-842dd6d9cafd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28c95bf3-7b55d\" (UID: \"783874cd-5bd5-40f4-829a-842dd6d9cafd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28c95bf3-7b55d"
Oct 30 19:12:18.596355 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:18.596326    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlxct\" (UniqueName: \"kubernetes.io/projected/783874cd-5bd5-40f4-829a-842dd6d9cafd-kube-api-access-hlxct\") pod \"maas-api-revoke-sub-28c95bf3-7b55d\" (UID: \"783874cd-5bd5-40f4-829a-842dd6d9cafd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28c95bf3-7b55d"
Oct 30 19:12:18.714213 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:18.712863    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28c95bf3-7b55d"
Oct 30 19:12:19.121474 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:19.121448    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28c95bf3-7b55d"]
Oct 30 19:12:19.123618 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:12:19.123590    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod783874cd_5bd5_40f4_829a_842dd6d9cafd.slice/crio-a5239100675467dab487a0ae254facd1a94b84dfbef7f536a56f155418c4f132 WatchSource:0}: Error finding container a5239100675467dab487a0ae254facd1a94b84dfbef7f536a56f155418c4f132: Status 404 returned error can't find the container with id a5239100675467dab487a0ae254facd1a94b84dfbef7f536a56f155418c4f132
Oct 30 19:12:19.377322 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:19.377229    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28c95bf3-7b55d" event={"ID":"783874cd-5bd5-40f4-829a-842dd6d9cafd","Type":"ContainerStarted","Data":"a5239100675467dab487a0ae254facd1a94b84dfbef7f536a56f155418c4f132"}
Oct 30 19:12:20.389779 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:20.389744    4525 generic.go:356] "Generic (PLEG): container finished" podID="783874cd-5bd5-40f4-829a-842dd6d9cafd" containerID="8535394a809a648f1856e311f611c42f46389279f6e1bb1c4b926d9548528fa3" exitCode=0
Oct 30 19:12:20.390255 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:20.389826    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28c95bf3-7b55d" event={"ID":"783874cd-5bd5-40f4-829a-842dd6d9cafd","Type":"ContainerDied","Data":"8535394a809a648f1856e311f611c42f46389279f6e1bb1c4b926d9548528fa3"}
Oct 30 19:12:21.705959 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:21.705924    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28c95bf3-7b55d"
Oct 30 19:12:21.872300 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:21.872267    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/783874cd-5bd5-40f4-829a-842dd6d9cafd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/783874cd-5bd5-40f4-829a-842dd6d9cafd-maas-api-xks-ca\") pod \"783874cd-5bd5-40f4-829a-842dd6d9cafd\" (UID: \"783874cd-5bd5-40f4-829a-842dd6d9cafd\") "
Oct 30 19:12:21.872543 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:21.872530    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/783874cd-5bd5-40f4-829a-842dd6d9cafd-kube-api-access-hlxct\" (UniqueName: \"kubernetes.io/projected/783874cd-5bd5-40f4-829a-842dd6d9cafd-kube-api-access-hlxct\") pod \"783874cd-5bd5-40f4-829a-842dd6d9cafd\" (UID: \"783874cd-5bd5-40f4-829a-842dd6d9cafd\") "
Oct 30 19:12:21.872600 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:21.872585    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/783874cd-5bd5-40f4-829a-842dd6d9cafd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/783874cd-5bd5-40f4-829a-842dd6d9cafd-maas-api-openshift-ca\") pod \"783874cd-5bd5-40f4-829a-842dd6d9cafd\" (UID: \"783874cd-5bd5-40f4-829a-842dd6d9cafd\") "
Oct 30 19:12:21.873369 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:21.873037    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/783874cd-5bd5-40f4-829a-842dd6d9cafd-maas-api-openshift-ca" pod "783874cd-5bd5-40f4-829a-842dd6d9cafd" (UID: "783874cd-5bd5-40f4-829a-842dd6d9cafd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:12:21.873489 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:21.873457    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/783874cd-5bd5-40f4-829a-842dd6d9cafd-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:21.874860 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:21.874833    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/783874cd-5bd5-40f4-829a-842dd6d9cafd-maas-api-xks-ca" pod "783874cd-5bd5-40f4-829a-842dd6d9cafd" (UID: "783874cd-5bd5-40f4-829a-842dd6d9cafd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:12:21.874860 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:21.874841    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/783874cd-5bd5-40f4-829a-842dd6d9cafd-kube-api-access-hlxct" pod "783874cd-5bd5-40f4-829a-842dd6d9cafd" (UID: "783874cd-5bd5-40f4-829a-842dd6d9cafd"). InnerVolumeSpecName "kube-api-access-hlxct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:12:21.975774 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:21.975646    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/783874cd-5bd5-40f4-829a-842dd6d9cafd-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:21.975774 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:21.975699    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlxct\" (UniqueName: \"kubernetes.io/projected/783874cd-5bd5-40f4-829a-842dd6d9cafd-kube-api-access-hlxct\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:22.413100 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:22.413073    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28c95bf3-7b55d"
Oct 30 19:12:22.413308 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:22.413108    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28c95bf3-7b55d" event={"ID":"783874cd-5bd5-40f4-829a-842dd6d9cafd","Type":"ContainerDied","Data":"a5239100675467dab487a0ae254facd1a94b84dfbef7f536a56f155418c4f132"}
Oct 30 19:12:22.413308 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:22.413151    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5239100675467dab487a0ae254facd1a94b84dfbef7f536a56f155418c4f132"
Oct 30 19:12:22.681817 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:22.681737    4525 scope.go:122] "RemoveContainer" containerID="e65e6c6222d7bcc31fa1cea35167935ec59fbe28951d1bb9303a3ccd2a4660db"
Oct 30 19:12:22.716762 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:22.716741    4525 scope.go:122] "RemoveContainer" containerID="97ac4b5b7c2cfaeb460e82e1b6b0d8d1f984219b2a61082166c7e661b3b665fb"
Oct 30 19:12:25.663771 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:25.663732    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09072f0e-8bsjp"]
Oct 30 19:12:25.665516 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:25.665485    4525 cpu_manager.go:414] "containerMap: removing container" podUID="783874cd-5bd5-40f4-829a-842dd6d9cafd" containerName="revoke-keys"
Oct 30 19:12:25.666124 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:25.665620    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="783874cd-5bd5-40f4-829a-842dd6d9cafd" containerName="revoke-keys"
Oct 30 19:12:25.666124 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:25.665842    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="783874cd-5bd5-40f4-829a-842dd6d9cafd" containerName="revoke-keys"
Oct 30 19:12:25.996741 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:25.984414    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09072f0e-8bsjp"]
Oct 30 19:12:25.998361 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:25.986831    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09072f0e-8bsjp"
Oct 30 19:12:26.003562 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:26.003515    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:12:26.003992 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:26.003955    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94hcn\""
Oct 30 19:12:26.071803 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:26.071770    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e9005e6-75e2-4244-8f75-3a2c7eaeee08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09072f0e-8bsjp\" (UID: \"1e9005e6-75e2-4244-8f75-3a2c7eaeee08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09072f0e-8bsjp"
Oct 30 19:12:26.072547 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:26.072021    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9nr9\" (UniqueName: \"kubernetes.io/projected/1e9005e6-75e2-4244-8f75-3a2c7eaeee08-kube-api-access-s9nr9\") pod \"maas-api-revoke-sub-09072f0e-8bsjp\" (UID: \"1e9005e6-75e2-4244-8f75-3a2c7eaeee08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09072f0e-8bsjp"
Oct 30 19:12:26.072547 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:26.072055    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e9005e6-75e2-4244-8f75-3a2c7eaeee08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09072f0e-8bsjp\" (UID: \"1e9005e6-75e2-4244-8f75-3a2c7eaeee08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09072f0e-8bsjp"
Oct 30 19:12:26.173493 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:26.173454    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e9005e6-75e2-4244-8f75-3a2c7eaeee08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09072f0e-8bsjp\" (UID: \"1e9005e6-75e2-4244-8f75-3a2c7eaeee08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09072f0e-8bsjp"
Oct 30 19:12:26.173654 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:26.173622    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9nr9\" (UniqueName: \"kubernetes.io/projected/1e9005e6-75e2-4244-8f75-3a2c7eaeee08-kube-api-access-s9nr9\") pod \"maas-api-revoke-sub-09072f0e-8bsjp\" (UID: \"1e9005e6-75e2-4244-8f75-3a2c7eaeee08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09072f0e-8bsjp"
Oct 30 19:12:26.173654 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:26.173647    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e9005e6-75e2-4244-8f75-3a2c7eaeee08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09072f0e-8bsjp\" (UID: \"1e9005e6-75e2-4244-8f75-3a2c7eaeee08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09072f0e-8bsjp"
Oct 30 19:12:26.174376 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:26.174348    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e9005e6-75e2-4244-8f75-3a2c7eaeee08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09072f0e-8bsjp\" (UID: \"1e9005e6-75e2-4244-8f75-3a2c7eaeee08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09072f0e-8bsjp"
Oct 30 19:12:26.176082 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:26.176063    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e9005e6-75e2-4244-8f75-3a2c7eaeee08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09072f0e-8bsjp\" (UID: \"1e9005e6-75e2-4244-8f75-3a2c7eaeee08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09072f0e-8bsjp"
Oct 30 19:12:26.181668 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:26.181649    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9nr9\" (UniqueName: \"kubernetes.io/projected/1e9005e6-75e2-4244-8f75-3a2c7eaeee08-kube-api-access-s9nr9\") pod \"maas-api-revoke-sub-09072f0e-8bsjp\" (UID: \"1e9005e6-75e2-4244-8f75-3a2c7eaeee08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09072f0e-8bsjp"
Oct 30 19:12:26.334060 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:26.333974    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09072f0e-8bsjp"
Oct 30 19:12:26.745085 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:26.744657    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09072f0e-8bsjp"]
Oct 30 19:12:27.872370 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:27.872318    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09072f0e-8bsjp" event={"ID":"1e9005e6-75e2-4244-8f75-3a2c7eaeee08","Type":"ContainerStarted","Data":"9b5893c5e59d54301b4ecb3892e96a125537d808ef6b62980a281faffc11aa78"}
Oct 30 19:12:27.873077 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:27.873048    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09072f0e-8bsjp" event={"ID":"1e9005e6-75e2-4244-8f75-3a2c7eaeee08","Type":"ContainerStarted","Data":"e331292b250b8e56dae488dc27fe6fb4a9e04902f57f844a611b901ae025d6f3"}
Oct 30 19:12:28.883168 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:28.883131    4525 generic.go:356] "Generic (PLEG): container finished" podID="1e9005e6-75e2-4244-8f75-3a2c7eaeee08" containerID="9b5893c5e59d54301b4ecb3892e96a125537d808ef6b62980a281faffc11aa78" exitCode=0
Oct 30 19:12:28.883604 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:28.883217    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09072f0e-8bsjp" event={"ID":"1e9005e6-75e2-4244-8f75-3a2c7eaeee08","Type":"ContainerDied","Data":"9b5893c5e59d54301b4ecb3892e96a125537d808ef6b62980a281faffc11aa78"}
Oct 30 19:12:30.105480 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:30.105453    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09072f0e-8bsjp"
Oct 30 19:12:30.208325 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:30.208243    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e9005e6-75e2-4244-8f75-3a2c7eaeee08-kube-api-access-s9nr9\" (UniqueName: \"kubernetes.io/projected/1e9005e6-75e2-4244-8f75-3a2c7eaeee08-kube-api-access-s9nr9\") pod \"1e9005e6-75e2-4244-8f75-3a2c7eaeee08\" (UID: \"1e9005e6-75e2-4244-8f75-3a2c7eaeee08\") "
Oct 30 19:12:30.208485 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:30.208355    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e9005e6-75e2-4244-8f75-3a2c7eaeee08-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e9005e6-75e2-4244-8f75-3a2c7eaeee08-maas-api-xks-ca\") pod \"1e9005e6-75e2-4244-8f75-3a2c7eaeee08\" (UID: \"1e9005e6-75e2-4244-8f75-3a2c7eaeee08\") "
Oct 30 19:12:30.208485 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:30.208473    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e9005e6-75e2-4244-8f75-3a2c7eaeee08-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e9005e6-75e2-4244-8f75-3a2c7eaeee08-maas-api-openshift-ca\") pod \"1e9005e6-75e2-4244-8f75-3a2c7eaeee08\" (UID: \"1e9005e6-75e2-4244-8f75-3a2c7eaeee08\") "
Oct 30 19:12:30.208840 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:30.208814    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e9005e6-75e2-4244-8f75-3a2c7eaeee08-maas-api-openshift-ca" pod "1e9005e6-75e2-4244-8f75-3a2c7eaeee08" (UID: "1e9005e6-75e2-4244-8f75-3a2c7eaeee08"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:12:30.209275 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:30.209257    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e9005e6-75e2-4244-8f75-3a2c7eaeee08-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:30.210555 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:30.210525    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e9005e6-75e2-4244-8f75-3a2c7eaeee08-kube-api-access-s9nr9" pod "1e9005e6-75e2-4244-8f75-3a2c7eaeee08" (UID: "1e9005e6-75e2-4244-8f75-3a2c7eaeee08"). InnerVolumeSpecName "kube-api-access-s9nr9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:12:30.210666 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:30.210647    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e9005e6-75e2-4244-8f75-3a2c7eaeee08-maas-api-xks-ca" pod "1e9005e6-75e2-4244-8f75-3a2c7eaeee08" (UID: "1e9005e6-75e2-4244-8f75-3a2c7eaeee08"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:12:30.338722 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:30.328115    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9nr9\" (UniqueName: \"kubernetes.io/projected/1e9005e6-75e2-4244-8f75-3a2c7eaeee08-kube-api-access-s9nr9\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:30.338722 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:30.328161    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e9005e6-75e2-4244-8f75-3a2c7eaeee08-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:30.916346 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:30.916314    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09072f0e-8bsjp"
Oct 30 19:12:30.916551 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:30.916341    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09072f0e-8bsjp" event={"ID":"1e9005e6-75e2-4244-8f75-3a2c7eaeee08","Type":"ContainerDied","Data":"e331292b250b8e56dae488dc27fe6fb4a9e04902f57f844a611b901ae025d6f3"}
Oct 30 19:12:30.916551 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:30.916404    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e331292b250b8e56dae488dc27fe6fb4a9e04902f57f844a611b901ae025d6f3"
Oct 30 19:12:36.025213 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:36.025175    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cf3c4bb-4zdf4"]
Oct 30 19:12:36.030322 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:36.030292    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cf3c4bb-4zdf4"]
Oct 30 19:12:37.646361 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:37.646327    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b963374-1226-411e-87b0-642297fad691" path="/var/lib/kubelet/pods/6b963374-1226-411e-87b0-642297fad691/volumes"
Oct 30 19:12:42.626623 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:42.626582    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b16e2a6-7z2nj"]
Oct 30 19:12:42.627819 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:42.627422    4525 cpu_manager.go:414] "containerMap: removing container" podUID="1e9005e6-75e2-4244-8f75-3a2c7eaeee08" containerName="revoke-keys"
Oct 30 19:12:42.627819 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:42.627437    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e9005e6-75e2-4244-8f75-3a2c7eaeee08" containerName="revoke-keys"
Oct 30 19:12:42.627819 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:42.627557    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e9005e6-75e2-4244-8f75-3a2c7eaeee08" containerName="revoke-keys"
Oct 30 19:12:42.631390 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:42.631369    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b16e2a6-7z2nj"
Oct 30 19:12:42.637052 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:42.637029    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:12:42.638658 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:42.637530    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94hcn\""
Oct 30 19:12:42.640311 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:42.640286    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b16e2a6-7z2nj"]
Oct 30 19:12:42.772655 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:42.772616    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkrgt\" (UniqueName: \"kubernetes.io/projected/85f38f2b-bce9-4d9f-9cfb-a0a6222f8d78-kube-api-access-bkrgt\") pod \"maas-api-revoke-sub-7b16e2a6-7z2nj\" (UID: \"85f38f2b-bce9-4d9f-9cfb-a0a6222f8d78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b16e2a6-7z2nj"
Oct 30 19:12:42.772655 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:42.772659    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85f38f2b-bce9-4d9f-9cfb-a0a6222f8d78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b16e2a6-7z2nj\" (UID: \"85f38f2b-bce9-4d9f-9cfb-a0a6222f8d78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b16e2a6-7z2nj"
Oct 30 19:12:42.772974 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:42.772718    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85f38f2b-bce9-4d9f-9cfb-a0a6222f8d78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b16e2a6-7z2nj\" (UID: \"85f38f2b-bce9-4d9f-9cfb-a0a6222f8d78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b16e2a6-7z2nj"
Oct 30 19:12:42.874615 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:42.874578    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkrgt\" (UniqueName: \"kubernetes.io/projected/85f38f2b-bce9-4d9f-9cfb-a0a6222f8d78-kube-api-access-bkrgt\") pod \"maas-api-revoke-sub-7b16e2a6-7z2nj\" (UID: \"85f38f2b-bce9-4d9f-9cfb-a0a6222f8d78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b16e2a6-7z2nj"
Oct 30 19:12:42.874615 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:42.874619    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85f38f2b-bce9-4d9f-9cfb-a0a6222f8d78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b16e2a6-7z2nj\" (UID: \"85f38f2b-bce9-4d9f-9cfb-a0a6222f8d78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b16e2a6-7z2nj"
Oct 30 19:12:42.874884 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:42.874662    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85f38f2b-bce9-4d9f-9cfb-a0a6222f8d78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b16e2a6-7z2nj\" (UID: \"85f38f2b-bce9-4d9f-9cfb-a0a6222f8d78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b16e2a6-7z2nj"
Oct 30 19:12:42.875335 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:42.875304    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85f38f2b-bce9-4d9f-9cfb-a0a6222f8d78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b16e2a6-7z2nj\" (UID: \"85f38f2b-bce9-4d9f-9cfb-a0a6222f8d78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b16e2a6-7z2nj"
Oct 30 19:12:42.877415 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:42.877359    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85f38f2b-bce9-4d9f-9cfb-a0a6222f8d78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b16e2a6-7z2nj\" (UID: \"85f38f2b-bce9-4d9f-9cfb-a0a6222f8d78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b16e2a6-7z2nj"
Oct 30 19:12:42.883225 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:42.883203    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkrgt\" (UniqueName: \"kubernetes.io/projected/85f38f2b-bce9-4d9f-9cfb-a0a6222f8d78-kube-api-access-bkrgt\") pod \"maas-api-revoke-sub-7b16e2a6-7z2nj\" (UID: \"85f38f2b-bce9-4d9f-9cfb-a0a6222f8d78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b16e2a6-7z2nj"
Oct 30 19:12:42.957830 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:42.957792    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b16e2a6-7z2nj"
Oct 30 19:12:43.274765 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:43.273587    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b16e2a6-7z2nj"]
Oct 30 19:12:44.094144 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:44.094100    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b16e2a6-7z2nj" event={"ID":"85f38f2b-bce9-4d9f-9cfb-a0a6222f8d78","Type":"ContainerStarted","Data":"5238ec5221253566eb50004f3a20db3418f46abaa2a8c7737bae9f1e6d0bfcd5"}
Oct 30 19:12:44.094562 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:44.094166    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b16e2a6-7z2nj" event={"ID":"85f38f2b-bce9-4d9f-9cfb-a0a6222f8d78","Type":"ContainerStarted","Data":"971c3bd6dc7e034d6a27b53d1154916caa9ab8f28b31cb00ec3288fb9d14dd75"}
Oct 30 19:12:44.117289 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:44.117234    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b16e2a6-7z2nj" podStartSLOduration=2.11722164 podStartE2EDuration="2.11722164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:12:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:12:44.111291785 +0000 UTC m=+2186.736961754" watchObservedRunningTime="2026-10-30 19:12:44.11722164 +0000 UTC m=+2186.742891604"
Oct 30 19:12:45.205706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:45.200044    4525 generic.go:356] "Generic (PLEG): container finished" podID="85f38f2b-bce9-4d9f-9cfb-a0a6222f8d78" containerID="5238ec5221253566eb50004f3a20db3418f46abaa2a8c7737bae9f1e6d0bfcd5" exitCode=0
Oct 30 19:12:45.205706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:45.200159    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b16e2a6-7z2nj" event={"ID":"85f38f2b-bce9-4d9f-9cfb-a0a6222f8d78","Type":"ContainerDied","Data":"5238ec5221253566eb50004f3a20db3418f46abaa2a8c7737bae9f1e6d0bfcd5"}
Oct 30 19:12:46.490102 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:46.490067    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b16e2a6-7z2nj"
Oct 30 19:12:46.521824 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:46.521766    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85f38f2b-bce9-4d9f-9cfb-a0a6222f8d78-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85f38f2b-bce9-4d9f-9cfb-a0a6222f8d78-maas-api-openshift-ca\") pod \"85f38f2b-bce9-4d9f-9cfb-a0a6222f8d78\" (UID: \"85f38f2b-bce9-4d9f-9cfb-a0a6222f8d78\") "
Oct 30 19:12:46.522028 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:46.521966    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85f38f2b-bce9-4d9f-9cfb-a0a6222f8d78-kube-api-access-bkrgt\" (UniqueName: \"kubernetes.io/projected/85f38f2b-bce9-4d9f-9cfb-a0a6222f8d78-kube-api-access-bkrgt\") pod \"85f38f2b-bce9-4d9f-9cfb-a0a6222f8d78\" (UID: \"85f38f2b-bce9-4d9f-9cfb-a0a6222f8d78\") "
Oct 30 19:12:46.523018 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:46.522236    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85f38f2b-bce9-4d9f-9cfb-a0a6222f8d78-maas-api-openshift-ca" pod "85f38f2b-bce9-4d9f-9cfb-a0a6222f8d78" (UID: "85f38f2b-bce9-4d9f-9cfb-a0a6222f8d78"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:12:46.523018 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:46.522874    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85f38f2b-bce9-4d9f-9cfb-a0a6222f8d78-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:46.526777 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:46.526747    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85f38f2b-bce9-4d9f-9cfb-a0a6222f8d78-kube-api-access-bkrgt" pod "85f38f2b-bce9-4d9f-9cfb-a0a6222f8d78" (UID: "85f38f2b-bce9-4d9f-9cfb-a0a6222f8d78"). InnerVolumeSpecName "kube-api-access-bkrgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:12:46.646710 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:46.645611    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85f38f2b-bce9-4d9f-9cfb-a0a6222f8d78-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85f38f2b-bce9-4d9f-9cfb-a0a6222f8d78-maas-api-xks-ca\") pod \"85f38f2b-bce9-4d9f-9cfb-a0a6222f8d78\" (UID: \"85f38f2b-bce9-4d9f-9cfb-a0a6222f8d78\") "
Oct 30 19:12:46.646710 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:46.646628    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkrgt\" (UniqueName: \"kubernetes.io/projected/85f38f2b-bce9-4d9f-9cfb-a0a6222f8d78-kube-api-access-bkrgt\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:46.738201 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:46.736903    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85f38f2b-bce9-4d9f-9cfb-a0a6222f8d78-maas-api-xks-ca" pod "85f38f2b-bce9-4d9f-9cfb-a0a6222f8d78" (UID: "85f38f2b-bce9-4d9f-9cfb-a0a6222f8d78"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:12:46.759161 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:46.759111    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85f38f2b-bce9-4d9f-9cfb-a0a6222f8d78-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:47.054874 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:47.054824    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf63472c-v2vbc"]
Oct 30 19:12:47.061802 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:47.061773    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf63472c-v2vbc"]
Oct 30 19:12:47.296469 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:47.296426    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b16e2a6-7z2nj"
Oct 30 19:12:47.296648 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:47.296461    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b16e2a6-7z2nj" event={"ID":"85f38f2b-bce9-4d9f-9cfb-a0a6222f8d78","Type":"ContainerDied","Data":"971c3bd6dc7e034d6a27b53d1154916caa9ab8f28b31cb00ec3288fb9d14dd75"}
Oct 30 19:12:47.296648 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:47.296514    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="971c3bd6dc7e034d6a27b53d1154916caa9ab8f28b31cb00ec3288fb9d14dd75"
Oct 30 19:12:47.405734 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:12:47.404275    4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d49l" containerName="extractor"
Oct 30 19:12:47.549325 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:12:47.549228    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="363cd59e-8ddd-43a6-91e9-f2c1b577948b" path="/var/lib/kubelet/pods/363cd59e-8ddd-43a6-91e9-f2c1b577948b/volumes"
Oct 30 19:13:02.173578 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:02.173495    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7501d6cf-xbk8t"]
Oct 30 19:13:02.174252 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:02.174236    4525 cpu_manager.go:414] "containerMap: removing container" podUID="85f38f2b-bce9-4d9f-9cfb-a0a6222f8d78" containerName="revoke-keys"
Oct 30 19:13:02.174299 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:02.174253    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85f38f2b-bce9-4d9f-9cfb-a0a6222f8d78" containerName="revoke-keys"
Oct 30 19:13:02.174410 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:02.174401    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="85f38f2b-bce9-4d9f-9cfb-a0a6222f8d78" containerName="revoke-keys"
Oct 30 19:13:02.179641 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:02.179621    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7501d6cf-xbk8t"
Oct 30 19:13:02.260711 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:02.259787    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7501d6cf-xbk8t"]
Oct 30 19:13:02.285057 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:02.285010    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94hcn\""
Oct 30 19:13:02.311734 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:02.287862    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:13:02.313415 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:02.313222    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6jtc\" (UniqueName: \"kubernetes.io/projected/1f526772-60e6-44cf-a476-de704cd4e3e9-kube-api-access-m6jtc\") pod \"maas-api-revoke-sub-7501d6cf-xbk8t\" (UID: \"1f526772-60e6-44cf-a476-de704cd4e3e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7501d6cf-xbk8t"
Oct 30 19:13:02.313915 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:02.313878    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f526772-60e6-44cf-a476-de704cd4e3e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7501d6cf-xbk8t\" (UID: \"1f526772-60e6-44cf-a476-de704cd4e3e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7501d6cf-xbk8t"
Oct 30 19:13:02.314517 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:02.314481    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f526772-60e6-44cf-a476-de704cd4e3e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7501d6cf-xbk8t\" (UID: \"1f526772-60e6-44cf-a476-de704cd4e3e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7501d6cf-xbk8t"
Oct 30 19:13:02.416388 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:02.416354    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6jtc\" (UniqueName: \"kubernetes.io/projected/1f526772-60e6-44cf-a476-de704cd4e3e9-kube-api-access-m6jtc\") pod \"maas-api-revoke-sub-7501d6cf-xbk8t\" (UID: \"1f526772-60e6-44cf-a476-de704cd4e3e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7501d6cf-xbk8t"
Oct 30 19:13:02.416580 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:02.416458    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f526772-60e6-44cf-a476-de704cd4e3e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7501d6cf-xbk8t\" (UID: \"1f526772-60e6-44cf-a476-de704cd4e3e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7501d6cf-xbk8t"
Oct 30 19:13:02.416655 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:02.416622    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f526772-60e6-44cf-a476-de704cd4e3e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7501d6cf-xbk8t\" (UID: \"1f526772-60e6-44cf-a476-de704cd4e3e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7501d6cf-xbk8t"
Oct 30 19:13:02.417244 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:02.417222    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f526772-60e6-44cf-a476-de704cd4e3e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7501d6cf-xbk8t\" (UID: \"1f526772-60e6-44cf-a476-de704cd4e3e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7501d6cf-xbk8t"
Oct 30 19:13:02.419154 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:02.419119    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f526772-60e6-44cf-a476-de704cd4e3e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7501d6cf-xbk8t\" (UID: \"1f526772-60e6-44cf-a476-de704cd4e3e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7501d6cf-xbk8t"
Oct 30 19:13:02.424620 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:02.424565    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6jtc\" (UniqueName: \"kubernetes.io/projected/1f526772-60e6-44cf-a476-de704cd4e3e9-kube-api-access-m6jtc\") pod \"maas-api-revoke-sub-7501d6cf-xbk8t\" (UID: \"1f526772-60e6-44cf-a476-de704cd4e3e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7501d6cf-xbk8t"
Oct 30 19:13:02.529435 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:02.529394    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7501d6cf-xbk8t"
Oct 30 19:13:02.815639 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:13:02.815607    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f526772_60e6_44cf_a476_de704cd4e3e9.slice/crio-677108465bc78ed625440683f213e680fd92ca55637624887ea6930c484e0843 WatchSource:0}: Error finding container 677108465bc78ed625440683f213e680fd92ca55637624887ea6930c484e0843: Status 404 returned error can't find the container with id 677108465bc78ed625440683f213e680fd92ca55637624887ea6930c484e0843
Oct 30 19:13:02.817422 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:02.817398    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7501d6cf-xbk8t"]
Oct 30 19:13:03.725467 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:03.725416    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7501d6cf-xbk8t" event={"ID":"1f526772-60e6-44cf-a476-de704cd4e3e9","Type":"ContainerStarted","Data":"8509ef207b63c1ec00f3b4c296da65415e618169653d2494cba3b10c7bb56063"}
Oct 30 19:13:03.744707 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:03.738081    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7501d6cf-xbk8t" event={"ID":"1f526772-60e6-44cf-a476-de704cd4e3e9","Type":"ContainerStarted","Data":"677108465bc78ed625440683f213e680fd92ca55637624887ea6930c484e0843"}
Oct 30 19:13:03.799750 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:03.792954    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7501d6cf-xbk8t" podStartSLOduration=1.792931486 podStartE2EDuration="1.792931486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:13:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:13:03.787935313 +0000 UTC m=+2206.413605278" watchObservedRunningTime="2026-10-30 19:13:03.792931486 +0000 UTC m=+2206.418601451"
Oct 30 19:13:04.737744 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:04.737707    4525 generic.go:356] "Generic (PLEG): container finished" podID="1f526772-60e6-44cf-a476-de704cd4e3e9" containerID="8509ef207b63c1ec00f3b4c296da65415e618169653d2494cba3b10c7bb56063" exitCode=0
Oct 30 19:13:04.738257 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:04.737808    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7501d6cf-xbk8t" event={"ID":"1f526772-60e6-44cf-a476-de704cd4e3e9","Type":"ContainerDied","Data":"8509ef207b63c1ec00f3b4c296da65415e618169653d2494cba3b10c7bb56063"}
Oct 30 19:13:06.023310 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:06.023278    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7501d6cf-xbk8t"
Oct 30 19:13:06.124647 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:06.124600    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f526772-60e6-44cf-a476-de704cd4e3e9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f526772-60e6-44cf-a476-de704cd4e3e9-maas-api-openshift-ca\") pod \"1f526772-60e6-44cf-a476-de704cd4e3e9\" (UID: \"1f526772-60e6-44cf-a476-de704cd4e3e9\") "
Oct 30 19:13:06.124887 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:06.124760    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f526772-60e6-44cf-a476-de704cd4e3e9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f526772-60e6-44cf-a476-de704cd4e3e9-maas-api-xks-ca\") pod \"1f526772-60e6-44cf-a476-de704cd4e3e9\" (UID: \"1f526772-60e6-44cf-a476-de704cd4e3e9\") "
Oct 30 19:13:06.124887 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:06.124836    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f526772-60e6-44cf-a476-de704cd4e3e9-kube-api-access-m6jtc\" (UniqueName: \"kubernetes.io/projected/1f526772-60e6-44cf-a476-de704cd4e3e9-kube-api-access-m6jtc\") pod \"1f526772-60e6-44cf-a476-de704cd4e3e9\" (UID: \"1f526772-60e6-44cf-a476-de704cd4e3e9\") "
Oct 30 19:13:06.125117 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:06.125087    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f526772-60e6-44cf-a476-de704cd4e3e9-maas-api-openshift-ca" pod "1f526772-60e6-44cf-a476-de704cd4e3e9" (UID: "1f526772-60e6-44cf-a476-de704cd4e3e9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:13:06.125533 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:06.125512    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f526772-60e6-44cf-a476-de704cd4e3e9-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:06.127283 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:06.127232    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f526772-60e6-44cf-a476-de704cd4e3e9-kube-api-access-m6jtc" pod "1f526772-60e6-44cf-a476-de704cd4e3e9" (UID: "1f526772-60e6-44cf-a476-de704cd4e3e9"). InnerVolumeSpecName "kube-api-access-m6jtc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:13:06.127622 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:06.127600    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f526772-60e6-44cf-a476-de704cd4e3e9-maas-api-xks-ca" pod "1f526772-60e6-44cf-a476-de704cd4e3e9" (UID: "1f526772-60e6-44cf-a476-de704cd4e3e9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:13:06.226986 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:06.226935    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f526772-60e6-44cf-a476-de704cd4e3e9-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:06.226986 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:06.226981    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6jtc\" (UniqueName: \"kubernetes.io/projected/1f526772-60e6-44cf-a476-de704cd4e3e9-kube-api-access-m6jtc\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:06.815045 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:06.815013    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7501d6cf-xbk8t"
Oct 30 19:13:06.815235 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:06.815045    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7501d6cf-xbk8t" event={"ID":"1f526772-60e6-44cf-a476-de704cd4e3e9","Type":"ContainerDied","Data":"677108465bc78ed625440683f213e680fd92ca55637624887ea6930c484e0843"}
Oct 30 19:13:06.815235 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:06.815092    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="677108465bc78ed625440683f213e680fd92ca55637624887ea6930c484e0843"
Oct 30 19:13:09.823278 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:09.822309    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65"]
Oct 30 19:13:09.823278 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:09.822603    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65" podUID="391c7032-c25c-47c3-925f-35ea421e07c5" containerName="istio-proxy" containerID="cri-o://b89bf1225edd646a777d562500d7c8f895cb29ea21cc764e998b9afa96b14f03" gracePeriod=30
Oct 30 19:13:10.015705 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:09.878815    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts" podUID="be6d4969-285b-4da0-883e-4847f2cbf3af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:13:10.413710 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:10.410658    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr" podUID="82bbbf85-ed93-48d2-9f17-49ac1f31c0a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 30 19:13:10.413710 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:10.411180    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj" podUID="84c99c49-1d60-4176-bda4-cad29c7d801c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Oct 30 19:13:10.413710 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:10.412211    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw" podUID="ac6852fd-01be-4d69-8c9b-0c591e50b889" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:13:10.456863 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:10.456808    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t" podUID="424fca14-6d8a-4663-86ec-0c91165f34ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:13:10.761338 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:10.758143    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb" podUID="12ebc8b9-c794-47b5-b87b-a63c4bd993e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:13:10.789939 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:10.769142    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv" podUID="595230e3-55f3-43eb-9a7b-6d89250af9b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:13:10.911504 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:10.910171    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts" podUID="be6d4969-285b-4da0-883e-4847f2cbf3af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Oct 30 19:13:10.932943 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:10.932637    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts"
Oct 30 19:13:11.361703 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:11.346764    4525 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv" podUID="595230e3-55f3-43eb-9a7b-6d89250af9b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:13:11.453732 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:11.452534    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw" podUID="ac6852fd-01be-4d69-8c9b-0c591e50b889" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:13:11.453732 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:11.453638    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw"
Oct 30 19:13:11.457246 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:11.457072    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj" podUID="84c99c49-1d60-4176-bda4-cad29c7d801c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:13:11.458277 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:11.458257    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj"
Oct 30 19:13:11.484721 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:11.484670    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts"
Oct 30 19:13:11.541430 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:11.541379    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw"
Oct 30 19:13:11.542483 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:11.542286    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj"
Oct 30 19:13:12.937435 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:12.937400    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e703ad97-92hq4"]
Oct 30 19:13:12.938226 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:12.938208    4525 cpu_manager.go:414] "containerMap: removing container" podUID="1f526772-60e6-44cf-a476-de704cd4e3e9" containerName="revoke-keys"
Oct 30 19:13:12.938280 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:12.938227    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f526772-60e6-44cf-a476-de704cd4e3e9" containerName="revoke-keys"
Oct 30 19:13:12.938380 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:12.938371    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f526772-60e6-44cf-a476-de704cd4e3e9" containerName="revoke-keys"
Oct 30 19:13:12.942119 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:12.942102    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e703ad97-92hq4"
Oct 30 19:13:12.947608 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:12.946213    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94hcn\""
Oct 30 19:13:12.947608 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:12.946192    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:13:12.950889 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:12.950859    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e703ad97-92hq4"]
Oct 30 19:13:12.956759 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:12.956728    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b46ae00-fd8c-4575-b843-7ca560d0f12c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e703ad97-92hq4\" (UID: \"2b46ae00-fd8c-4575-b843-7ca560d0f12c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e703ad97-92hq4"
Oct 30 19:13:12.956918 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:12.956892    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b46ae00-fd8c-4575-b843-7ca560d0f12c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e703ad97-92hq4\" (UID: \"2b46ae00-fd8c-4575-b843-7ca560d0f12c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e703ad97-92hq4"
Oct 30 19:13:12.957054 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:12.957037    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5t8fw\" (UniqueName: \"kubernetes.io/projected/2b46ae00-fd8c-4575-b843-7ca560d0f12c-kube-api-access-5t8fw\") pod \"maas-api-revoke-sub-e703ad97-92hq4\" (UID: \"2b46ae00-fd8c-4575-b843-7ca560d0f12c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e703ad97-92hq4"
Oct 30 19:13:13.058742 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:13.058704    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b46ae00-fd8c-4575-b843-7ca560d0f12c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e703ad97-92hq4\" (UID: \"2b46ae00-fd8c-4575-b843-7ca560d0f12c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e703ad97-92hq4"
Oct 30 19:13:13.059620 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:13.058801    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5t8fw\" (UniqueName: \"kubernetes.io/projected/2b46ae00-fd8c-4575-b843-7ca560d0f12c-kube-api-access-5t8fw\") pod \"maas-api-revoke-sub-e703ad97-92hq4\" (UID: \"2b46ae00-fd8c-4575-b843-7ca560d0f12c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e703ad97-92hq4"
Oct 30 19:13:13.059620 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:13.058954    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b46ae00-fd8c-4575-b843-7ca560d0f12c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e703ad97-92hq4\" (UID: \"2b46ae00-fd8c-4575-b843-7ca560d0f12c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e703ad97-92hq4"
Oct 30 19:13:13.059620 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:13.059307    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b46ae00-fd8c-4575-b843-7ca560d0f12c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e703ad97-92hq4\" (UID: \"2b46ae00-fd8c-4575-b843-7ca560d0f12c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e703ad97-92hq4"
Oct 30 19:13:13.061524 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:13.061499    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b46ae00-fd8c-4575-b843-7ca560d0f12c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e703ad97-92hq4\" (UID: \"2b46ae00-fd8c-4575-b843-7ca560d0f12c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e703ad97-92hq4"
Oct 30 19:13:13.067409 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:13.067386    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t8fw\" (UniqueName: \"kubernetes.io/projected/2b46ae00-fd8c-4575-b843-7ca560d0f12c-kube-api-access-5t8fw\") pod \"maas-api-revoke-sub-e703ad97-92hq4\" (UID: \"2b46ae00-fd8c-4575-b843-7ca560d0f12c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e703ad97-92hq4"
Oct 30 19:13:13.292120 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:13.292028    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e703ad97-92hq4"
Oct 30 19:13:13.726591 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:13.726561    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e703ad97-92hq4"]
Oct 30 19:13:13.729830 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:13:13.729802    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b46ae00_fd8c_4575_b843_7ca560d0f12c.slice/crio-53d3d7cb7576336827abe38b8a9b6bf548b1b1f50d7c72df326fcc4bcf265db9 WatchSource:0}: Error finding container 53d3d7cb7576336827abe38b8a9b6bf548b1b1f50d7c72df326fcc4bcf265db9: Status 404 returned error can't find the container with id 53d3d7cb7576336827abe38b8a9b6bf548b1b1f50d7c72df326fcc4bcf265db9
Oct 30 19:13:14.818551 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:14.818507    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e703ad97-92hq4" event={"ID":"2b46ae00-fd8c-4575-b843-7ca560d0f12c","Type":"ContainerStarted","Data":"537f0c8c39c9527732edc6d6f161d66f89e01d1f7ad124d0907c52b8824d22a8"}
Oct 30 19:13:14.820045 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:14.818634    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e703ad97-92hq4" event={"ID":"2b46ae00-fd8c-4575-b843-7ca560d0f12c","Type":"ContainerStarted","Data":"53d3d7cb7576336827abe38b8a9b6bf548b1b1f50d7c72df326fcc4bcf265db9"}
Oct 30 19:13:14.837750 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:14.837667    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e703ad97-92hq4" podStartSLOduration=2.837654864 podStartE2EDuration="2.837654864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:13:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:13:14.835672033 +0000 UTC m=+2217.461341998" watchObservedRunningTime="2026-10-30 19:13:14.837654864 +0000 UTC m=+2217.463324827"
Oct 30 19:13:15.424981 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:15.424938    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-qv6r8"]
Oct 30 19:13:15.427840 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:15.425181    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-qv6r8" podUID="36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b" containerName="istio-proxy" containerID="cri-o://32ecebc6ea5d4f7b9a8c28cd0fe1175887c16eabc9e5ce09032bd44d03278131" gracePeriod=30
Oct 30 19:13:15.658400 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:15.658369    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65"
Oct 30 19:13:15.727823 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:15.727790    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-f3d6a5/payload-processing-external-model-e2e-extmod-b-f3d6a5-f7d7npjtw"]
Oct 30 19:13:15.728610 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:15.728564    4525 cpu_manager.go:414] "containerMap: removing container" podUID="391c7032-c25c-47c3-925f-35ea421e07c5" containerName="istio-proxy"
Oct 30 19:13:15.728610 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:15.728578    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="391c7032-c25c-47c3-925f-35ea421e07c5" containerName="istio-proxy"
Oct 30 19:13:15.728844 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:15.728705    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="391c7032-c25c-47c3-925f-35ea421e07c5" containerName="istio-proxy"
Oct 30 19:13:15.734429 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:15.734402    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-1dd09d/payload-processing-external-model-e2e-extmod-a-1dd09d-5445l9g92"]
Oct 30 19:13:15.736704 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:15.736656    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f3d6a5/payload-processing-external-model-e2e-extmod-b-f3d6a5-f7d7npjtw"
Oct 30 19:13:15.740131 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:15.740110    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-1dd09d/payload-processing-external-model-e2e-extmod-a-1dd09d-5445l9g92"
Oct 30 19:13:15.742843 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:15.742813    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-1dd09d\"/\"openshift-service-ca.crt\""
Oct 30 19:13:15.742991 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:15.742976    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-f3d6a5\"/\"agc-cred-b58707117777\""
Oct 30 19:13:15.743081 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:15.743073    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-f3d6a5\"/\"payload-processing-external-model-e2e--eb883441-dockercfg-2ql8r\""
Oct 30 19:13:15.743205 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:15.743184    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-1dd09d\"/\"payload-processing-external-model-e2e--6f690741-dockercfg-lzt76\""
Oct 30 19:13:15.743520 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:15.743501    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-f3d6a5\"/\"openshift-service-ca.crt\""
Oct 30 19:13:15.743603 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:15.743562    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-f3d6a5\"/\"routing-overlay\""
Oct 30 19:13:15.743603 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:15.743601    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-f3d6a5\"/\"payload-processing-external-model-plugins-e2e-extmod-b-f3d6a5\""
Oct 30 19:13:15.743725 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:15.743657    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-f3d6a5\"/\"kube-root-ca.crt\""
Oct 30 19:13:15.743725 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:15.743709    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-f3d6a5/payload-processing-external-model-e2e-extmod-b-f3d6a5-f7d7npjtw"]
Oct 30 19:13:15.751023 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:15.750996    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-1dd09d/payload-processing-external-model-e2e-extmod-a-1dd09d-5445l9g92"]
Oct 30 19:13:15.751472 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:15.751451    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-1dd09d\"/\"kube-root-ca.crt\""
Oct 30 19:13:15.751725 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:15.751658    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-1dd09d\"/\"payload-processing-external-model-plugins-e2e-extmod-a-1dd09d\""
Oct 30 19:13:15.751923 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:15.751900    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-1dd09d\"/\"routing-overlay\""
Oct 30 19:13:15.752024 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:15.751913    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-1dd09d\"/\"e2e-cmp-legacy-c5c176-provider-cred\""
Oct 30 19:13:15.832543 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:15.832505    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/391c7032-c25c-47c3-925f-35ea421e07c5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/391c7032-c25c-47c3-925f-35ea421e07c5-workload-certs\") pod \"391c7032-c25c-47c3-925f-35ea421e07c5\" (UID: \"391c7032-c25c-47c3-925f-35ea421e07c5\") "
Oct 30 19:13:15.833139 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:15.832588    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/391c7032-c25c-47c3-925f-35ea421e07c5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/391c7032-c25c-47c3-925f-35ea421e07c5-credential-socket\") pod \"391c7032-c25c-47c3-925f-35ea421e07c5\" (UID: \"391c7032-c25c-47c3-925f-35ea421e07c5\") "
Oct 30 19:13:15.833139 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:15.832798    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/391c7032-c25c-47c3-925f-35ea421e07c5-kube-api-access-4t8n4\" (UniqueName: \"kubernetes.io/projected/391c7032-c25c-47c3-925f-35ea421e07c5-kube-api-access-4t8n4\") pod \"391c7032-c25c-47c3-925f-35ea421e07c5\" (UID: \"391c7032-c25c-47c3-925f-35ea421e07c5\") "
Oct 30 19:13:15.833139 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:15.832888    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/391c7032-c25c-47c3-925f-35ea421e07c5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/391c7032-c25c-47c3-925f-35ea421e07c5-istiod-ca-cert\") pod \"391c7032-c25c-47c3-925f-35ea421e07c5\" (UID: \"391c7032-c25c-47c3-925f-35ea421e07c5\") "
Oct 30 19:13:15.833139 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:15.832987    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/391c7032-c25c-47c3-925f-35ea421e07c5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/391c7032-c25c-47c3-925f-35ea421e07c5-istio-podinfo\") pod \"391c7032-c25c-47c3-925f-35ea421e07c5\" (UID: \"391c7032-c25c-47c3-925f-35ea421e07c5\") "
Oct 30 19:13:15.833139 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:15.833059    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/391c7032-c25c-47c3-925f-35ea421e07c5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/391c7032-c25c-47c3-925f-35ea421e07c5-workload-socket\") pod \"391c7032-c25c-47c3-925f-35ea421e07c5\" (UID: \"391c7032-c25c-47c3-925f-35ea421e07c5\") "
Oct 30 19:13:15.833472 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:15.833241    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/391c7032-c25c-47c3-925f-35ea421e07c5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/391c7032-c25c-47c3-925f-35ea421e07c5-istio-envoy\") pod \"391c7032-c25c-47c3-925f-35ea421e07c5\" (UID: \"391c7032-c25c-47c3-925f-35ea421e07c5\") "
Oct 30 19:13:15.833472 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:15.833344    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/391c7032-c25c-47c3-925f-35ea421e07c5-istio-token\" (UniqueName: \"kubernetes.io/projected/391c7032-c25c-47c3-925f-35ea421e07c5-istio-token\") pod \"391c7032-c25c-47c3-925f-35ea421e07c5\" (UID: \"391c7032-c25c-47c3-925f-35ea421e07c5\") "
Oct 30 19:13:15.834128 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:15.834094    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/391c7032-c25c-47c3-925f-35ea421e07c5-istiod-ca-cert" pod "391c7032-c25c-47c3-925f-35ea421e07c5" (UID: "391c7032-c25c-47c3-925f-35ea421e07c5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:13:15.834428 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:15.834401    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/391c7032-c25c-47c3-925f-35ea421e07c5-workload-socket" pod "391c7032-c25c-47c3-925f-35ea421e07c5" (UID: "391c7032-c25c-47c3-925f-35ea421e07c5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:15.835000 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:15.834632    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/391c7032-c25c-47c3-925f-35ea421e07c5-workload-certs" pod "391c7032-c25c-47c3-925f-35ea421e07c5" (UID: "391c7032-c25c-47c3-925f-35ea421e07c5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:15.835236 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:15.835219    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/391c7032-c25c-47c3-925f-35ea421e07c5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/391c7032-c25c-47c3-925f-35ea421e07c5-istio-data\") pod \"391c7032-c25c-47c3-925f-35ea421e07c5\" (UID: \"391c7032-c25c-47c3-925f-35ea421e07c5\") "
Oct 30 19:13:15.835987 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:15.835657    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/391c7032-c25c-47c3-925f-35ea421e07c5-istio-data" pod "391c7032-c25c-47c3-925f-35ea421e07c5" (UID: "391c7032-c25c-47c3-925f-35ea421e07c5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:15.836618 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:15.836593    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c52512ff-3354-4aba-9d39-99a3084e02af-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-f3d6a5-f7d7npjtw\" (UID: \"c52512ff-3354-4aba-9d39-99a3084e02af\") " pod="ai-tenant-e2e-extmod-b-f3d6a5/payload-processing-external-model-e2e-extmod-b-f3d6a5-f7d7npjtw"
Oct 30 19:13:15.837018 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:15.836980    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqs7b\" (UniqueName: \"kubernetes.io/projected/0da000be-c37d-4fe0-87b5-9e15525341c8-kube-api-access-wqs7b\") pod \"payload-processing-external-model-e2e-extmod-a-1dd09d-5445l9g92\" (UID: \"0da000be-c37d-4fe0-87b5-9e15525341c8\") " pod="ai-tenant-e2e-extmod-a-1dd09d/payload-processing-external-model-e2e-extmod-a-1dd09d-5445l9g92"
Oct 30 19:13:15.840199 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:15.839008    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0da000be-c37d-4fe0-87b5-9e15525341c8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-1dd09d-5445l9g92\" (UID: \"0da000be-c37d-4fe0-87b5-9e15525341c8\") " pod="ai-tenant-e2e-extmod-a-1dd09d/payload-processing-external-model-e2e-extmod-a-1dd09d-5445l9g92"
Oct 30 19:13:15.840199 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:15.839075    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0da000be-c37d-4fe0-87b5-9e15525341c8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1dd09d-5445l9g92\" (UID: \"0da000be-c37d-4fe0-87b5-9e15525341c8\") " pod="ai-tenant-e2e-extmod-a-1dd09d/payload-processing-external-model-e2e-extmod-a-1dd09d-5445l9g92"
Oct 30 19:13:15.840199 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:15.839141    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c52512ff-3354-4aba-9d39-99a3084e02af-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f3d6a5-f7d7npjtw\" (UID: \"c52512ff-3354-4aba-9d39-99a3084e02af\") " pod="ai-tenant-e2e-extmod-b-f3d6a5/payload-processing-external-model-e2e-extmod-b-f3d6a5-f7d7npjtw"
Oct 30 19:13:15.840199 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:15.839296    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpbd2\" (UniqueName: \"kubernetes.io/projected/c52512ff-3354-4aba-9d39-99a3084e02af-kube-api-access-lpbd2\") pod \"payload-processing-external-model-e2e-extmod-b-f3d6a5-f7d7npjtw\" (UID: \"c52512ff-3354-4aba-9d39-99a3084e02af\") " pod="ai-tenant-e2e-extmod-b-f3d6a5/payload-processing-external-model-e2e-extmod-b-f3d6a5-f7d7npjtw"
Oct 30 19:13:15.840199 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:15.839330    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c52512ff-3354-4aba-9d39-99a3084e02af-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f3d6a5-f7d7npjtw\" (UID: \"c52512ff-3354-4aba-9d39-99a3084e02af\") " pod="ai-tenant-e2e-extmod-b-f3d6a5/payload-processing-external-model-e2e-extmod-b-f3d6a5-f7d7npjtw"
Oct 30 19:13:15.840199 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:15.839369    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0da000be-c37d-4fe0-87b5-9e15525341c8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1dd09d-5445l9g92\" (UID: \"0da000be-c37d-4fe0-87b5-9e15525341c8\") " pod="ai-tenant-e2e-extmod-a-1dd09d/payload-processing-external-model-e2e-extmod-a-1dd09d-5445l9g92"
Oct 30 19:13:15.840199 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:15.839527    4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/391c7032-c25c-47c3-925f-35ea421e07c5-istiod-ca-cert\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:15.840199 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:15.839545    4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/391c7032-c25c-47c3-925f-35ea421e07c5-workload-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:15.840199 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:15.839560    4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/391c7032-c25c-47c3-925f-35ea421e07c5-istio-data\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:15.840199 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:15.839575    4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/391c7032-c25c-47c3-925f-35ea421e07c5-workload-certs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:15.841352 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:15.840515    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/391c7032-c25c-47c3-925f-35ea421e07c5-istio-envoy" pod "391c7032-c25c-47c3-925f-35ea421e07c5" (UID: "391c7032-c25c-47c3-925f-35ea421e07c5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:15.841352 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:15.840618    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/391c7032-c25c-47c3-925f-35ea421e07c5-istio-token" pod "391c7032-c25c-47c3-925f-35ea421e07c5" (UID: "391c7032-c25c-47c3-925f-35ea421e07c5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:13:15.841352 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:15.840767    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/391c7032-c25c-47c3-925f-35ea421e07c5-istio-podinfo" pod "391c7032-c25c-47c3-925f-35ea421e07c5" (UID: "391c7032-c25c-47c3-925f-35ea421e07c5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:13:15.841352 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:15.841285    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/391c7032-c25c-47c3-925f-35ea421e07c5-credential-socket" pod "391c7032-c25c-47c3-925f-35ea421e07c5" (UID: "391c7032-c25c-47c3-925f-35ea421e07c5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:15.843500 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:15.843448    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/391c7032-c25c-47c3-925f-35ea421e07c5-kube-api-access-4t8n4" pod "391c7032-c25c-47c3-925f-35ea421e07c5" (UID: "391c7032-c25c-47c3-925f-35ea421e07c5"). InnerVolumeSpecName "kube-api-access-4t8n4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:13:15.852310 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:15.851620    4525 generic.go:356] "Generic (PLEG): container finished" podID="2b46ae00-fd8c-4575-b843-7ca560d0f12c" containerID="537f0c8c39c9527732edc6d6f161d66f89e01d1f7ad124d0907c52b8824d22a8" exitCode=0
Oct 30 19:13:15.852310 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:15.851739    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e703ad97-92hq4" event={"ID":"2b46ae00-fd8c-4575-b843-7ca560d0f12c","Type":"ContainerDied","Data":"537f0c8c39c9527732edc6d6f161d66f89e01d1f7ad124d0907c52b8824d22a8"}
Oct 30 19:13:15.853456 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:15.853436    4525 generic.go:356] "Generic (PLEG): container finished" podID="391c7032-c25c-47c3-925f-35ea421e07c5" containerID="b89bf1225edd646a777d562500d7c8f895cb29ea21cc764e998b9afa96b14f03" exitCode=0
Oct 30 19:13:15.853563 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:15.853493    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65"
Oct 30 19:13:15.853563 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:15.853515    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65" event={"ID":"391c7032-c25c-47c3-925f-35ea421e07c5","Type":"ContainerDied","Data":"b89bf1225edd646a777d562500d7c8f895cb29ea21cc764e998b9afa96b14f03"}
Oct 30 19:13:15.854176 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:15.853566    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65" event={"ID":"391c7032-c25c-47c3-925f-35ea421e07c5","Type":"ContainerDied","Data":"5bec36c891864617167a987a03ef7ea50fb05bff58edcc80e112232b04495f1e"}
Oct 30 19:13:15.854176 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:15.853596    4525 scope.go:122] "RemoveContainer" containerID="b89bf1225edd646a777d562500d7c8f895cb29ea21cc764e998b9afa96b14f03"
Oct 30 19:13:15.908757 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:15.907846    4525 scope.go:122] "RemoveContainer" containerID="b89bf1225edd646a777d562500d7c8f895cb29ea21cc764e998b9afa96b14f03"
Oct 30 19:13:15.908757 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:13:15.908331    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b89bf1225edd646a777d562500d7c8f895cb29ea21cc764e998b9afa96b14f03\": container with ID starting with b89bf1225edd646a777d562500d7c8f895cb29ea21cc764e998b9afa96b14f03 not found: ID does not exist" containerID="b89bf1225edd646a777d562500d7c8f895cb29ea21cc764e998b9afa96b14f03"
Oct 30 19:13:15.908757 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:15.908375    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b89bf1225edd646a777d562500d7c8f895cb29ea21cc764e998b9afa96b14f03"} err="failed to get container status \"b89bf1225edd646a777d562500d7c8f895cb29ea21cc764e998b9afa96b14f03\": rpc error: code = NotFound desc = could not find container \"b89bf1225edd646a777d562500d7c8f895cb29ea21cc764e998b9afa96b14f03\": container with ID starting with b89bf1225edd646a777d562500d7c8f895cb29ea21cc764e998b9afa96b14f03 not found: ID does not exist"
Oct 30 19:13:15.924751 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:15.924720    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65"]
Oct 30 19:13:15.932231 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:15.932202    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-wmh65"]
Oct 30 19:13:15.940959 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:15.940926    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0da000be-c37d-4fe0-87b5-9e15525341c8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-1dd09d-5445l9g92\" (UID: \"0da000be-c37d-4fe0-87b5-9e15525341c8\") " pod="ai-tenant-e2e-extmod-a-1dd09d/payload-processing-external-model-e2e-extmod-a-1dd09d-5445l9g92"
Oct 30 19:13:15.941148 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:15.940967    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0da000be-c37d-4fe0-87b5-9e15525341c8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1dd09d-5445l9g92\" (UID: \"0da000be-c37d-4fe0-87b5-9e15525341c8\") " pod="ai-tenant-e2e-extmod-a-1dd09d/payload-processing-external-model-e2e-extmod-a-1dd09d-5445l9g92"
Oct 30 19:13:15.941148 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:15.941009    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c52512ff-3354-4aba-9d39-99a3084e02af-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f3d6a5-f7d7npjtw\" (UID: \"c52512ff-3354-4aba-9d39-99a3084e02af\") " pod="ai-tenant-e2e-extmod-b-f3d6a5/payload-processing-external-model-e2e-extmod-b-f3d6a5-f7d7npjtw"
Oct 30 19:13:15.941148 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:15.941088    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpbd2\" (UniqueName: \"kubernetes.io/projected/c52512ff-3354-4aba-9d39-99a3084e02af-kube-api-access-lpbd2\") pod \"payload-processing-external-model-e2e-extmod-b-f3d6a5-f7d7npjtw\" (UID: \"c52512ff-3354-4aba-9d39-99a3084e02af\") " pod="ai-tenant-e2e-extmod-b-f3d6a5/payload-processing-external-model-e2e-extmod-b-f3d6a5-f7d7npjtw"
Oct 30 19:13:15.941332 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:15.941198    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c52512ff-3354-4aba-9d39-99a3084e02af-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f3d6a5-f7d7npjtw\" (UID: \"c52512ff-3354-4aba-9d39-99a3084e02af\") " pod="ai-tenant-e2e-extmod-b-f3d6a5/payload-processing-external-model-e2e-extmod-b-f3d6a5-f7d7npjtw"
Oct 30 19:13:15.941332 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:15.941242    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0da000be-c37d-4fe0-87b5-9e15525341c8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1dd09d-5445l9g92\" (UID: \"0da000be-c37d-4fe0-87b5-9e15525341c8\") " pod="ai-tenant-e2e-extmod-a-1dd09d/payload-processing-external-model-e2e-extmod-a-1dd09d-5445l9g92"
Oct 30 19:13:15.941902 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:15.941466    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c52512ff-3354-4aba-9d39-99a3084e02af-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-f3d6a5-f7d7npjtw\" (UID: \"c52512ff-3354-4aba-9d39-99a3084e02af\") " pod="ai-tenant-e2e-extmod-b-f3d6a5/payload-processing-external-model-e2e-extmod-b-f3d6a5-f7d7npjtw"
Oct 30 19:13:15.941902 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:15.941533    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqs7b\" (UniqueName: \"kubernetes.io/projected/0da000be-c37d-4fe0-87b5-9e15525341c8-kube-api-access-wqs7b\") pod \"payload-processing-external-model-e2e-extmod-a-1dd09d-5445l9g92\" (UID: \"0da000be-c37d-4fe0-87b5-9e15525341c8\") " pod="ai-tenant-e2e-extmod-a-1dd09d/payload-processing-external-model-e2e-extmod-a-1dd09d-5445l9g92"
Oct 30 19:13:15.941902 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:15.941776    4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/391c7032-c25c-47c3-925f-35ea421e07c5-istio-envoy\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:15.941902 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:15.941797    4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/391c7032-c25c-47c3-925f-35ea421e07c5-istio-token\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:15.941902 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:15.941816    4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/391c7032-c25c-47c3-925f-35ea421e07c5-credential-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:15.941902 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:15.941833    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4t8n4\" (UniqueName: \"kubernetes.io/projected/391c7032-c25c-47c3-925f-35ea421e07c5-kube-api-access-4t8n4\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:15.941902 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:15.941852    4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/391c7032-c25c-47c3-925f-35ea421e07c5-istio-podinfo\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:15.941902 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:15.941831    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c52512ff-3354-4aba-9d39-99a3084e02af-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f3d6a5-f7d7npjtw\" (UID: \"c52512ff-3354-4aba-9d39-99a3084e02af\") " pod="ai-tenant-e2e-extmod-b-f3d6a5/payload-processing-external-model-e2e-extmod-b-f3d6a5-f7d7npjtw"
Oct 30 19:13:15.942400 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:15.942061    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0da000be-c37d-4fe0-87b5-9e15525341c8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1dd09d-5445l9g92\" (UID: \"0da000be-c37d-4fe0-87b5-9e15525341c8\") " pod="ai-tenant-e2e-extmod-a-1dd09d/payload-processing-external-model-e2e-extmod-a-1dd09d-5445l9g92"
Oct 30 19:13:15.942400 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:15.942119    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0da000be-c37d-4fe0-87b5-9e15525341c8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1dd09d-5445l9g92\" (UID: \"0da000be-c37d-4fe0-87b5-9e15525341c8\") " pod="ai-tenant-e2e-extmod-a-1dd09d/payload-processing-external-model-e2e-extmod-a-1dd09d-5445l9g92"
Oct 30 19:13:15.942400 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:15.942175    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c52512ff-3354-4aba-9d39-99a3084e02af-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f3d6a5-f7d7npjtw\" (UID: \"c52512ff-3354-4aba-9d39-99a3084e02af\") " pod="ai-tenant-e2e-extmod-b-f3d6a5/payload-processing-external-model-e2e-extmod-b-f3d6a5-f7d7npjtw"
Oct 30 19:13:15.944467 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:15.944438    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c52512ff-3354-4aba-9d39-99a3084e02af-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-f3d6a5-f7d7npjtw\" (UID: \"c52512ff-3354-4aba-9d39-99a3084e02af\") " pod="ai-tenant-e2e-extmod-b-f3d6a5/payload-processing-external-model-e2e-extmod-b-f3d6a5-f7d7npjtw"
Oct 30 19:13:15.944995 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:15.944977    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0da000be-c37d-4fe0-87b5-9e15525341c8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-1dd09d-5445l9g92\" (UID: \"0da000be-c37d-4fe0-87b5-9e15525341c8\") " pod="ai-tenant-e2e-extmod-a-1dd09d/payload-processing-external-model-e2e-extmod-a-1dd09d-5445l9g92"
Oct 30 19:13:15.949882 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:15.949845    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpbd2\" (UniqueName: \"kubernetes.io/projected/c52512ff-3354-4aba-9d39-99a3084e02af-kube-api-access-lpbd2\") pod \"payload-processing-external-model-e2e-extmod-b-f3d6a5-f7d7npjtw\" (UID: \"c52512ff-3354-4aba-9d39-99a3084e02af\") " pod="ai-tenant-e2e-extmod-b-f3d6a5/payload-processing-external-model-e2e-extmod-b-f3d6a5-f7d7npjtw"
Oct 30 19:13:15.950273 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:15.950250    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqs7b\" (UniqueName: \"kubernetes.io/projected/0da000be-c37d-4fe0-87b5-9e15525341c8-kube-api-access-wqs7b\") pod \"payload-processing-external-model-e2e-extmod-a-1dd09d-5445l9g92\" (UID: \"0da000be-c37d-4fe0-87b5-9e15525341c8\") " pod="ai-tenant-e2e-extmod-a-1dd09d/payload-processing-external-model-e2e-extmod-a-1dd09d-5445l9g92"
Oct 30 19:13:16.070563 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:16.067028    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f3d6a5/payload-processing-external-model-e2e-extmod-b-f3d6a5-f7d7npjtw"
Oct 30 19:13:16.087704 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:16.071826    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-1dd09d/payload-processing-external-model-e2e-extmod-a-1dd09d-5445l9g92"
Oct 30 19:13:16.144384 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:16.143662    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-zvv2m"]
Oct 30 19:13:16.144384 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:16.143950    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-zvv2m" podUID="7c2697b3-2f09-4c2e-a6b6-abd10d93835c" containerName="istio-proxy" containerID="cri-o://d79e73b4d3461f461b871b6ad2c34e85d4c522c2dc7fe1a6dca1e2f8a61dbd6b" gracePeriod=30
Oct 30 19:13:16.541779 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:13:16.541749    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc52512ff_3354_4aba_9d39_99a3084e02af.slice/crio-68c8e680874ec1dd0928db7da95af8dfd6efaf763b3c7c3744da85d3307a5282 WatchSource:0}: Error finding container 68c8e680874ec1dd0928db7da95af8dfd6efaf763b3c7c3744da85d3307a5282: Status 404 returned error can't find the container with id 68c8e680874ec1dd0928db7da95af8dfd6efaf763b3c7c3744da85d3307a5282
Oct 30 19:13:16.543526 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:16.542805    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-f3d6a5/payload-processing-external-model-e2e-extmod-b-f3d6a5-f7d7npjtw"]
Oct 30 19:13:16.545359 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:13:16.545332    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0da000be_c37d_4fe0_87b5_9e15525341c8.slice/crio-7f9ce19ca9bdb416428d8996176dcac24e343e3a387106b282b501d9e7e08a09 WatchSource:0}: Error finding container 7f9ce19ca9bdb416428d8996176dcac24e343e3a387106b282b501d9e7e08a09: Status 404 returned error can't find the container with id 7f9ce19ca9bdb416428d8996176dcac24e343e3a387106b282b501d9e7e08a09
Oct 30 19:13:16.551301 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:16.551275    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-1dd09d/payload-processing-external-model-e2e-extmod-a-1dd09d-5445l9g92"]
Oct 30 19:13:16.889904 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:16.885364    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f3d6a5/payload-processing-external-model-e2e-extmod-b-f3d6a5-f7d7npjtw" event={"ID":"c52512ff-3354-4aba-9d39-99a3084e02af","Type":"ContainerStarted","Data":"d15f22c40ccf75d8c2265fe3d023ccc0401bf6843bf5a0a4a3d5b37d0ebc0dff"}
Oct 30 19:13:16.889904 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:16.885445    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f3d6a5/payload-processing-external-model-e2e-extmod-b-f3d6a5-f7d7npjtw" event={"ID":"c52512ff-3354-4aba-9d39-99a3084e02af","Type":"ContainerStarted","Data":"68c8e680874ec1dd0928db7da95af8dfd6efaf763b3c7c3744da85d3307a5282"}
Oct 30 19:13:16.889904 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:16.886872    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-f3d6a5/payload-processing-external-model-e2e-extmod-b-f3d6a5-f7d7npjtw"
Oct 30 19:13:16.889904 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:16.889464    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-1dd09d/payload-processing-external-model-e2e-extmod-a-1dd09d-5445l9g92" event={"ID":"0da000be-c37d-4fe0-87b5-9e15525341c8","Type":"ContainerStarted","Data":"0abbdc2b2a07258ea865019f035bfafe47ff17ab8836f5a9679790bde1285e74"}
Oct 30 19:13:16.889904 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:16.889513    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-1dd09d/payload-processing-external-model-e2e-extmod-a-1dd09d-5445l9g92" event={"ID":"0da000be-c37d-4fe0-87b5-9e15525341c8","Type":"ContainerStarted","Data":"7f9ce19ca9bdb416428d8996176dcac24e343e3a387106b282b501d9e7e08a09"}
Oct 30 19:13:16.890707 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:16.890393    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-1dd09d/payload-processing-external-model-e2e-extmod-a-1dd09d-5445l9g92"
Oct 30 19:13:16.915461 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:16.915399    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-f3d6a5/payload-processing-external-model-e2e-extmod-b-f3d6a5-f7d7npjtw" podStartSLOduration=23.915380759 podStartE2EDuration="23.915380759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:12:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:13:16.906105534 +0000 UTC m=+2219.531775501" watchObservedRunningTime="2026-10-30 19:13:16.915380759 +0000 UTC m=+2219.541050723"
Oct 30 19:13:16.933018 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:16.932966    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-1dd09d/payload-processing-external-model-e2e-extmod-a-1dd09d-5445l9g92" podStartSLOduration=23.932954381000002 podStartE2EDuration="23.932954381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:12:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:13:16.927516599 +0000 UTC m=+2219.553186567" watchObservedRunningTime="2026-10-30 19:13:16.932954381 +0000 UTC m=+2219.558624344"
Oct 30 19:13:17.228519 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:17.228488    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e703ad97-92hq4"
Oct 30 19:13:17.247155 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:17.247123    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b46ae00-fd8c-4575-b843-7ca560d0f12c-kube-api-access-5t8fw\" (UniqueName: \"kubernetes.io/projected/2b46ae00-fd8c-4575-b843-7ca560d0f12c-kube-api-access-5t8fw\") pod \"2b46ae00-fd8c-4575-b843-7ca560d0f12c\" (UID: \"2b46ae00-fd8c-4575-b843-7ca560d0f12c\") "
Oct 30 19:13:17.247408 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:17.247347    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b46ae00-fd8c-4575-b843-7ca560d0f12c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b46ae00-fd8c-4575-b843-7ca560d0f12c-maas-api-openshift-ca\") pod \"2b46ae00-fd8c-4575-b843-7ca560d0f12c\" (UID: \"2b46ae00-fd8c-4575-b843-7ca560d0f12c\") "
Oct 30 19:13:17.247492 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:17.247470    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b46ae00-fd8c-4575-b843-7ca560d0f12c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b46ae00-fd8c-4575-b843-7ca560d0f12c-maas-api-xks-ca\") pod \"2b46ae00-fd8c-4575-b843-7ca560d0f12c\" (UID: \"2b46ae00-fd8c-4575-b843-7ca560d0f12c\") "
Oct 30 19:13:17.247797 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:17.247771    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b46ae00-fd8c-4575-b843-7ca560d0f12c-maas-api-openshift-ca" pod "2b46ae00-fd8c-4575-b843-7ca560d0f12c" (UID: "2b46ae00-fd8c-4575-b843-7ca560d0f12c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:13:17.248404 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:17.248374    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b46ae00-fd8c-4575-b843-7ca560d0f12c-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:17.251751 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:17.251667    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b46ae00-fd8c-4575-b843-7ca560d0f12c-maas-api-xks-ca" pod "2b46ae00-fd8c-4575-b843-7ca560d0f12c" (UID: "2b46ae00-fd8c-4575-b843-7ca560d0f12c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:13:17.251869 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:17.251792    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b46ae00-fd8c-4575-b843-7ca560d0f12c-kube-api-access-5t8fw" pod "2b46ae00-fd8c-4575-b843-7ca560d0f12c" (UID: "2b46ae00-fd8c-4575-b843-7ca560d0f12c"). InnerVolumeSpecName "kube-api-access-5t8fw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:13:17.350212 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:17.350176    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b46ae00-fd8c-4575-b843-7ca560d0f12c-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:17.350212 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:17.350208    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5t8fw\" (UniqueName: \"kubernetes.io/projected/2b46ae00-fd8c-4575-b843-7ca560d0f12c-kube-api-access-5t8fw\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:17.404467 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:13:17.404426    4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d49l" containerName="extractor"
Oct 30 19:13:17.681911 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:17.681871    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="391c7032-c25c-47c3-925f-35ea421e07c5" path="/var/lib/kubelet/pods/391c7032-c25c-47c3-925f-35ea421e07c5/volumes"
Oct 30 19:13:17.903615 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:17.903581    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e703ad97-92hq4" event={"ID":"2b46ae00-fd8c-4575-b843-7ca560d0f12c","Type":"ContainerDied","Data":"53d3d7cb7576336827abe38b8a9b6bf548b1b1f50d7c72df326fcc4bcf265db9"}
Oct 30 19:13:17.904112 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:17.903641    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53d3d7cb7576336827abe38b8a9b6bf548b1b1f50d7c72df326fcc4bcf265db9"
Oct 30 19:13:17.904112 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:17.903644    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e703ad97-92hq4"
Oct 30 19:13:19.446857 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:19.446819    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:13:19.447365 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:19.446886    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45" podUID="d2f9a313-f069-4e48-8cdc-f75c6a285a7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:13:20.669035 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:20.668990    4525 patch_prober.go:29] interesting pod/e2e-worker-w0-9de872-openshift-default-77cf8df555-qv6r8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 19:13:20.669572 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:20.669049    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-qv6r8" podUID="36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 19:13:20.949176 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:20.949141    4525 generic.go:356] "Generic (PLEG): container finished" podID="36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b" containerID="32ecebc6ea5d4f7b9a8c28cd0fe1175887c16eabc9e5ce09032bd44d03278131" exitCode=0
Oct 30 19:13:20.949350 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:20.949219    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-qv6r8" event={"ID":"36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b","Type":"ContainerDied","Data":"32ecebc6ea5d4f7b9a8c28cd0fe1175887c16eabc9e5ce09032bd44d03278131"}
Oct 30 19:13:21.499304 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:21.498921    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-qv6r8"
Oct 30 19:13:21.614018 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:21.610958    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b-istio-envoy\") pod \"36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b\" (UID: \"36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b\") "
Oct 30 19:13:21.614018 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:21.611056    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b-credential-socket\") pod \"36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b\" (UID: \"36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b\") "
Oct 30 19:13:21.614018 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:21.611111    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b-istio-podinfo\") pod \"36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b\" (UID: \"36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b\") "
Oct 30 19:13:21.614018 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:21.611147    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b-workload-certs\") pod \"36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b\" (UID: \"36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b\") "
Oct 30 19:13:21.644552 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:21.618970    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b-credential-socket" pod "36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b" (UID: "36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:21.644552 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:21.620236    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b-workload-certs" pod "36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b" (UID: "36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:21.679026 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:21.673507    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b-istio-podinfo" pod "36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b" (UID: "36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:13:21.679026 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:21.678095    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b-istio-envoy" pod "36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b" (UID: "36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:21.699841 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:21.699650    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-96zzd"]
Oct 30 19:13:21.700939 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:21.700913    4525 cpu_manager.go:414] "containerMap: removing container" podUID="2b46ae00-fd8c-4575-b843-7ca560d0f12c" containerName="revoke-keys"
Oct 30 19:13:21.700939 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:21.700939    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b46ae00-fd8c-4575-b843-7ca560d0f12c" containerName="revoke-keys"
Oct 30 19:13:21.701135 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:21.700968    4525 cpu_manager.go:414] "containerMap: removing container" podUID="36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b" containerName="istio-proxy"
Oct 30 19:13:21.701135 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:21.700978    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b" containerName="istio-proxy"
Oct 30 19:13:21.701252 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:21.701206    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b46ae00-fd8c-4575-b843-7ca560d0f12c" containerName="revoke-keys"
Oct 30 19:13:21.701252 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:21.701223    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b" containerName="istio-proxy"
Oct 30 19:13:21.706864 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:21.706838    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-96zzd"]
Oct 30 19:13:21.706990 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:21.706963    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-96zzd"
Oct 30 19:13:21.712959 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:21.712932    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b-workload-socket\") pod \"36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b\" (UID: \"36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b\") "
Oct 30 19:13:21.713094 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:21.713083    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b-istio-token\" (UniqueName: \"kubernetes.io/projected/36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b-istio-token\") pod \"36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b\" (UID: \"36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b\") "
Oct 30 19:13:21.713221 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:21.713164    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b-istiod-ca-cert\") pod \"36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b\" (UID: \"36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b\") "
Oct 30 19:13:21.713363 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:21.713341    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b-kube-api-access-jtpbq\" (UniqueName: \"kubernetes.io/projected/36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b-kube-api-access-jtpbq\") pod \"36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b\" (UID: \"36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b\") "
Oct 30 19:13:21.713509 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:21.713491    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b-istio-data\") pod \"36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b\" (UID: \"36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b\") "
Oct 30 19:13:21.713730 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:21.713708    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b-workload-socket" pod "36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b" (UID: "36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:21.713899 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:21.713871    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a-workload-socket\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-96zzd\" (UID: \"33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-96zzd"
Oct 30 19:13:21.713974 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:21.713926    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b-istiod-ca-cert" pod "36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b" (UID: "36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:13:21.713974 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:21.713937    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a-istiod-ca-cert\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-96zzd\" (UID: \"33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-96zzd"
Oct 30 19:13:21.713974 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:21.713964    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hhqw\" (UniqueName: \"kubernetes.io/projected/33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a-kube-api-access-7hhqw\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-96zzd\" (UID: \"33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-96zzd"
Oct 30 19:13:21.714155 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:21.714117    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a-istio-data\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-96zzd\" (UID: \"33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-96zzd"
Oct 30 19:13:21.714217 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:21.714164    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a-istio-podinfo\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-96zzd\" (UID: \"33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-96zzd"
Oct 30 19:13:21.714217 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:21.714201    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b-istio-data" pod "36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b" (UID: "36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:21.714334 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:21.714306    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a-istio-envoy\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-96zzd\" (UID: \"33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-96zzd"
Oct 30 19:13:21.714439 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:21.714417    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a-credential-socket\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-96zzd\" (UID: \"33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-96zzd"
Oct 30 19:13:21.714560 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:21.714542    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a-istio-token\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-96zzd\" (UID: \"33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-96zzd"
Oct 30 19:13:21.714636 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:21.714592    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a-workload-certs\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-96zzd\" (UID: \"33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-96zzd"
Oct 30 19:13:21.714758 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:21.714726    4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b-istio-data\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:21.714758 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:21.714747    4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b-workload-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:21.714931 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:21.714762    4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b-istiod-ca-cert\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:21.714931 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:21.714778    4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b-istio-envoy\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:21.714931 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:21.714791    4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b-credential-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:21.714931 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:21.714805    4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b-istio-podinfo\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:21.714931 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:21.714818    4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b-workload-certs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:21.729968 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:21.728753    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b-kube-api-access-jtpbq" pod "36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b" (UID: "36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b"). InnerVolumeSpecName "kube-api-access-jtpbq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:13:21.737038 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:21.737008    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b-istio-token" pod "36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b" (UID: "36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:13:21.815656 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:21.815617    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a-istio-data\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-96zzd\" (UID: \"33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-96zzd"
Oct 30 19:13:21.815656 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:21.815664    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a-istio-podinfo\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-96zzd\" (UID: \"33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-96zzd"
Oct 30 19:13:21.815945 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:21.815733    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a-istio-envoy\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-96zzd\" (UID: \"33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-96zzd"
Oct 30 19:13:21.816137 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:21.816070    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a-istio-data\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-96zzd\" (UID: \"33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-96zzd"
Oct 30 19:13:21.816317 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:21.816188    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a-credential-socket\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-96zzd\" (UID: \"33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-96zzd"
Oct 30 19:13:21.816460 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:21.816439    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a-istio-token\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-96zzd\" (UID: \"33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-96zzd"
Oct 30 19:13:21.816643 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:21.816482    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a-credential-socket\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-96zzd\" (UID: \"33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-96zzd"
Oct 30 19:13:21.816643 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:21.816489    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a-workload-certs\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-96zzd\" (UID: \"33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-96zzd"
Oct 30 19:13:21.816643 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:21.816577    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a-workload-socket\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-96zzd\" (UID: \"33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-96zzd"
Oct 30 19:13:21.816643 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:21.816623    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a-istiod-ca-cert\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-96zzd\" (UID: \"33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-96zzd"
Oct 30 19:13:21.816998 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:21.816646    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hhqw\" (UniqueName: \"kubernetes.io/projected/33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a-kube-api-access-7hhqw\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-96zzd\" (UID: \"33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-96zzd"
Oct 30 19:13:21.816998 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:21.816815    4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b-istio-token\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:21.816998 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:21.816836    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtpbq\" (UniqueName: \"kubernetes.io/projected/36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b-kube-api-access-jtpbq\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:21.816998 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:21.816973    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a-workload-socket\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-96zzd\" (UID: \"33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-96zzd"
Oct 30 19:13:21.817247 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:21.817200    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a-workload-certs\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-96zzd\" (UID: \"33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-96zzd"
Oct 30 19:13:21.817315 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:21.817249    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a-istiod-ca-cert\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-96zzd\" (UID: \"33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-96zzd"
Oct 30 19:13:21.818929 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:21.818893    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a-istio-podinfo\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-96zzd\" (UID: \"33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-96zzd"
Oct 30 19:13:21.819232 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:21.819214    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a-istio-envoy\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-96zzd\" (UID: \"33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-96zzd"
Oct 30 19:13:21.826373 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:21.826347    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a-istio-token\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-96zzd\" (UID: \"33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-96zzd"
Oct 30 19:13:21.826497 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:21.826478    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hhqw\" (UniqueName: \"kubernetes.io/projected/33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a-kube-api-access-7hhqw\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-96zzd\" (UID: \"33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-96zzd"
Oct 30 19:13:21.961108 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:21.961075    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-qv6r8"
Oct 30 19:13:21.961284 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:21.961074    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-qv6r8" event={"ID":"36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b","Type":"ContainerDied","Data":"2f147dfbb3333d054aa4a906b2f5dae5695193bcd99649cb391c88dd3430af3a"}
Oct 30 19:13:21.961391 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:21.961285    4525 scope.go:122] "RemoveContainer" containerID="32ecebc6ea5d4f7b9a8c28cd0fe1175887c16eabc9e5ce09032bd44d03278131"
Oct 30 19:13:21.964084 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:21.964058    4525 generic.go:356] "Generic (PLEG): container finished" podID="7c2697b3-2f09-4c2e-a6b6-abd10d93835c" containerID="d79e73b4d3461f461b871b6ad2c34e85d4c522c2dc7fe1a6dca1e2f8a61dbd6b" exitCode=0
Oct 30 19:13:21.964225 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:21.964160    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-zvv2m" event={"ID":"7c2697b3-2f09-4c2e-a6b6-abd10d93835c","Type":"ContainerDied","Data":"d79e73b4d3461f461b871b6ad2c34e85d4c522c2dc7fe1a6dca1e2f8a61dbd6b"}
Oct 30 19:13:21.996229 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:21.996196    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-qv6r8"]
Oct 30 19:13:22.002929 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:22.002900    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-qv6r8"]
Oct 30 19:13:22.098162 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:22.098121    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-96zzd"
Oct 30 19:13:22.379061 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:22.378567    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-zvv2m"
Oct 30 19:13:22.439692 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:22.438472    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c2697b3-2f09-4c2e-a6b6-abd10d93835c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c2697b3-2f09-4c2e-a6b6-abd10d93835c-istio-envoy\") pod \"7c2697b3-2f09-4c2e-a6b6-abd10d93835c\" (UID: \"7c2697b3-2f09-4c2e-a6b6-abd10d93835c\") "
Oct 30 19:13:22.439692 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:22.438550    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c2697b3-2f09-4c2e-a6b6-abd10d93835c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c2697b3-2f09-4c2e-a6b6-abd10d93835c-workload-certs\") pod \"7c2697b3-2f09-4c2e-a6b6-abd10d93835c\" (UID: \"7c2697b3-2f09-4c2e-a6b6-abd10d93835c\") "
Oct 30 19:13:22.439692 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:22.438598    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c2697b3-2f09-4c2e-a6b6-abd10d93835c-kube-api-access-gxfnp\" (UniqueName: \"kubernetes.io/projected/7c2697b3-2f09-4c2e-a6b6-abd10d93835c-kube-api-access-gxfnp\") pod \"7c2697b3-2f09-4c2e-a6b6-abd10d93835c\" (UID: \"7c2697b3-2f09-4c2e-a6b6-abd10d93835c\") "
Oct 30 19:13:22.439692 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:22.438753    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c2697b3-2f09-4c2e-a6b6-abd10d93835c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c2697b3-2f09-4c2e-a6b6-abd10d93835c-credential-socket\") pod \"7c2697b3-2f09-4c2e-a6b6-abd10d93835c\" (UID: \"7c2697b3-2f09-4c2e-a6b6-abd10d93835c\") "
Oct 30 19:13:22.439692 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:22.438886    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c2697b3-2f09-4c2e-a6b6-abd10d93835c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c2697b3-2f09-4c2e-a6b6-abd10d93835c-istiod-ca-cert\") pod \"7c2697b3-2f09-4c2e-a6b6-abd10d93835c\" (UID: \"7c2697b3-2f09-4c2e-a6b6-abd10d93835c\") "
Oct 30 19:13:22.439692 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:22.438919    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c2697b3-2f09-4c2e-a6b6-abd10d93835c-istio-token\" (UniqueName: \"kubernetes.io/projected/7c2697b3-2f09-4c2e-a6b6-abd10d93835c-istio-token\") pod \"7c2697b3-2f09-4c2e-a6b6-abd10d93835c\" (UID: \"7c2697b3-2f09-4c2e-a6b6-abd10d93835c\") "
Oct 30 19:13:22.439692 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:22.439048    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c2697b3-2f09-4c2e-a6b6-abd10d93835c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c2697b3-2f09-4c2e-a6b6-abd10d93835c-istio-data\") pod \"7c2697b3-2f09-4c2e-a6b6-abd10d93835c\" (UID: \"7c2697b3-2f09-4c2e-a6b6-abd10d93835c\") "
Oct 30 19:13:22.439692 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:22.439205    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7c2697b3-2f09-4c2e-a6b6-abd10d93835c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c2697b3-2f09-4c2e-a6b6-abd10d93835c-istio-podinfo\") pod \"7c2697b3-2f09-4c2e-a6b6-abd10d93835c\" (UID: \"7c2697b3-2f09-4c2e-a6b6-abd10d93835c\") "
Oct 30 19:13:22.439692 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:22.439309    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c2697b3-2f09-4c2e-a6b6-abd10d93835c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c2697b3-2f09-4c2e-a6b6-abd10d93835c-workload-socket\") pod \"7c2697b3-2f09-4c2e-a6b6-abd10d93835c\" (UID: \"7c2697b3-2f09-4c2e-a6b6-abd10d93835c\") "
Oct 30 19:13:22.439692 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:22.439613    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c2697b3-2f09-4c2e-a6b6-abd10d93835c-credential-socket" pod "7c2697b3-2f09-4c2e-a6b6-abd10d93835c" (UID: "7c2697b3-2f09-4c2e-a6b6-abd10d93835c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:22.444053 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:22.443359    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c2697b3-2f09-4c2e-a6b6-abd10d93835c-istio-data" pod "7c2697b3-2f09-4c2e-a6b6-abd10d93835c" (UID: "7c2697b3-2f09-4c2e-a6b6-abd10d93835c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:22.444053 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:22.443696    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c2697b3-2f09-4c2e-a6b6-abd10d93835c-istiod-ca-cert" pod "7c2697b3-2f09-4c2e-a6b6-abd10d93835c" (UID: "7c2697b3-2f09-4c2e-a6b6-abd10d93835c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:13:22.445118 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:22.444608    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c2697b3-2f09-4c2e-a6b6-abd10d93835c-workload-socket" pod "7c2697b3-2f09-4c2e-a6b6-abd10d93835c" (UID: "7c2697b3-2f09-4c2e-a6b6-abd10d93835c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:22.445737 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:22.445264    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-d7mth"]
Oct 30 19:13:22.449071 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:22.446595    4525 cpu_manager.go:414] "containerMap: removing container" podUID="7c2697b3-2f09-4c2e-a6b6-abd10d93835c" containerName="istio-proxy"
Oct 30 19:13:22.449071 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:22.446620    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c2697b3-2f09-4c2e-a6b6-abd10d93835c" containerName="istio-proxy"
Oct 30 19:13:22.449071 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:22.446872    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c2697b3-2f09-4c2e-a6b6-abd10d93835c" containerName="istio-proxy"
Oct 30 19:13:22.451924 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:22.449339    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c2697b3-2f09-4c2e-a6b6-abd10d93835c-workload-certs" pod "7c2697b3-2f09-4c2e-a6b6-abd10d93835c" (UID: "7c2697b3-2f09-4c2e-a6b6-abd10d93835c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:22.452863 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:22.452839    4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c2697b3-2f09-4c2e-a6b6-abd10d93835c-credential-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:22.453066 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:22.452994    4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c2697b3-2f09-4c2e-a6b6-abd10d93835c-istiod-ca-cert\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:22.453066 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:22.453016    4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c2697b3-2f09-4c2e-a6b6-abd10d93835c-istio-data\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:22.453066 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:22.453031    4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c2697b3-2f09-4c2e-a6b6-abd10d93835c-workload-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:22.453066 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:22.453047    4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c2697b3-2f09-4c2e-a6b6-abd10d93835c-workload-certs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:22.474656 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:22.465714    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-d7mth"
Oct 30 19:13:22.474656 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:22.473796    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-d7mth"]
Oct 30 19:13:22.482316 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:22.481484    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c2697b3-2f09-4c2e-a6b6-abd10d93835c-istio-token" pod "7c2697b3-2f09-4c2e-a6b6-abd10d93835c" (UID: "7c2697b3-2f09-4c2e-a6b6-abd10d93835c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:13:22.482316 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:22.481923    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c2697b3-2f09-4c2e-a6b6-abd10d93835c-kube-api-access-gxfnp" pod "7c2697b3-2f09-4c2e-a6b6-abd10d93835c" (UID: "7c2697b3-2f09-4c2e-a6b6-abd10d93835c"). InnerVolumeSpecName "kube-api-access-gxfnp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:13:22.503235 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:22.503005    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c2697b3-2f09-4c2e-a6b6-abd10d93835c-istio-envoy" pod "7c2697b3-2f09-4c2e-a6b6-abd10d93835c" (UID: "7c2697b3-2f09-4c2e-a6b6-abd10d93835c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:22.514160 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:22.514125    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c2697b3-2f09-4c2e-a6b6-abd10d93835c-istio-podinfo" pod "7c2697b3-2f09-4c2e-a6b6-abd10d93835c" (UID: "7c2697b3-2f09-4c2e-a6b6-abd10d93835c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:13:22.555019 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:22.554931    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adb63756-9989-4246-84d1-5fc1d1c3ebd2-workload-certs\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-d7mth\" (UID: \"adb63756-9989-4246-84d1-5fc1d1c3ebd2\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-d7mth"
Oct 30 19:13:22.555188 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:22.555098    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adb63756-9989-4246-84d1-5fc1d1c3ebd2-istio-token\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-d7mth\" (UID: \"adb63756-9989-4246-84d1-5fc1d1c3ebd2\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-d7mth"
Oct 30 19:13:22.555188 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:22.555144    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adb63756-9989-4246-84d1-5fc1d1c3ebd2-istio-envoy\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-d7mth\" (UID: \"adb63756-9989-4246-84d1-5fc1d1c3ebd2\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-d7mth"
Oct 30 19:13:22.555188 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:22.555179    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb63756-9989-4246-84d1-5fc1d1c3ebd2-credential-socket\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-d7mth\" (UID: \"adb63756-9989-4246-84d1-5fc1d1c3ebd2\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-d7mth"
Oct 30 19:13:22.555402 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:22.555253    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dzln\" (UniqueName: \"kubernetes.io/projected/adb63756-9989-4246-84d1-5fc1d1c3ebd2-kube-api-access-5dzln\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-d7mth\" (UID: \"adb63756-9989-4246-84d1-5fc1d1c3ebd2\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-d7mth"
Oct 30 19:13:22.555402 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:22.555337    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adb63756-9989-4246-84d1-5fc1d1c3ebd2-istio-data\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-d7mth\" (UID: \"adb63756-9989-4246-84d1-5fc1d1c3ebd2\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-d7mth"
Oct 30 19:13:22.555402 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:22.555391    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adb63756-9989-4246-84d1-5fc1d1c3ebd2-istio-podinfo\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-d7mth\" (UID: \"adb63756-9989-4246-84d1-5fc1d1c3ebd2\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-d7mth"
Oct 30 19:13:22.555572 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:22.555409    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb63756-9989-4246-84d1-5fc1d1c3ebd2-workload-socket\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-d7mth\" (UID: \"adb63756-9989-4246-84d1-5fc1d1c3ebd2\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-d7mth"
Oct 30 19:13:22.555572 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:22.555470    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adb63756-9989-4246-84d1-5fc1d1c3ebd2-istiod-ca-cert\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-d7mth\" (UID: \"adb63756-9989-4246-84d1-5fc1d1c3ebd2\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-d7mth"
Oct 30 19:13:22.555572 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:22.555542    4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c2697b3-2f09-4c2e-a6b6-abd10d93835c-istio-envoy\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:22.555572 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:22.555554    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxfnp\" (UniqueName: \"kubernetes.io/projected/7c2697b3-2f09-4c2e-a6b6-abd10d93835c-kube-api-access-gxfnp\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:22.555572 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:22.555565    4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c2697b3-2f09-4c2e-a6b6-abd10d93835c-istio-token\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:22.556504 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:22.555575    4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c2697b3-2f09-4c2e-a6b6-abd10d93835c-istio-podinfo\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:22.657246 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:22.657214    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adb63756-9989-4246-84d1-5fc1d1c3ebd2-istio-data\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-d7mth\" (UID: \"adb63756-9989-4246-84d1-5fc1d1c3ebd2\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-d7mth"
Oct 30 19:13:22.657382 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:22.657302    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adb63756-9989-4246-84d1-5fc1d1c3ebd2-istio-podinfo\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-d7mth\" (UID: \"adb63756-9989-4246-84d1-5fc1d1c3ebd2\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-d7mth"
Oct 30 19:13:22.657382 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:22.657336    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb63756-9989-4246-84d1-5fc1d1c3ebd2-workload-socket\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-d7mth\" (UID: \"adb63756-9989-4246-84d1-5fc1d1c3ebd2\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-d7mth"
Oct 30 19:13:22.657522 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:22.657500    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adb63756-9989-4246-84d1-5fc1d1c3ebd2-istiod-ca-cert\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-d7mth\" (UID: \"adb63756-9989-4246-84d1-5fc1d1c3ebd2\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-d7mth"
Oct 30 19:13:22.657586 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:22.657558    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adb63756-9989-4246-84d1-5fc1d1c3ebd2-workload-certs\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-d7mth\" (UID: \"adb63756-9989-4246-84d1-5fc1d1c3ebd2\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-d7mth"
Oct 30 19:13:22.657725 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:22.657701    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adb63756-9989-4246-84d1-5fc1d1c3ebd2-istio-data\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-d7mth\" (UID: \"adb63756-9989-4246-84d1-5fc1d1c3ebd2\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-d7mth"
Oct 30 19:13:22.657870 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:22.657807    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adb63756-9989-4246-84d1-5fc1d1c3ebd2-istio-token\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-d7mth\" (UID: \"adb63756-9989-4246-84d1-5fc1d1c3ebd2\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-d7mth"
Oct 30 19:13:22.658033 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:22.657873    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adb63756-9989-4246-84d1-5fc1d1c3ebd2-istio-envoy\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-d7mth\" (UID: \"adb63756-9989-4246-84d1-5fc1d1c3ebd2\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-d7mth"
Oct 30 19:13:22.658033 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:22.657949    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb63756-9989-4246-84d1-5fc1d1c3ebd2-workload-socket\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-d7mth\" (UID: \"adb63756-9989-4246-84d1-5fc1d1c3ebd2\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-d7mth"
Oct 30 19:13:22.658256 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:22.658099    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adb63756-9989-4246-84d1-5fc1d1c3ebd2-workload-certs\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-d7mth\" (UID: \"adb63756-9989-4246-84d1-5fc1d1c3ebd2\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-d7mth"
Oct 30 19:13:22.658256 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:22.657957    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb63756-9989-4246-84d1-5fc1d1c3ebd2-credential-socket\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-d7mth\" (UID: \"adb63756-9989-4246-84d1-5fc1d1c3ebd2\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-d7mth"
Oct 30 19:13:22.658256 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:22.658252    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dzln\" (UniqueName: \"kubernetes.io/projected/adb63756-9989-4246-84d1-5fc1d1c3ebd2-kube-api-access-5dzln\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-d7mth\" (UID: \"adb63756-9989-4246-84d1-5fc1d1c3ebd2\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-d7mth"
Oct 30 19:13:22.658554 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:22.658475    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adb63756-9989-4246-84d1-5fc1d1c3ebd2-istiod-ca-cert\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-d7mth\" (UID: \"adb63756-9989-4246-84d1-5fc1d1c3ebd2\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-d7mth"
Oct 30 19:13:22.658554 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:22.658530    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb63756-9989-4246-84d1-5fc1d1c3ebd2-credential-socket\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-d7mth\" (UID: \"adb63756-9989-4246-84d1-5fc1d1c3ebd2\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-d7mth"
Oct 30 19:13:22.661298 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:22.661277    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adb63756-9989-4246-84d1-5fc1d1c3ebd2-istio-envoy\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-d7mth\" (UID: \"adb63756-9989-4246-84d1-5fc1d1c3ebd2\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-d7mth"
Oct 30 19:13:22.662579 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:22.662553    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adb63756-9989-4246-84d1-5fc1d1c3ebd2-istio-podinfo\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-d7mth\" (UID: \"adb63756-9989-4246-84d1-5fc1d1c3ebd2\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-d7mth"
Oct 30 19:13:22.665774 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:22.665746    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dzln\" (UniqueName: \"kubernetes.io/projected/adb63756-9989-4246-84d1-5fc1d1c3ebd2-kube-api-access-5dzln\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-d7mth\" (UID: \"adb63756-9989-4246-84d1-5fc1d1c3ebd2\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-d7mth"
Oct 30 19:13:22.666180 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:22.666159    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adb63756-9989-4246-84d1-5fc1d1c3ebd2-istio-token\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-d7mth\" (UID: \"adb63756-9989-4246-84d1-5fc1d1c3ebd2\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-d7mth"
Oct 30 19:13:22.731358 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:13:22.731332    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33b5fccf_6fc7_4a25_9338_9c3c4a5ff14a.slice/crio-895cd7380f11e5ccbf85c44a94f65c9592b1222cdc2e0c384c9c646962784a85 WatchSource:0}: Error finding container 895cd7380f11e5ccbf85c44a94f65c9592b1222cdc2e0c384c9c646962784a85: Status 404 returned error can't find the container with id 895cd7380f11e5ccbf85c44a94f65c9592b1222cdc2e0c384c9c646962784a85
Oct 30 19:13:22.732769 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:22.732745    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-96zzd"]
Oct 30 19:13:22.734940 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:22.734909    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:13:22.735040 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:22.734983    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:13:22.735040 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:22.735013    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:13:22.762376 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:22.762349    4525 scope.go:122] "RemoveContainer" containerID="312e333d6833330acb4fcde0951ab2063af3f0ba5ba832bcc8921676f139bfea"
Oct 30 19:13:22.791280 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:22.791254    4525 scope.go:122] "RemoveContainer" containerID="d79e73b4d3461f461b871b6ad2c34e85d4c522c2dc7fe1a6dca1e2f8a61dbd6b"
Oct 30 19:13:22.808894 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:22.808866    4525 scope.go:122] "RemoveContainer" containerID="c1b1837a21fb838f29057657902b42219cde8c8aa30b962129613f976b555ed2"
Oct 30 19:13:22.815793 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:22.815768    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-d7mth"
Oct 30 19:13:22.839089 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:22.839062    4525 scope.go:122] "RemoveContainer" containerID="ca7a3ef6e3ed3e2f7e6aa4eb9b83abf888232928ac66586c7f4a951d28676c88"
Oct 30 19:13:22.878476 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:22.878450    4525 scope.go:122] "RemoveContainer" containerID="88a248e06f67fae48e93b755d8f14656928e2f2db358bc2cecf2bd459687de84"
Oct 30 19:13:22.988623 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:22.988569    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-96zzd" event={"ID":"33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a","Type":"ContainerStarted","Data":"de89b1a37146d36976ac5ab7bc6418454b59b538499c9cbdbb20a2f768a8caec"}
Oct 30 19:13:22.989424 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:22.988875    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-96zzd" event={"ID":"33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a","Type":"ContainerStarted","Data":"895cd7380f11e5ccbf85c44a94f65c9592b1222cdc2e0c384c9c646962784a85"}
Oct 30 19:13:22.995441 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:22.992266    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-zvv2m" event={"ID":"7c2697b3-2f09-4c2e-a6b6-abd10d93835c","Type":"ContainerDied","Data":"b3133f42a09fd34ff6c4909079014495b217d3b5fdb1e61124b9392d0a57af12"}
Oct 30 19:13:22.995441 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:22.992363    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-zvv2m"
Oct 30 19:13:23.046972 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:23.046420    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-96zzd" podStartSLOduration=49.046387654 podStartE2EDuration="49.046387654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:12:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:13:23.036580856 +0000 UTC m=+2225.662250857" watchObservedRunningTime="2026-10-30 19:13:23.046387654 +0000 UTC m=+2225.672057623"
Oct 30 19:13:23.169864 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:23.169838    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-96zzd"
Oct 30 19:13:23.171770 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:23.170847    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-96zzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" start-of-body=
Oct 30 19:13:23.171770 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:23.170928    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-96zzd" podUID="33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused"
Oct 30 19:13:23.402131 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:23.194787    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-zvv2m"]
Oct 30 19:13:23.402131 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:23.202312    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-d7mth"]
Oct 30 19:13:23.402131 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:23.207357    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-zvv2m"]
Oct 30 19:13:23.402131 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:13:23.210835    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadb63756_9989_4246_84d1_5fc1d1c3ebd2.slice/crio-6a0e88c728dfeea36e794a058121d6bc4d4e8db42c489d34765d00737ea66422 WatchSource:0}: Error finding container 6a0e88c728dfeea36e794a058121d6bc4d4e8db42c489d34765d00737ea66422: Status 404 returned error can't find the container with id 6a0e88c728dfeea36e794a058121d6bc4d4e8db42c489d34765d00737ea66422
Oct 30 19:13:23.402131 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:23.214991    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:13:23.402131 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:23.215048    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:13:23.402131 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:23.215074    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:13:23.566580 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:23.566541    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b" path="/var/lib/kubelet/pods/36c2a8f4-b96a-4b4c-a996-e97e2fe6c90b/volumes"
Oct 30 19:13:23.567538 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:23.567502    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c2697b3-2f09-4c2e-a6b6-abd10d93835c" path="/var/lib/kubelet/pods/7c2697b3-2f09-4c2e-a6b6-abd10d93835c/volumes"
Oct 30 19:13:24.024735 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:24.024664    4525 generic.go:356] "Generic (PLEG): container finished" podID="d2f9a313-f069-4e48-8cdc-f75c6a285a7c" containerID="4e0c7213e1372f284371fd7df3e88f0b367fecefa141b4f62b24c7e75b267acf" exitCode=0
Oct 30 19:13:24.026272 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:24.026227    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45" event={"ID":"d2f9a313-f069-4e48-8cdc-f75c6a285a7c","Type":"ContainerDied","Data":"4e0c7213e1372f284371fd7df3e88f0b367fecefa141b4f62b24c7e75b267acf"}
Oct 30 19:13:24.027219 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:24.027186    4525 scope.go:122] "RemoveContainer" containerID="4e0c7213e1372f284371fd7df3e88f0b367fecefa141b4f62b24c7e75b267acf"
Oct 30 19:13:24.030822 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:24.030779    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-d7mth" event={"ID":"adb63756-9989-4246-84d1-5fc1d1c3ebd2","Type":"ContainerStarted","Data":"231338f743ae0e3cf6392e485d1590fc563014f25840d830674da9e82b25c850"}
Oct 30 19:13:24.030934 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:24.030868    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-d7mth" event={"ID":"adb63756-9989-4246-84d1-5fc1d1c3ebd2","Type":"ContainerStarted","Data":"6a0e88c728dfeea36e794a058121d6bc4d4e8db42c489d34765d00737ea66422"}
Oct 30 19:13:24.031068 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:24.031035    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:13:24.031153 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:24.031134    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:13:24.031263 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:24.031189    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:13:24.146846 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:24.122839    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-96zzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" start-of-body=
Oct 30 19:13:24.146846 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:24.122901    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-96zzd" podUID="33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused"
Oct 30 19:13:24.146846 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:24.146457    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-8wnxx"]
Oct 30 19:13:24.146846 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:24.146719    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-8wnxx" podUID="72e67919-78bc-4e8f-b8f1-13e34b8f54ce" containerName="istio-proxy" containerID="cri-o://e1029d0a9968cfcbd673cd6acfed8b749433a60f7dfa3558245729d65976c53f" gracePeriod=30
Oct 30 19:13:24.161971 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:24.154298    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-d7mth" podStartSLOduration=50.154280312 podStartE2EDuration="50.154280312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:12:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:13:24.104829342 +0000 UTC m=+2226.730499307" watchObservedRunningTime="2026-10-30 19:13:24.154280312 +0000 UTC m=+2226.779950277"
Oct 30 19:13:24.849418 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:24.848936    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-d7mth"
Oct 30 19:13:24.850564 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:24.850535    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-d7mth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" start-of-body=
Oct 30 19:13:24.850867 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:24.850824    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-d7mth" podUID="adb63756-9989-4246-84d1-5fc1d1c3ebd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused"
Oct 30 19:13:24.967003 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:24.966957    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-8wnxx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:13:24.967204 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:24.967025    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-8wnxx" podUID="72e67919-78bc-4e8f-b8f1-13e34b8f54ce" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:13:25.046522 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:25.046477    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45" event={"ID":"d2f9a313-f069-4e48-8cdc-f75c6a285a7c","Type":"ContainerStarted","Data":"5b8c6c1e4d153d4c504d2beac49d7d788e072865b13d8580aa8a54e737d1558b"}
Oct 30 19:13:25.099124 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:25.099083    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-96zzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" start-of-body=
Oct 30 19:13:25.099467 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:25.099400    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-96zzd" podUID="33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused"
Oct 30 19:13:25.374002 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:25.373953    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f009fff9-76lm9"]
Oct 30 19:13:25.443061 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:25.443004    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body=
Oct 30 19:13:25.443278 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:25.443091    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45" podUID="d2f9a313-f069-4e48-8cdc-f75c6a285a7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused"
Oct 30 19:13:25.451835 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:25.451803    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f009fff9-76lm9"]
Oct 30 19:13:25.452026 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:25.451901    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45"
Oct 30 19:13:25.452120 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:25.452061    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f009fff9-76lm9"
Oct 30 19:13:25.473221 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:25.473174    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:13:25.473651 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:25.473627    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94hcn\""
Oct 30 19:13:25.566700 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:25.565995    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52c00ed7-ee9c-422c-964d-e6c41b832d71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f009fff9-76lm9\" (UID: \"52c00ed7-ee9c-422c-964d-e6c41b832d71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f009fff9-76lm9"
Oct 30 19:13:25.566700 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:25.566085    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mksds\" (UniqueName: \"kubernetes.io/projected/52c00ed7-ee9c-422c-964d-e6c41b832d71-kube-api-access-mksds\") pod \"maas-api-revoke-sub-f009fff9-76lm9\" (UID: \"52c00ed7-ee9c-422c-964d-e6c41b832d71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f009fff9-76lm9"
Oct 30 19:13:25.566700 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:25.566255    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52c00ed7-ee9c-422c-964d-e6c41b832d71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f009fff9-76lm9\" (UID: \"52c00ed7-ee9c-422c-964d-e6c41b832d71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f009fff9-76lm9"
Oct 30 19:13:25.601721 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:25.601523    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw"]
Oct 30 19:13:25.601950 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:25.601822    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw" podUID="3330720a-f130-4cd2-8156-643a79a24235" containerName="istio-proxy" containerID="cri-o://f2e5fb680cdbedf57e345dcaca5ac439e1d270920ec383be3498906e46da14c6" gracePeriod=30
Oct 30 19:13:25.688429 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:25.683502    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52c00ed7-ee9c-422c-964d-e6c41b832d71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f009fff9-76lm9\" (UID: \"52c00ed7-ee9c-422c-964d-e6c41b832d71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f009fff9-76lm9"
Oct 30 19:13:25.688429 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:25.683901    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52c00ed7-ee9c-422c-964d-e6c41b832d71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f009fff9-76lm9\" (UID: \"52c00ed7-ee9c-422c-964d-e6c41b832d71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f009fff9-76lm9"
Oct 30 19:13:25.688429 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:25.683965    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mksds\" (UniqueName: \"kubernetes.io/projected/52c00ed7-ee9c-422c-964d-e6c41b832d71-kube-api-access-mksds\") pod \"maas-api-revoke-sub-f009fff9-76lm9\" (UID: \"52c00ed7-ee9c-422c-964d-e6c41b832d71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f009fff9-76lm9"
Oct 30 19:13:25.745762 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:25.711837    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52c00ed7-ee9c-422c-964d-e6c41b832d71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f009fff9-76lm9\" (UID: \"52c00ed7-ee9c-422c-964d-e6c41b832d71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f009fff9-76lm9"
Oct 30 19:13:25.751457 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:25.749937    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mksds\" (UniqueName: \"kubernetes.io/projected/52c00ed7-ee9c-422c-964d-e6c41b832d71-kube-api-access-mksds\") pod \"maas-api-revoke-sub-f009fff9-76lm9\" (UID: \"52c00ed7-ee9c-422c-964d-e6c41b832d71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f009fff9-76lm9"
Oct 30 19:13:25.799777 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:25.786207    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52c00ed7-ee9c-422c-964d-e6c41b832d71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f009fff9-76lm9\" (UID: \"52c00ed7-ee9c-422c-964d-e6c41b832d71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f009fff9-76lm9"
Oct 30 19:13:25.856711 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:25.856466    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-d7mth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" start-of-body=
Oct 30 19:13:25.856711 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:25.856528    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-d7mth" podUID="adb63756-9989-4246-84d1-5fc1d1c3ebd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused"
Oct 30 19:13:25.868079 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:25.868037    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f009fff9-76lm9"
Oct 30 19:13:26.176706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:26.171745    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-96zzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" start-of-body=
Oct 30 19:13:26.176706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:26.171830    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-96zzd" podUID="33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused"
Oct 30 19:13:26.544117 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:26.544076    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body=
Oct 30 19:13:26.544402 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:26.544374    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45" podUID="d2f9a313-f069-4e48-8cdc-f75c6a285a7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused"
Oct 30 19:13:26.821024 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:26.820977    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-d7mth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" start-of-body=
Oct 30 19:13:26.821304 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:26.821275    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-d7mth" podUID="adb63756-9989-4246-84d1-5fc1d1c3ebd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused"
Oct 30 19:13:26.970011 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:26.969955    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f009fff9-76lm9"]
Oct 30 19:13:27.154836 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:27.154796    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-96zzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" start-of-body=
Oct 30 19:13:27.155236 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:27.155205    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-96zzd" podUID="33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused"
Oct 30 19:13:27.412960 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:27.412879    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body=
Oct 30 19:13:27.446817 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:27.446773    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45" podUID="d2f9a313-f069-4e48-8cdc-f75c6a285a7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused"
Oct 30 19:13:27.503213 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:27.503174    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f009fff9-76lm9" event={"ID":"52c00ed7-ee9c-422c-964d-e6c41b832d71","Type":"ContainerStarted","Data":"e584e4cb6c8e6ffbc0afe79546b9a43de53fb0f923c2c96242a323478ddda9ed"}
Oct 30 19:13:27.847723 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:27.834423    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-d7mth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" start-of-body=
Oct 30 19:13:27.847723 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:27.834488    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-d7mth" podUID="adb63756-9989-4246-84d1-5fc1d1c3ebd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused"
Oct 30 19:13:28.103796 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:28.099537    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-96zzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" start-of-body=
Oct 30 19:13:28.103796 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:28.099596    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-96zzd" podUID="33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused"
Oct 30 19:13:28.410144 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:28.410099    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body=
Oct 30 19:13:28.410457 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:28.410422    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45" podUID="d2f9a313-f069-4e48-8cdc-f75c6a285a7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused"
Oct 30 19:13:28.566224 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:28.566183    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f009fff9-76lm9" event={"ID":"52c00ed7-ee9c-422c-964d-e6c41b832d71","Type":"ContainerStarted","Data":"4fde6f098ba03db33e070586182a199d7c0e5c56406e63b51724aad49ac6a98e"}
Oct 30 19:13:28.619719 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:28.619422    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f009fff9-76lm9" podStartSLOduration=3.619400647 podStartE2EDuration="3.619400647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:13:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:13:28.619116765 +0000 UTC m=+2231.244786728" watchObservedRunningTime="2026-10-30 19:13:28.619400647 +0000 UTC m=+2231.245070615"
Oct 30 19:13:28.821765 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:28.817092    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-d7mth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" start-of-body=
Oct 30 19:13:28.821765 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:28.817153    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-d7mth" podUID="adb63756-9989-4246-84d1-5fc1d1c3ebd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused"
Oct 30 19:13:28.958965 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:28.958908    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-1dd09d/payload-processing-external-model-e2e-extmod-a-1dd09d-5445l9g92"
Oct 30 19:13:28.970607 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:28.965641    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-f3d6a5/payload-processing-external-model-e2e-extmod-b-f3d6a5-f7d7npjtw"
Oct 30 19:13:29.116766 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:29.105090    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-96zzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" start-of-body=
Oct 30 19:13:29.116766 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:29.105146    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-96zzd" podUID="33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused"
Oct 30 19:13:29.372339 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:29.372217    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-dhclt"]
Oct 30 19:13:29.374089 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:29.374016    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-dhclt" podUID="0e0636e7-6f61-4843-844d-fb9d7bb527da" containerName="istio-proxy" containerID="cri-o://744818fd672a41f9671e8a439aa5cfef93a5f459b0ed208ee19d37e71d823797" gracePeriod=30
Oct 30 19:13:29.625241 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:29.600431    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body=
Oct 30 19:13:29.625241 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:29.600493    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45" podUID="d2f9a313-f069-4e48-8cdc-f75c6a285a7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused"
Oct 30 19:13:29.860782 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:29.859972    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-d7mth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" start-of-body=
Oct 30 19:13:29.860782 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:29.860036    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-d7mth" podUID="adb63756-9989-4246-84d1-5fc1d1c3ebd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused"
Oct 30 19:13:29.946806 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:29.946312    4525 generic.go:356] "Generic (PLEG): container finished" podID="72e67919-78bc-4e8f-b8f1-13e34b8f54ce" containerID="e1029d0a9968cfcbd673cd6acfed8b749433a60f7dfa3558245729d65976c53f" exitCode=0
Oct 30 19:13:29.946806 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:29.946737    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-8wnxx" event={"ID":"72e67919-78bc-4e8f-b8f1-13e34b8f54ce","Type":"ContainerDied","Data":"e1029d0a9968cfcbd673cd6acfed8b749433a60f7dfa3558245729d65976c53f"}
Oct 30 19:13:29.949698 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:29.949654    4525 generic.go:356] "Generic (PLEG): container finished" podID="52c00ed7-ee9c-422c-964d-e6c41b832d71" containerID="4fde6f098ba03db33e070586182a199d7c0e5c56406e63b51724aad49ac6a98e" exitCode=0
Oct 30 19:13:29.949913 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:29.949892    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f009fff9-76lm9" event={"ID":"52c00ed7-ee9c-422c-964d-e6c41b832d71","Type":"ContainerDied","Data":"4fde6f098ba03db33e070586182a199d7c0e5c56406e63b51724aad49ac6a98e"}
Oct 30 19:13:29.952272 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:29.952247    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-6pb9d"]
Oct 30 19:13:29.952773 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:29.952720    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-6pb9d" podUID="21d3419d-d29b-4e2b-8598-fe844028161e" containerName="istio-proxy" containerID="cri-o://a4836020c1f29976f62b485d0adb521ab0a7fddc503f4cbb0259bae8cdd1da30" gracePeriod=30
Oct 30 19:13:30.462947 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:30.462487    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-96zzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" start-of-body=
Oct 30 19:13:30.462947 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:30.462546    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-96zzd" podUID="33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused"
Oct 30 19:13:30.469727 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:30.467571    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body=
Oct 30 19:13:30.469727 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:30.467640    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45" podUID="d2f9a313-f069-4e48-8cdc-f75c6a285a7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused"
Oct 30 19:13:30.638224 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:30.638054    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-8wnxx"
Oct 30 19:13:30.674221 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:30.674173    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-9v7v9"]
Oct 30 19:13:30.707708 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:30.707634    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72e67919-78bc-4e8f-b8f1-13e34b8f54ce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72e67919-78bc-4e8f-b8f1-13e34b8f54ce-istio-data\") pod \"72e67919-78bc-4e8f-b8f1-13e34b8f54ce\" (UID: \"72e67919-78bc-4e8f-b8f1-13e34b8f54ce\") "
Oct 30 19:13:30.707900 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:30.707794    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72e67919-78bc-4e8f-b8f1-13e34b8f54ce-kube-api-access-4nc69\" (UniqueName: \"kubernetes.io/projected/72e67919-78bc-4e8f-b8f1-13e34b8f54ce-kube-api-access-4nc69\") pod \"72e67919-78bc-4e8f-b8f1-13e34b8f54ce\" (UID: \"72e67919-78bc-4e8f-b8f1-13e34b8f54ce\") "
Oct 30 19:13:30.712703 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:30.708031    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72e67919-78bc-4e8f-b8f1-13e34b8f54ce-istio-token\" (UniqueName: \"kubernetes.io/projected/72e67919-78bc-4e8f-b8f1-13e34b8f54ce-istio-token\") pod \"72e67919-78bc-4e8f-b8f1-13e34b8f54ce\" (UID: \"72e67919-78bc-4e8f-b8f1-13e34b8f54ce\") "
Oct 30 19:13:30.712703 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:30.708230    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72e67919-78bc-4e8f-b8f1-13e34b8f54ce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72e67919-78bc-4e8f-b8f1-13e34b8f54ce-credential-socket\") pod \"72e67919-78bc-4e8f-b8f1-13e34b8f54ce\" (UID: \"72e67919-78bc-4e8f-b8f1-13e34b8f54ce\") "
Oct 30 19:13:30.712703 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:30.708355    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72e67919-78bc-4e8f-b8f1-13e34b8f54ce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72e67919-78bc-4e8f-b8f1-13e34b8f54ce-istio-envoy\") pod \"72e67919-78bc-4e8f-b8f1-13e34b8f54ce\" (UID: \"72e67919-78bc-4e8f-b8f1-13e34b8f54ce\") "
Oct 30 19:13:30.712703 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:30.708490    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72e67919-78bc-4e8f-b8f1-13e34b8f54ce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72e67919-78bc-4e8f-b8f1-13e34b8f54ce-workload-certs\") pod \"72e67919-78bc-4e8f-b8f1-13e34b8f54ce\" (UID: \"72e67919-78bc-4e8f-b8f1-13e34b8f54ce\") "
Oct 30 19:13:30.716762 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:30.714494    4525 cpu_manager.go:414] "containerMap: removing container" podUID="72e67919-78bc-4e8f-b8f1-13e34b8f54ce" containerName="istio-proxy"
Oct 30 19:13:30.716762 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:30.714526    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72e67919-78bc-4e8f-b8f1-13e34b8f54ce" containerName="istio-proxy"
Oct 30 19:13:30.716762 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:30.715864    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="72e67919-78bc-4e8f-b8f1-13e34b8f54ce" containerName="istio-proxy"
Oct 30 19:13:30.718359 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:30.717337    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72e67919-78bc-4e8f-b8f1-13e34b8f54ce-istio-data" pod "72e67919-78bc-4e8f-b8f1-13e34b8f54ce" (UID: "72e67919-78bc-4e8f-b8f1-13e34b8f54ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:30.718359 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:30.718138    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72e67919-78bc-4e8f-b8f1-13e34b8f54ce-credential-socket" pod "72e67919-78bc-4e8f-b8f1-13e34b8f54ce" (UID: "72e67919-78bc-4e8f-b8f1-13e34b8f54ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:30.718811 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:30.718713    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72e67919-78bc-4e8f-b8f1-13e34b8f54ce-workload-certs" pod "72e67919-78bc-4e8f-b8f1-13e34b8f54ce" (UID: "72e67919-78bc-4e8f-b8f1-13e34b8f54ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:30.720279 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:30.719398    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72e67919-78bc-4e8f-b8f1-13e34b8f54ce-workload-socket" pod "72e67919-78bc-4e8f-b8f1-13e34b8f54ce" (UID: "72e67919-78bc-4e8f-b8f1-13e34b8f54ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:30.728720 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:30.728583    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72e67919-78bc-4e8f-b8f1-13e34b8f54ce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72e67919-78bc-4e8f-b8f1-13e34b8f54ce-workload-socket\") pod \"72e67919-78bc-4e8f-b8f1-13e34b8f54ce\" (UID: \"72e67919-78bc-4e8f-b8f1-13e34b8f54ce\") "
Oct 30 19:13:30.728720 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:30.728659    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/72e67919-78bc-4e8f-b8f1-13e34b8f54ce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72e67919-78bc-4e8f-b8f1-13e34b8f54ce-istio-podinfo\") pod \"72e67919-78bc-4e8f-b8f1-13e34b8f54ce\" (UID: \"72e67919-78bc-4e8f-b8f1-13e34b8f54ce\") "
Oct 30 19:13:30.728904 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:30.728781    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72e67919-78bc-4e8f-b8f1-13e34b8f54ce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72e67919-78bc-4e8f-b8f1-13e34b8f54ce-istiod-ca-cert\") pod \"72e67919-78bc-4e8f-b8f1-13e34b8f54ce\" (UID: \"72e67919-78bc-4e8f-b8f1-13e34b8f54ce\") "
Oct 30 19:13:30.735716 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:30.734397    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72e67919-78bc-4e8f-b8f1-13e34b8f54ce-istiod-ca-cert" pod "72e67919-78bc-4e8f-b8f1-13e34b8f54ce" (UID: "72e67919-78bc-4e8f-b8f1-13e34b8f54ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:13:30.736674 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:30.736652    4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72e67919-78bc-4e8f-b8f1-13e34b8f54ce-istio-data\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:30.736674 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:30.736720    4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72e67919-78bc-4e8f-b8f1-13e34b8f54ce-credential-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:30.736674 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:30.736738    4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72e67919-78bc-4e8f-b8f1-13e34b8f54ce-workload-certs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:30.736674 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:30.736752    4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72e67919-78bc-4e8f-b8f1-13e34b8f54ce-workload-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:30.736674 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:30.736766    4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72e67919-78bc-4e8f-b8f1-13e34b8f54ce-istiod-ca-cert\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:30.738454 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:30.738431    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72e67919-78bc-4e8f-b8f1-13e34b8f54ce-istio-token" pod "72e67919-78bc-4e8f-b8f1-13e34b8f54ce" (UID: "72e67919-78bc-4e8f-b8f1-13e34b8f54ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:13:30.743749 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:30.742726    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-9v7v9"
Oct 30 19:13:30.784052 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:30.784003    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/72e67919-78bc-4e8f-b8f1-13e34b8f54ce-istio-podinfo" pod "72e67919-78bc-4e8f-b8f1-13e34b8f54ce" (UID: "72e67919-78bc-4e8f-b8f1-13e34b8f54ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:13:30.789138 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:30.789108    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72e67919-78bc-4e8f-b8f1-13e34b8f54ce-kube-api-access-4nc69" pod "72e67919-78bc-4e8f-b8f1-13e34b8f54ce" (UID: "72e67919-78bc-4e8f-b8f1-13e34b8f54ce"). InnerVolumeSpecName "kube-api-access-4nc69". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:13:30.799738 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:30.793458    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72e67919-78bc-4e8f-b8f1-13e34b8f54ce-istio-envoy" pod "72e67919-78bc-4e8f-b8f1-13e34b8f54ce" (UID: "72e67919-78bc-4e8f-b8f1-13e34b8f54ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:30.821927 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:30.821721    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-9v7v9"]
Oct 30 19:13:30.838760 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:30.838716    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0856440-337e-4507-ad1a-bdce910e1989-workload-certs\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-9v7v9\" (UID: \"a0856440-337e-4507-ad1a-bdce910e1989\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-9v7v9"
Oct 30 19:13:30.838900 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:30.838804    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0856440-337e-4507-ad1a-bdce910e1989-istio-envoy\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-9v7v9\" (UID: \"a0856440-337e-4507-ad1a-bdce910e1989\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-9v7v9"
Oct 30 19:13:30.838900 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:30.838866    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0856440-337e-4507-ad1a-bdce910e1989-istio-podinfo\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-9v7v9\" (UID: \"a0856440-337e-4507-ad1a-bdce910e1989\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-9v7v9"
Oct 30 19:13:30.839026 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:30.838992    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0856440-337e-4507-ad1a-bdce910e1989-istio-data\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-9v7v9\" (UID: \"a0856440-337e-4507-ad1a-bdce910e1989\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-9v7v9"
Oct 30 19:13:30.839097 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:30.839024    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0856440-337e-4507-ad1a-bdce910e1989-istio-token\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-9v7v9\" (UID: \"a0856440-337e-4507-ad1a-bdce910e1989\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-9v7v9"
Oct 30 19:13:30.839152 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:30.839139    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0856440-337e-4507-ad1a-bdce910e1989-istiod-ca-cert\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-9v7v9\" (UID: \"a0856440-337e-4507-ad1a-bdce910e1989\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-9v7v9"
Oct 30 19:13:30.839546 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:30.839241    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjgjr\" (UniqueName: \"kubernetes.io/projected/a0856440-337e-4507-ad1a-bdce910e1989-kube-api-access-vjgjr\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-9v7v9\" (UID: \"a0856440-337e-4507-ad1a-bdce910e1989\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-9v7v9"
Oct 30 19:13:30.839546 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:30.839297    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0856440-337e-4507-ad1a-bdce910e1989-credential-socket\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-9v7v9\" (UID: \"a0856440-337e-4507-ad1a-bdce910e1989\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-9v7v9"
Oct 30 19:13:30.839546 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:30.839373    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0856440-337e-4507-ad1a-bdce910e1989-workload-socket\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-9v7v9\" (UID: \"a0856440-337e-4507-ad1a-bdce910e1989\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-9v7v9"
Oct 30 19:13:30.839546 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:30.839466    4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72e67919-78bc-4e8f-b8f1-13e34b8f54ce-istio-token\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:30.839546 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:30.839482    4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72e67919-78bc-4e8f-b8f1-13e34b8f54ce-istio-envoy\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:30.839546 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:30.839495    4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72e67919-78bc-4e8f-b8f1-13e34b8f54ce-istio-podinfo\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:30.839546 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:30.839510    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nc69\" (UniqueName: \"kubernetes.io/projected/72e67919-78bc-4e8f-b8f1-13e34b8f54ce-kube-api-access-4nc69\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:30.961773 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:30.961550    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0856440-337e-4507-ad1a-bdce910e1989-workload-socket\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-9v7v9\" (UID: \"a0856440-337e-4507-ad1a-bdce910e1989\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-9v7v9"
Oct 30 19:13:30.962032 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:30.961805    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0856440-337e-4507-ad1a-bdce910e1989-workload-certs\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-9v7v9\" (UID: \"a0856440-337e-4507-ad1a-bdce910e1989\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-9v7v9"
Oct 30 19:13:30.962032 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:30.961908    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0856440-337e-4507-ad1a-bdce910e1989-istio-envoy\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-9v7v9\" (UID: \"a0856440-337e-4507-ad1a-bdce910e1989\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-9v7v9"
Oct 30 19:13:30.962032 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:30.962023    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0856440-337e-4507-ad1a-bdce910e1989-istio-podinfo\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-9v7v9\" (UID: \"a0856440-337e-4507-ad1a-bdce910e1989\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-9v7v9"
Oct 30 19:13:30.962214 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:30.962196    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0856440-337e-4507-ad1a-bdce910e1989-istio-data\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-9v7v9\" (UID: \"a0856440-337e-4507-ad1a-bdce910e1989\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-9v7v9"
Oct 30 19:13:30.962271 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:30.962228    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0856440-337e-4507-ad1a-bdce910e1989-istio-token\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-9v7v9\" (UID: \"a0856440-337e-4507-ad1a-bdce910e1989\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-9v7v9"
Oct 30 19:13:30.962348 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:30.962311    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0856440-337e-4507-ad1a-bdce910e1989-istiod-ca-cert\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-9v7v9\" (UID: \"a0856440-337e-4507-ad1a-bdce910e1989\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-9v7v9"
Oct 30 19:13:30.962994 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:30.962972    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0856440-337e-4507-ad1a-bdce910e1989-workload-socket\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-9v7v9\" (UID: \"a0856440-337e-4507-ad1a-bdce910e1989\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-9v7v9"
Oct 30 19:13:30.963174 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:30.963063    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0856440-337e-4507-ad1a-bdce910e1989-workload-certs\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-9v7v9\" (UID: \"a0856440-337e-4507-ad1a-bdce910e1989\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-9v7v9"
Oct 30 19:13:30.963371 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:30.963306    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0856440-337e-4507-ad1a-bdce910e1989-istio-data\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-9v7v9\" (UID: \"a0856440-337e-4507-ad1a-bdce910e1989\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-9v7v9"
Oct 30 19:13:30.980762 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:30.963958    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjgjr\" (UniqueName: \"kubernetes.io/projected/a0856440-337e-4507-ad1a-bdce910e1989-kube-api-access-vjgjr\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-9v7v9\" (UID: \"a0856440-337e-4507-ad1a-bdce910e1989\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-9v7v9"
Oct 30 19:13:30.980762 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:30.964031    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0856440-337e-4507-ad1a-bdce910e1989-credential-socket\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-9v7v9\" (UID: \"a0856440-337e-4507-ad1a-bdce910e1989\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-9v7v9"
Oct 30 19:13:30.980762 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:30.967518    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0856440-337e-4507-ad1a-bdce910e1989-istiod-ca-cert\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-9v7v9\" (UID: \"a0856440-337e-4507-ad1a-bdce910e1989\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-9v7v9"
Oct 30 19:13:30.980762 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:30.974873    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0856440-337e-4507-ad1a-bdce910e1989-credential-socket\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-9v7v9\" (UID: \"a0856440-337e-4507-ad1a-bdce910e1989\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-9v7v9"
Oct 30 19:13:30.992036 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:30.986733    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0856440-337e-4507-ad1a-bdce910e1989-istio-podinfo\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-9v7v9\" (UID: \"a0856440-337e-4507-ad1a-bdce910e1989\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-9v7v9"
Oct 30 19:13:31.026515 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:31.026407    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjgjr\" (UniqueName: \"kubernetes.io/projected/a0856440-337e-4507-ad1a-bdce910e1989-kube-api-access-vjgjr\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-9v7v9\" (UID: \"a0856440-337e-4507-ad1a-bdce910e1989\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-9v7v9"
Oct 30 19:13:31.027242 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:31.027010    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0856440-337e-4507-ad1a-bdce910e1989-istio-envoy\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-9v7v9\" (UID: \"a0856440-337e-4507-ad1a-bdce910e1989\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-9v7v9"
Oct 30 19:13:31.032703 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:31.030219    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0856440-337e-4507-ad1a-bdce910e1989-istio-token\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-9v7v9\" (UID: \"a0856440-337e-4507-ad1a-bdce910e1989\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-9v7v9"
Oct 30 19:13:31.036241 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:31.035388    4525 generic.go:356] "Generic (PLEG): container finished" podID="3330720a-f130-4cd2-8156-643a79a24235" containerID="f2e5fb680cdbedf57e345dcaca5ac439e1d270920ec383be3498906e46da14c6" exitCode=0
Oct 30 19:13:31.036241 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:31.035570    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw" event={"ID":"3330720a-f130-4cd2-8156-643a79a24235","Type":"ContainerDied","Data":"f2e5fb680cdbedf57e345dcaca5ac439e1d270920ec383be3498906e46da14c6"}
Oct 30 19:13:31.036241 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:31.035628    4525 scope.go:122] "RemoveContainer" containerID="8f922ecb3e1d7b4a7c194ed789ba3fbba1a01f20884f74726110792f014f4b2e"
Oct 30 19:13:31.040312 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:31.040242    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-8wnxx"
Oct 30 19:13:31.040312 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:31.040253    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-8wnxx" event={"ID":"72e67919-78bc-4e8f-b8f1-13e34b8f54ce","Type":"ContainerDied","Data":"99a2002736016d93fa48c438db3f1b8c38ddc6f611881105c4ce4f026fa63b1f"}
Oct 30 19:13:31.049828 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:31.049800    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-d7mth"
Oct 30 19:13:31.050700 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:31.050635    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-d7mth"
Oct 30 19:13:31.053165 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:31.053150    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-d7mth"
Oct 30 19:13:31.063740 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:31.063673    4525 scope.go:122] "RemoveContainer" containerID="e1029d0a9968cfcbd673cd6acfed8b749433a60f7dfa3558245729d65976c53f"
Oct 30 19:13:31.126728 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:31.126010    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-96zzd"
Oct 30 19:13:31.127449 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:31.127403    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-96zzd"
Oct 30 19:13:31.135005 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:31.131902    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-96zzd"
Oct 30 19:13:31.143707 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:31.143635    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-8wnxx"]
Oct 30 19:13:31.150660 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:31.150637    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-8wnxx"]
Oct 30 19:13:31.223047 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:31.222982    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-229cp"]
Oct 30 19:13:31.223895 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:31.223862    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-229cp" podUID="5d9a2656-6bae-4277-a7f3-b43e8143dbd7" containerName="istio-proxy" containerID="cri-o://7396b8db18338af05d47281d1cfb35fe280b6afbe7bd7a122a987ac114fa04f7" gracePeriod=30
Oct 30 19:13:31.234831 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:31.234464    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-9v7v9"
Oct 30 19:13:31.369812 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:31.369776    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw"
Oct 30 19:13:31.425572 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:31.425538    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body=
Oct 30 19:13:31.427623 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:31.427590    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45" podUID="d2f9a313-f069-4e48-8cdc-f75c6a285a7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused"
Oct 30 19:13:31.446526 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:31.446502    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f009fff9-76lm9"
Oct 30 19:13:31.461706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:31.459842    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-ecc266-559cbb854f-8wvqg"]
Oct 30 19:13:31.461706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:31.461091    4525 cpu_manager.go:414] "containerMap: removing container" podUID="3330720a-f130-4cd2-8156-643a79a24235" containerName="istio-proxy"
Oct 30 19:13:31.461706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:31.461112    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3330720a-f130-4cd2-8156-643a79a24235" containerName="istio-proxy"
Oct 30 19:13:31.461706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:31.461143    4525 cpu_manager.go:414] "containerMap: removing container" podUID="52c00ed7-ee9c-422c-964d-e6c41b832d71" containerName="revoke-keys"
Oct 30 19:13:31.461706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:31.461151    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52c00ed7-ee9c-422c-964d-e6c41b832d71" containerName="revoke-keys"
Oct 30 19:13:31.461706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:31.461339    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="3330720a-f130-4cd2-8156-643a79a24235" containerName="istio-proxy"
Oct 30 19:13:31.461706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:31.461375    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="3330720a-f130-4cd2-8156-643a79a24235" containerName="istio-proxy"
Oct 30 19:13:31.461706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:31.461390    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="52c00ed7-ee9c-422c-964d-e6c41b832d71" containerName="revoke-keys"
Oct 30 19:13:31.469000 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:31.468978    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ecc266-559cbb854f-8wvqg"
Oct 30 19:13:31.472047 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:31.472013    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-ecc266-559cbb854f-8wvqg"]
Oct 30 19:13:31.473484 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:31.473459    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-ecc266-serving-cert\""
Oct 30 19:13:31.482708 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:31.475777    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3330720a-f130-4cd2-8156-643a79a24235-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3330720a-f130-4cd2-8156-643a79a24235-istio-podinfo\") pod \"3330720a-f130-4cd2-8156-643a79a24235\" (UID: \"3330720a-f130-4cd2-8156-643a79a24235\") "
Oct 30 19:13:31.509306 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:31.475822    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3330720a-f130-4cd2-8156-643a79a24235-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3330720a-f130-4cd2-8156-643a79a24235-istiod-ca-cert\") pod \"3330720a-f130-4cd2-8156-643a79a24235\" (UID: \"3330720a-f130-4cd2-8156-643a79a24235\") "
Oct 30 19:13:31.509306 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:31.482983    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3330720a-f130-4cd2-8156-643a79a24235-istio-token\" (UniqueName: \"kubernetes.io/projected/3330720a-f130-4cd2-8156-643a79a24235-istio-token\") pod \"3330720a-f130-4cd2-8156-643a79a24235\" (UID: \"3330720a-f130-4cd2-8156-643a79a24235\") "
Oct 30 19:13:31.509306 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:31.483114    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3330720a-f130-4cd2-8156-643a79a24235-kube-api-access-k27m6\" (UniqueName: \"kubernetes.io/projected/3330720a-f130-4cd2-8156-643a79a24235-kube-api-access-k27m6\") pod \"3330720a-f130-4cd2-8156-643a79a24235\" (UID: \"3330720a-f130-4cd2-8156-643a79a24235\") "
Oct 30 19:13:31.509306 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:31.483169    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3330720a-f130-4cd2-8156-643a79a24235-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3330720a-f130-4cd2-8156-643a79a24235-workload-socket\") pod \"3330720a-f130-4cd2-8156-643a79a24235\" (UID: \"3330720a-f130-4cd2-8156-643a79a24235\") "
Oct 30 19:13:31.509306 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:31.483323    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3330720a-f130-4cd2-8156-643a79a24235-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3330720a-f130-4cd2-8156-643a79a24235-istio-envoy\") pod \"3330720a-f130-4cd2-8156-643a79a24235\" (UID: \"3330720a-f130-4cd2-8156-643a79a24235\") "
Oct 30 19:13:31.509306 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:31.483399    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3330720a-f130-4cd2-8156-643a79a24235-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3330720a-f130-4cd2-8156-643a79a24235-istio-data\") pod \"3330720a-f130-4cd2-8156-643a79a24235\" (UID: \"3330720a-f130-4cd2-8156-643a79a24235\") "
Oct 30 19:13:31.509306 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:31.483506    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3330720a-f130-4cd2-8156-643a79a24235-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3330720a-f130-4cd2-8156-643a79a24235-workload-certs\") pod \"3330720a-f130-4cd2-8156-643a79a24235\" (UID: \"3330720a-f130-4cd2-8156-643a79a24235\") "
Oct 30 19:13:31.509306 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:31.483629    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3330720a-f130-4cd2-8156-643a79a24235-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3330720a-f130-4cd2-8156-643a79a24235-credential-socket\") pod \"3330720a-f130-4cd2-8156-643a79a24235\" (UID: \"3330720a-f130-4cd2-8156-643a79a24235\") "
Oct 30 19:13:31.509306 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:31.508254    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3330720a-f130-4cd2-8156-643a79a24235-istiod-ca-cert" pod "3330720a-f130-4cd2-8156-643a79a24235" (UID: "3330720a-f130-4cd2-8156-643a79a24235"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:13:31.510382 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:31.510091    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3330720a-f130-4cd2-8156-643a79a24235-credential-socket" pod "3330720a-f130-4cd2-8156-643a79a24235" (UID: "3330720a-f130-4cd2-8156-643a79a24235"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:31.529409 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:31.511376    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3330720a-f130-4cd2-8156-643a79a24235-workload-certs" pod "3330720a-f130-4cd2-8156-643a79a24235" (UID: "3330720a-f130-4cd2-8156-643a79a24235"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:31.529409 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:31.512656    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3330720a-f130-4cd2-8156-643a79a24235-istio-data" pod "3330720a-f130-4cd2-8156-643a79a24235" (UID: "3330720a-f130-4cd2-8156-643a79a24235"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:31.529729 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:31.529639    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3330720a-f130-4cd2-8156-643a79a24235-istio-envoy" pod "3330720a-f130-4cd2-8156-643a79a24235" (UID: "3330720a-f130-4cd2-8156-643a79a24235"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:31.533030 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:31.532992    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3330720a-f130-4cd2-8156-643a79a24235-istio-token" pod "3330720a-f130-4cd2-8156-643a79a24235" (UID: "3330720a-f130-4cd2-8156-643a79a24235"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:13:31.533861 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:31.533509    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3330720a-f130-4cd2-8156-643a79a24235-istio-podinfo" pod "3330720a-f130-4cd2-8156-643a79a24235" (UID: "3330720a-f130-4cd2-8156-643a79a24235"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:13:31.535265 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:31.535224    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3330720a-f130-4cd2-8156-643a79a24235-kube-api-access-k27m6" pod "3330720a-f130-4cd2-8156-643a79a24235" (UID: "3330720a-f130-4cd2-8156-643a79a24235"). InnerVolumeSpecName "kube-api-access-k27m6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:13:31.539429 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:31.539389    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3330720a-f130-4cd2-8156-643a79a24235-workload-socket" pod "3330720a-f130-4cd2-8156-643a79a24235" (UID: "3330720a-f130-4cd2-8156-643a79a24235"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:31.574475 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:31.572667    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72e67919-78bc-4e8f-b8f1-13e34b8f54ce" path="/var/lib/kubelet/pods/72e67919-78bc-4e8f-b8f1-13e34b8f54ce/volumes"
Oct 30 19:13:31.607706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:31.590621    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52c00ed7-ee9c-422c-964d-e6c41b832d71-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52c00ed7-ee9c-422c-964d-e6c41b832d71-maas-api-openshift-ca\") pod \"52c00ed7-ee9c-422c-964d-e6c41b832d71\" (UID: \"52c00ed7-ee9c-422c-964d-e6c41b832d71\") "
Oct 30 19:13:31.607706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:31.590699    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52c00ed7-ee9c-422c-964d-e6c41b832d71-kube-api-access-mksds\" (UniqueName: \"kubernetes.io/projected/52c00ed7-ee9c-422c-964d-e6c41b832d71-kube-api-access-mksds\") pod \"52c00ed7-ee9c-422c-964d-e6c41b832d71\" (UID: \"52c00ed7-ee9c-422c-964d-e6c41b832d71\") "
Oct 30 19:13:31.607706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:31.590792    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52c00ed7-ee9c-422c-964d-e6c41b832d71-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52c00ed7-ee9c-422c-964d-e6c41b832d71-maas-api-xks-ca\") pod \"52c00ed7-ee9c-422c-964d-e6c41b832d71\" (UID: \"52c00ed7-ee9c-422c-964d-e6c41b832d71\") "
Oct 30 19:13:31.607706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:31.591110    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc021748-e220-42d9-9e39-029115dade4b-metrics-certs\") pod \"maas-api-e2e-extmod-b-ecc266-559cbb854f-8wvqg\" (UID: \"bc021748-e220-42d9-9e39-029115dade4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ecc266-559cbb854f-8wvqg"
Oct 30 19:13:31.607706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:31.591175    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc021748-e220-42d9-9e39-029115dade4b-maas-api-tls\") pod \"maas-api-e2e-extmod-b-ecc266-559cbb854f-8wvqg\" (UID: \"bc021748-e220-42d9-9e39-029115dade4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ecc266-559cbb854f-8wvqg"
Oct 30 19:13:31.607706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:31.591462    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc021748-e220-42d9-9e39-029115dade4b-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-ecc266-559cbb854f-8wvqg\" (UID: \"bc021748-e220-42d9-9e39-029115dade4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ecc266-559cbb854f-8wvqg"
Oct 30 19:13:31.607706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:31.591492    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6szt\" (UniqueName: \"kubernetes.io/projected/bc021748-e220-42d9-9e39-029115dade4b-kube-api-access-c6szt\") pod \"maas-api-e2e-extmod-b-ecc266-559cbb854f-8wvqg\" (UID: \"bc021748-e220-42d9-9e39-029115dade4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ecc266-559cbb854f-8wvqg"
Oct 30 19:13:31.607706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:31.591781    4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3330720a-f130-4cd2-8156-643a79a24235-istio-podinfo\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:31.607706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:31.591803    4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3330720a-f130-4cd2-8156-643a79a24235-istiod-ca-cert\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:31.607706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:31.591818    4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3330720a-f130-4cd2-8156-643a79a24235-istio-token\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:31.607706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:31.591831    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k27m6\" (UniqueName: \"kubernetes.io/projected/3330720a-f130-4cd2-8156-643a79a24235-kube-api-access-k27m6\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:31.607706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:31.591846    4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3330720a-f130-4cd2-8156-643a79a24235-workload-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:31.607706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:31.591861    4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3330720a-f130-4cd2-8156-643a79a24235-istio-envoy\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:31.607706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:31.591873    4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3330720a-f130-4cd2-8156-643a79a24235-istio-data\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:31.607706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:31.591887    4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3330720a-f130-4cd2-8156-643a79a24235-workload-certs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:31.607706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:31.591900    4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3330720a-f130-4cd2-8156-643a79a24235-credential-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:31.607706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:31.599907    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52c00ed7-ee9c-422c-964d-e6c41b832d71-maas-api-openshift-ca" pod "52c00ed7-ee9c-422c-964d-e6c41b832d71" (UID: "52c00ed7-ee9c-422c-964d-e6c41b832d71"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:13:31.616138 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:31.616110    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52c00ed7-ee9c-422c-964d-e6c41b832d71-kube-api-access-mksds" pod "52c00ed7-ee9c-422c-964d-e6c41b832d71" (UID: "52c00ed7-ee9c-422c-964d-e6c41b832d71"). InnerVolumeSpecName "kube-api-access-mksds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:13:31.617868 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:31.617841    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52c00ed7-ee9c-422c-964d-e6c41b832d71-maas-api-xks-ca" pod "52c00ed7-ee9c-422c-964d-e6c41b832d71" (UID: "52c00ed7-ee9c-422c-964d-e6c41b832d71"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:13:31.737377 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:31.724373    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc021748-e220-42d9-9e39-029115dade4b-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-ecc266-559cbb854f-8wvqg\" (UID: \"bc021748-e220-42d9-9e39-029115dade4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ecc266-559cbb854f-8wvqg"
Oct 30 19:13:31.737377 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:31.724422    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6szt\" (UniqueName: \"kubernetes.io/projected/bc021748-e220-42d9-9e39-029115dade4b-kube-api-access-c6szt\") pod \"maas-api-e2e-extmod-b-ecc266-559cbb854f-8wvqg\" (UID: \"bc021748-e220-42d9-9e39-029115dade4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ecc266-559cbb854f-8wvqg"
Oct 30 19:13:31.737377 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:31.726114    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc021748-e220-42d9-9e39-029115dade4b-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-ecc266-559cbb854f-8wvqg\" (UID: \"bc021748-e220-42d9-9e39-029115dade4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ecc266-559cbb854f-8wvqg"
Oct 30 19:13:31.742576 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:31.741289    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc021748-e220-42d9-9e39-029115dade4b-metrics-certs\") pod \"maas-api-e2e-extmod-b-ecc266-559cbb854f-8wvqg\" (UID: \"bc021748-e220-42d9-9e39-029115dade4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ecc266-559cbb854f-8wvqg"
Oct 30 19:13:31.742576 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:31.741657    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc021748-e220-42d9-9e39-029115dade4b-maas-api-tls\") pod \"maas-api-e2e-extmod-b-ecc266-559cbb854f-8wvqg\" (UID: \"bc021748-e220-42d9-9e39-029115dade4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ecc266-559cbb854f-8wvqg"
Oct 30 19:13:31.743330 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:31.743226    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52c00ed7-ee9c-422c-964d-e6c41b832d71-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:31.743330 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:31.743268    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mksds\" (UniqueName: \"kubernetes.io/projected/52c00ed7-ee9c-422c-964d-e6c41b832d71-kube-api-access-mksds\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:31.743330 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:31.743285    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52c00ed7-ee9c-422c-964d-e6c41b832d71-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:31.758712 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:31.758660    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-9v7v9"]
Oct 30 19:13:31.760843 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:31.760787    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc021748-e220-42d9-9e39-029115dade4b-maas-api-tls\") pod \"maas-api-e2e-extmod-b-ecc266-559cbb854f-8wvqg\" (UID: \"bc021748-e220-42d9-9e39-029115dade4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ecc266-559cbb854f-8wvqg"
Oct 30 19:13:31.766183 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:31.762218    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6szt\" (UniqueName: \"kubernetes.io/projected/bc021748-e220-42d9-9e39-029115dade4b-kube-api-access-c6szt\") pod \"maas-api-e2e-extmod-b-ecc266-559cbb854f-8wvqg\" (UID: \"bc021748-e220-42d9-9e39-029115dade4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ecc266-559cbb854f-8wvqg"
Oct 30 19:13:31.766183 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:31.764667    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc021748-e220-42d9-9e39-029115dade4b-metrics-certs\") pod \"maas-api-e2e-extmod-b-ecc266-559cbb854f-8wvqg\" (UID: \"bc021748-e220-42d9-9e39-029115dade4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ecc266-559cbb854f-8wvqg"
Oct 30 19:13:31.772216 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:31.772181    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:13:31.772327 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:31.772284    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:13:31.772391 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:31.772337    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:13:32.043429 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:32.043153    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ecc266-559cbb854f-8wvqg"
Oct 30 19:13:32.060148 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:32.060099    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f009fff9-76lm9"
Oct 30 19:13:32.060334 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:32.060112    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f009fff9-76lm9" event={"ID":"52c00ed7-ee9c-422c-964d-e6c41b832d71","Type":"ContainerDied","Data":"e584e4cb6c8e6ffbc0afe79546b9a43de53fb0f923c2c96242a323478ddda9ed"}
Oct 30 19:13:32.060334 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:32.060241    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e584e4cb6c8e6ffbc0afe79546b9a43de53fb0f923c2c96242a323478ddda9ed"
Oct 30 19:13:32.062847 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:32.062822    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-9v7v9" event={"ID":"a0856440-337e-4507-ad1a-bdce910e1989","Type":"ContainerStarted","Data":"d5d74098339e199a238c446030a53edb63529d268797ed9df5a630493f0ebcd0"}
Oct 30 19:13:32.062969 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:32.062867    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-9v7v9" event={"ID":"a0856440-337e-4507-ad1a-bdce910e1989","Type":"ContainerStarted","Data":"6f9e2954da2f9fb4a249e60fef1c6bc5429fb96ec0b3958fed2ffd57ef6c3d41"}
Oct 30 19:13:32.066808 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:32.066790    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw"
Oct 30 19:13:32.066910 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:32.066836    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw" event={"ID":"3330720a-f130-4cd2-8156-643a79a24235","Type":"ContainerDied","Data":"c83d9ca472d9d0b63c49603a86297e718e5701ec0664f0efbf8ccbdc50fce136"}
Oct 30 19:13:32.066910 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:32.066887    4525 scope.go:122] "RemoveContainer" containerID="f2e5fb680cdbedf57e345dcaca5ac439e1d270920ec383be3498906e46da14c6"
Oct 30 19:13:32.084666 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:32.084603    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-9v7v9" podStartSLOduration=73.084582949 podStartE2EDuration="1m13.084582949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:12:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:13:32.083668723 +0000 UTC m=+2234.709338680" watchObservedRunningTime="2026-10-30 19:13:32.084582949 +0000 UTC m=+2234.710252914"
Oct 30 19:13:32.104145 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:32.104107    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw"]
Oct 30 19:13:32.110936 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:32.110900    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-dcd326-openshift-default-d787c4f6f-rvcsw"]
Oct 30 19:13:32.235433 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:32.235403    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-9v7v9"
Oct 30 19:13:32.236818 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:32.236788    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-9v7v9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body=
Oct 30 19:13:32.236960 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:32.236850    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-9v7v9" podUID="a0856440-337e-4507-ad1a-bdce910e1989" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused"
Oct 30 19:13:32.358335 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:13:32.358299    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc021748_e220_42d9_9e39_029115dade4b.slice/crio-a2a761a5adef1d2dff520551fa19ebec3d04806b531ac3a7448656ee8a116709 WatchSource:0}: Error finding container a2a761a5adef1d2dff520551fa19ebec3d04806b531ac3a7448656ee8a116709: Status 404 returned error can't find the container with id a2a761a5adef1d2dff520551fa19ebec3d04806b531ac3a7448656ee8a116709
Oct 30 19:13:32.358483 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:32.358460    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-ecc266-559cbb854f-8wvqg"]
Oct 30 19:13:32.413609 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:32.413576    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45"
Oct 30 19:13:32.414015 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:32.413994    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45"
Oct 30 19:13:32.414616 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:32.414594    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45"
Oct 30 19:13:33.096708 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:33.090836    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ecc266-559cbb854f-8wvqg" event={"ID":"bc021748-e220-42d9-9e39-029115dade4b","Type":"ContainerStarted","Data":"255e3f0fb10a28ee3a6022a6d455dde70711138dccc92715cedec8617113d664"}
Oct 30 19:13:33.096708 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:33.090957    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ecc266-559cbb854f-8wvqg" event={"ID":"bc021748-e220-42d9-9e39-029115dade4b","Type":"ContainerStarted","Data":"a2a761a5adef1d2dff520551fa19ebec3d04806b531ac3a7448656ee8a116709"}
Oct 30 19:13:33.096708 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:33.092345    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ecc266-559cbb854f-8wvqg"
Oct 30 19:13:33.239828 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:33.239195    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-9v7v9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body=
Oct 30 19:13:33.239828 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:33.239258    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-9v7v9" podUID="a0856440-337e-4507-ad1a-bdce910e1989" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused"
Oct 30 19:13:33.549133 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:33.549051    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3330720a-f130-4cd2-8156-643a79a24235" path="/var/lib/kubelet/pods/3330720a-f130-4cd2-8156-643a79a24235/volumes"
Oct 30 19:13:34.236128 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:34.236087    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-9v7v9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body=
Oct 30 19:13:34.236545 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:34.236156    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-9v7v9" podUID="a0856440-337e-4507-ad1a-bdce910e1989" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused"
Oct 30 19:13:34.951063 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:34.951036    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-dhclt"
Oct 30 19:13:35.020806 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.020119    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ecc266-559cbb854f-8wvqg" podStartSLOduration=155.474576059 podStartE2EDuration="2m36.020095859s" totalImagesPullingTime="545.5198ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:10:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:13:33.139040157 +0000 UTC m=+2235.764710122" watchObservedRunningTime="2026-10-30 19:13:35.020095859 +0000 UTC m=+2237.645765823"
Oct 30 19:13:35.034146 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.034115    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw"]
Oct 30 19:13:35.035402 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.035375    4525 cpu_manager.go:414] "containerMap: removing container" podUID="0e0636e7-6f61-4843-844d-fb9d7bb527da" containerName="istio-proxy"
Oct 30 19:13:35.035578 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.035563    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e0636e7-6f61-4843-844d-fb9d7bb527da" containerName="istio-proxy"
Oct 30 19:13:35.035672 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.035662    4525 cpu_manager.go:414] "containerMap: removing container" podUID="3330720a-f130-4cd2-8156-643a79a24235" containerName="istio-proxy"
Oct 30 19:13:35.035782 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.035768    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3330720a-f130-4cd2-8156-643a79a24235" containerName="istio-proxy"
Oct 30 19:13:35.036178 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.036160    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e0636e7-6f61-4843-844d-fb9d7bb527da" containerName="istio-proxy"
Oct 30 19:13:35.049873 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.049847    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw"]
Oct 30 19:13:35.050232 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.049979    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw"
Oct 30 19:13:35.132232 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.132140    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e0636e7-6f61-4843-844d-fb9d7bb527da-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e0636e7-6f61-4843-844d-fb9d7bb527da-workload-certs\") pod \"0e0636e7-6f61-4843-844d-fb9d7bb527da\" (UID: \"0e0636e7-6f61-4843-844d-fb9d7bb527da\") "
Oct 30 19:13:35.141308 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.141258    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e0636e7-6f61-4843-844d-fb9d7bb527da-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e0636e7-6f61-4843-844d-fb9d7bb527da-credential-socket\") pod \"0e0636e7-6f61-4843-844d-fb9d7bb527da\" (UID: \"0e0636e7-6f61-4843-844d-fb9d7bb527da\") "
Oct 30 19:13:35.141565 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.134204    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e0636e7-6f61-4843-844d-fb9d7bb527da-workload-certs" pod "0e0636e7-6f61-4843-844d-fb9d7bb527da" (UID: "0e0636e7-6f61-4843-844d-fb9d7bb527da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:35.141669 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.141532    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e0636e7-6f61-4843-844d-fb9d7bb527da-credential-socket" pod "0e0636e7-6f61-4843-844d-fb9d7bb527da" (UID: "0e0636e7-6f61-4843-844d-fb9d7bb527da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:35.143068 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.143039    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e0636e7-6f61-4843-844d-fb9d7bb527da-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e0636e7-6f61-4843-844d-fb9d7bb527da-workload-socket\") pod \"0e0636e7-6f61-4843-844d-fb9d7bb527da\" (UID: \"0e0636e7-6f61-4843-844d-fb9d7bb527da\") "
Oct 30 19:13:35.143197 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.143107    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e0636e7-6f61-4843-844d-fb9d7bb527da-kube-api-access-ldkb5\" (UniqueName: \"kubernetes.io/projected/0e0636e7-6f61-4843-844d-fb9d7bb527da-kube-api-access-ldkb5\") pod \"0e0636e7-6f61-4843-844d-fb9d7bb527da\" (UID: \"0e0636e7-6f61-4843-844d-fb9d7bb527da\") "
Oct 30 19:13:35.143398 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.143268    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e0636e7-6f61-4843-844d-fb9d7bb527da-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e0636e7-6f61-4843-844d-fb9d7bb527da-istio-data\") pod \"0e0636e7-6f61-4843-844d-fb9d7bb527da\" (UID: \"0e0636e7-6f61-4843-844d-fb9d7bb527da\") "
Oct 30 19:13:35.143398 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.143334    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e0636e7-6f61-4843-844d-fb9d7bb527da-istio-token\" (UniqueName: \"kubernetes.io/projected/0e0636e7-6f61-4843-844d-fb9d7bb527da-istio-token\") pod \"0e0636e7-6f61-4843-844d-fb9d7bb527da\" (UID: \"0e0636e7-6f61-4843-844d-fb9d7bb527da\") "
Oct 30 19:13:35.143538 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.143410    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e0636e7-6f61-4843-844d-fb9d7bb527da-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e0636e7-6f61-4843-844d-fb9d7bb527da-istio-envoy\") pod \"0e0636e7-6f61-4843-844d-fb9d7bb527da\" (UID: \"0e0636e7-6f61-4843-844d-fb9d7bb527da\") "
Oct 30 19:13:35.143612 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.143574    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0e0636e7-6f61-4843-844d-fb9d7bb527da-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e0636e7-6f61-4843-844d-fb9d7bb527da-istio-podinfo\") pod \"0e0636e7-6f61-4843-844d-fb9d7bb527da\" (UID: \"0e0636e7-6f61-4843-844d-fb9d7bb527da\") "
Oct 30 19:13:35.143672 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.143616    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e0636e7-6f61-4843-844d-fb9d7bb527da-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e0636e7-6f61-4843-844d-fb9d7bb527da-istiod-ca-cert\") pod \"0e0636e7-6f61-4843-844d-fb9d7bb527da\" (UID: \"0e0636e7-6f61-4843-844d-fb9d7bb527da\") "
Oct 30 19:13:35.144746 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.144715    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd0977b0-c75d-41ee-b911-4e71bd8c9b3a-istio-podinfo\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw\" (UID: \"bd0977b0-c75d-41ee-b911-4e71bd8c9b3a\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw"
Oct 30 19:13:35.144915 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.144894    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd0977b0-c75d-41ee-b911-4e71bd8c9b3a-workload-socket\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw\" (UID: \"bd0977b0-c75d-41ee-b911-4e71bd8c9b3a\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw"
Oct 30 19:13:35.144993 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.144955    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd0977b0-c75d-41ee-b911-4e71bd8c9b3a-credential-socket\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw\" (UID: \"bd0977b0-c75d-41ee-b911-4e71bd8c9b3a\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw"
Oct 30 19:13:35.145049 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.145010    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd0977b0-c75d-41ee-b911-4e71bd8c9b3a-workload-certs\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw\" (UID: \"bd0977b0-c75d-41ee-b911-4e71bd8c9b3a\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw"
Oct 30 19:13:35.145123 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.145107    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8psn\" (UniqueName: \"kubernetes.io/projected/bd0977b0-c75d-41ee-b911-4e71bd8c9b3a-kube-api-access-h8psn\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw\" (UID: \"bd0977b0-c75d-41ee-b911-4e71bd8c9b3a\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw"
Oct 30 19:13:35.145196 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.145183    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd0977b0-c75d-41ee-b911-4e71bd8c9b3a-istio-envoy\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw\" (UID: \"bd0977b0-c75d-41ee-b911-4e71bd8c9b3a\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw"
Oct 30 19:13:35.145338 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.145320    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd0977b0-c75d-41ee-b911-4e71bd8c9b3a-istiod-ca-cert\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw\" (UID: \"bd0977b0-c75d-41ee-b911-4e71bd8c9b3a\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw"
Oct 30 19:13:35.145401 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.145380    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd0977b0-c75d-41ee-b911-4e71bd8c9b3a-istio-data\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw\" (UID: \"bd0977b0-c75d-41ee-b911-4e71bd8c9b3a\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw"
Oct 30 19:13:35.145469 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.145450    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd0977b0-c75d-41ee-b911-4e71bd8c9b3a-istio-token\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw\" (UID: \"bd0977b0-c75d-41ee-b911-4e71bd8c9b3a\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw"
Oct 30 19:13:35.145726 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.145707    4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e0636e7-6f61-4843-844d-fb9d7bb527da-workload-certs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:35.145792 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.145740    4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e0636e7-6f61-4843-844d-fb9d7bb527da-credential-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:35.146917 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.146337    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e0636e7-6f61-4843-844d-fb9d7bb527da-kube-api-access-ldkb5" pod "0e0636e7-6f61-4843-844d-fb9d7bb527da" (UID: "0e0636e7-6f61-4843-844d-fb9d7bb527da"). InnerVolumeSpecName "kube-api-access-ldkb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:13:35.146917 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.146569    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e0636e7-6f61-4843-844d-fb9d7bb527da-workload-socket" pod "0e0636e7-6f61-4843-844d-fb9d7bb527da" (UID: "0e0636e7-6f61-4843-844d-fb9d7bb527da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:35.147432 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.147401    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e0636e7-6f61-4843-844d-fb9d7bb527da-istio-data" pod "0e0636e7-6f61-4843-844d-fb9d7bb527da" (UID: "0e0636e7-6f61-4843-844d-fb9d7bb527da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:35.149672 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.149644    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e0636e7-6f61-4843-844d-fb9d7bb527da-istiod-ca-cert" pod "0e0636e7-6f61-4843-844d-fb9d7bb527da" (UID: "0e0636e7-6f61-4843-844d-fb9d7bb527da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:13:35.150378 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.150348    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e0636e7-6f61-4843-844d-fb9d7bb527da-istio-envoy" pod "0e0636e7-6f61-4843-844d-fb9d7bb527da" (UID: "0e0636e7-6f61-4843-844d-fb9d7bb527da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:35.155819 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.154956    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e0636e7-6f61-4843-844d-fb9d7bb527da-istio-token" pod "0e0636e7-6f61-4843-844d-fb9d7bb527da" (UID: "0e0636e7-6f61-4843-844d-fb9d7bb527da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:13:35.157742 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.157711    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0e0636e7-6f61-4843-844d-fb9d7bb527da-istio-podinfo" pod "0e0636e7-6f61-4843-844d-fb9d7bb527da" (UID: "0e0636e7-6f61-4843-844d-fb9d7bb527da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:13:35.160961 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.160931    4525 generic.go:356] "Generic (PLEG): container finished" podID="0e0636e7-6f61-4843-844d-fb9d7bb527da" containerID="744818fd672a41f9671e8a439aa5cfef93a5f459b0ed208ee19d37e71d823797" exitCode=0
Oct 30 19:13:35.161419 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.161401    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-dhclt"
Oct 30 19:13:35.161794 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.161769    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-dhclt" event={"ID":"0e0636e7-6f61-4843-844d-fb9d7bb527da","Type":"ContainerDied","Data":"744818fd672a41f9671e8a439aa5cfef93a5f459b0ed208ee19d37e71d823797"}
Oct 30 19:13:35.161936 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.161918    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-dhclt" event={"ID":"0e0636e7-6f61-4843-844d-fb9d7bb527da","Type":"ContainerDied","Data":"2d010365bfca6d534b7527883645987a3349dc0870c581235140c207df2266dd"}
Oct 30 19:13:35.162025 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.161960    4525 scope.go:122] "RemoveContainer" containerID="744818fd672a41f9671e8a439aa5cfef93a5f459b0ed208ee19d37e71d823797"
Oct 30 19:13:35.164255 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.164234    4525 generic.go:356] "Generic (PLEG): container finished" podID="21d3419d-d29b-4e2b-8598-fe844028161e" containerID="a4836020c1f29976f62b485d0adb521ab0a7fddc503f4cbb0259bae8cdd1da30" exitCode=0
Oct 30 19:13:35.164464 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.164278    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-6pb9d" event={"ID":"21d3419d-d29b-4e2b-8598-fe844028161e","Type":"ContainerDied","Data":"a4836020c1f29976f62b485d0adb521ab0a7fddc503f4cbb0259bae8cdd1da30"}
Oct 30 19:13:35.209211 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.209165    4525 scope.go:122] "RemoveContainer" containerID="744818fd672a41f9671e8a439aa5cfef93a5f459b0ed208ee19d37e71d823797"
Oct 30 19:13:35.211038 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:13:35.209600    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"744818fd672a41f9671e8a439aa5cfef93a5f459b0ed208ee19d37e71d823797\": container with ID starting with 744818fd672a41f9671e8a439aa5cfef93a5f459b0ed208ee19d37e71d823797 not found: ID does not exist" containerID="744818fd672a41f9671e8a439aa5cfef93a5f459b0ed208ee19d37e71d823797"
Oct 30 19:13:35.211038 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.209634    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"744818fd672a41f9671e8a439aa5cfef93a5f459b0ed208ee19d37e71d823797"} err="failed to get container status \"744818fd672a41f9671e8a439aa5cfef93a5f459b0ed208ee19d37e71d823797\": rpc error: code = NotFound desc = could not find container \"744818fd672a41f9671e8a439aa5cfef93a5f459b0ed208ee19d37e71d823797\": container with ID starting with 744818fd672a41f9671e8a439aa5cfef93a5f459b0ed208ee19d37e71d823797 not found: ID does not exist"
Oct 30 19:13:35.213885 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.213858    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-dhclt"]
Oct 30 19:13:35.219345 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.219323    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-68cb9b4b95-dhclt"]
Oct 30 19:13:35.236618 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.236188    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-9v7v9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body=
Oct 30 19:13:35.236618 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.236247    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-9v7v9" podUID="a0856440-337e-4507-ad1a-bdce910e1989" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused"
Oct 30 19:13:35.246634 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.246605    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd0977b0-c75d-41ee-b911-4e71bd8c9b3a-istio-envoy\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw\" (UID: \"bd0977b0-c75d-41ee-b911-4e71bd8c9b3a\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw"
Oct 30 19:13:35.246785 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.246746    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd0977b0-c75d-41ee-b911-4e71bd8c9b3a-istiod-ca-cert\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw\" (UID: \"bd0977b0-c75d-41ee-b911-4e71bd8c9b3a\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw"
Oct 30 19:13:35.246851 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.246795    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd0977b0-c75d-41ee-b911-4e71bd8c9b3a-istio-data\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw\" (UID: \"bd0977b0-c75d-41ee-b911-4e71bd8c9b3a\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw"
Oct 30 19:13:35.246910 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.246894    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd0977b0-c75d-41ee-b911-4e71bd8c9b3a-istio-token\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw\" (UID: \"bd0977b0-c75d-41ee-b911-4e71bd8c9b3a\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw"
Oct 30 19:13:35.248731 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.247106    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd0977b0-c75d-41ee-b911-4e71bd8c9b3a-istio-podinfo\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw\" (UID: \"bd0977b0-c75d-41ee-b911-4e71bd8c9b3a\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw"
Oct 30 19:13:35.248731 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.247236    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd0977b0-c75d-41ee-b911-4e71bd8c9b3a-workload-socket\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw\" (UID: \"bd0977b0-c75d-41ee-b911-4e71bd8c9b3a\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw"
Oct 30 19:13:35.248731 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.247282    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd0977b0-c75d-41ee-b911-4e71bd8c9b3a-credential-socket\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw\" (UID: \"bd0977b0-c75d-41ee-b911-4e71bd8c9b3a\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw"
Oct 30 19:13:35.248731 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.247319    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd0977b0-c75d-41ee-b911-4e71bd8c9b3a-workload-certs\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw\" (UID: \"bd0977b0-c75d-41ee-b911-4e71bd8c9b3a\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw"
Oct 30 19:13:35.248731 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.247430    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8psn\" (UniqueName: \"kubernetes.io/projected/bd0977b0-c75d-41ee-b911-4e71bd8c9b3a-kube-api-access-h8psn\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw\" (UID: \"bd0977b0-c75d-41ee-b911-4e71bd8c9b3a\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw"
Oct 30 19:13:35.248731 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.247548    4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e0636e7-6f61-4843-844d-fb9d7bb527da-istio-data\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:35.248731 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.247568    4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e0636e7-6f61-4843-844d-fb9d7bb527da-istio-token\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:35.248731 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.247583    4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e0636e7-6f61-4843-844d-fb9d7bb527da-istio-envoy\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:35.248731 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.247598    4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e0636e7-6f61-4843-844d-fb9d7bb527da-istio-podinfo\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:35.248731 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.247611    4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e0636e7-6f61-4843-844d-fb9d7bb527da-istiod-ca-cert\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:35.248731 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.247627    4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e0636e7-6f61-4843-844d-fb9d7bb527da-workload-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:35.248731 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.247641    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldkb5\" (UniqueName: \"kubernetes.io/projected/0e0636e7-6f61-4843-844d-fb9d7bb527da-kube-api-access-ldkb5\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:35.248731 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.247728    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd0977b0-c75d-41ee-b911-4e71bd8c9b3a-istiod-ca-cert\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw\" (UID: \"bd0977b0-c75d-41ee-b911-4e71bd8c9b3a\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw"
Oct 30 19:13:35.248731 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.248160    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd0977b0-c75d-41ee-b911-4e71bd8c9b3a-workload-socket\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw\" (UID: \"bd0977b0-c75d-41ee-b911-4e71bd8c9b3a\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw"
Oct 30 19:13:35.248731 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.248389    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd0977b0-c75d-41ee-b911-4e71bd8c9b3a-istio-data\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw\" (UID: \"bd0977b0-c75d-41ee-b911-4e71bd8c9b3a\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw"
Oct 30 19:13:35.248731 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.248427    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd0977b0-c75d-41ee-b911-4e71bd8c9b3a-credential-socket\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw\" (UID: \"bd0977b0-c75d-41ee-b911-4e71bd8c9b3a\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw"
Oct 30 19:13:35.249650 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.248970    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd0977b0-c75d-41ee-b911-4e71bd8c9b3a-workload-certs\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw\" (UID: \"bd0977b0-c75d-41ee-b911-4e71bd8c9b3a\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw"
Oct 30 19:13:35.251132 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.251087    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd0977b0-c75d-41ee-b911-4e71bd8c9b3a-istio-envoy\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw\" (UID: \"bd0977b0-c75d-41ee-b911-4e71bd8c9b3a\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw"
Oct 30 19:13:35.254535 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.254509    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd0977b0-c75d-41ee-b911-4e71bd8c9b3a-istio-podinfo\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw\" (UID: \"bd0977b0-c75d-41ee-b911-4e71bd8c9b3a\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw"
Oct 30 19:13:35.259510 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.259456    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8psn\" (UniqueName: \"kubernetes.io/projected/bd0977b0-c75d-41ee-b911-4e71bd8c9b3a-kube-api-access-h8psn\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw\" (UID: \"bd0977b0-c75d-41ee-b911-4e71bd8c9b3a\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw"
Oct 30 19:13:35.260086 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.260041    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd0977b0-c75d-41ee-b911-4e71bd8c9b3a-istio-token\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw\" (UID: \"bd0977b0-c75d-41ee-b911-4e71bd8c9b3a\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw"
Oct 30 19:13:35.421749 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.412274    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw"
Oct 30 19:13:35.520003 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.519975    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-6pb9d"
Oct 30 19:13:35.559556 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.559527    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21d3419d-d29b-4e2b-8598-fe844028161e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21d3419d-d29b-4e2b-8598-fe844028161e-istiod-ca-cert\") pod \"21d3419d-d29b-4e2b-8598-fe844028161e\" (UID: \"21d3419d-d29b-4e2b-8598-fe844028161e\") "
Oct 30 19:13:35.560257 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.560236    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21d3419d-d29b-4e2b-8598-fe844028161e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21d3419d-d29b-4e2b-8598-fe844028161e-credential-socket\") pod \"21d3419d-d29b-4e2b-8598-fe844028161e\" (UID: \"21d3419d-d29b-4e2b-8598-fe844028161e\") "
Oct 30 19:13:35.563714 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.563143    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21d3419d-d29b-4e2b-8598-fe844028161e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21d3419d-d29b-4e2b-8598-fe844028161e-istio-data\") pod \"21d3419d-d29b-4e2b-8598-fe844028161e\" (UID: \"21d3419d-d29b-4e2b-8598-fe844028161e\") "
Oct 30 19:13:35.565467 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.565448    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21d3419d-d29b-4e2b-8598-fe844028161e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21d3419d-d29b-4e2b-8598-fe844028161e-workload-socket\") pod \"21d3419d-d29b-4e2b-8598-fe844028161e\" (UID: \"21d3419d-d29b-4e2b-8598-fe844028161e\") "
Oct 30 19:13:35.566106 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.566087    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/21d3419d-d29b-4e2b-8598-fe844028161e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21d3419d-d29b-4e2b-8598-fe844028161e-istio-podinfo\") pod \"21d3419d-d29b-4e2b-8598-fe844028161e\" (UID: \"21d3419d-d29b-4e2b-8598-fe844028161e\") "
Oct 30 19:13:35.566345 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.566328    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21d3419d-d29b-4e2b-8598-fe844028161e-kube-api-access-jvpks\" (UniqueName: \"kubernetes.io/projected/21d3419d-d29b-4e2b-8598-fe844028161e-kube-api-access-jvpks\") pod \"21d3419d-d29b-4e2b-8598-fe844028161e\" (UID: \"21d3419d-d29b-4e2b-8598-fe844028161e\") "
Oct 30 19:13:35.566471 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.566459    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21d3419d-d29b-4e2b-8598-fe844028161e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21d3419d-d29b-4e2b-8598-fe844028161e-workload-certs\") pod \"21d3419d-d29b-4e2b-8598-fe844028161e\" (UID: \"21d3419d-d29b-4e2b-8598-fe844028161e\") "
Oct 30 19:13:35.567093 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.566603    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21d3419d-d29b-4e2b-8598-fe844028161e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21d3419d-d29b-4e2b-8598-fe844028161e-istio-envoy\") pod \"21d3419d-d29b-4e2b-8598-fe844028161e\" (UID: \"21d3419d-d29b-4e2b-8598-fe844028161e\") "
Oct 30 19:13:35.567093 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.563607    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e0636e7-6f61-4843-844d-fb9d7bb527da" path="/var/lib/kubelet/pods/0e0636e7-6f61-4843-844d-fb9d7bb527da/volumes"
Oct 30 19:13:35.567374 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.560124    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21d3419d-d29b-4e2b-8598-fe844028161e-istiod-ca-cert" pod "21d3419d-d29b-4e2b-8598-fe844028161e" (UID: "21d3419d-d29b-4e2b-8598-fe844028161e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:13:35.567459 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.563040    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21d3419d-d29b-4e2b-8598-fe844028161e-credential-socket" pod "21d3419d-d29b-4e2b-8598-fe844028161e" (UID: "21d3419d-d29b-4e2b-8598-fe844028161e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:35.567547 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.563991    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21d3419d-d29b-4e2b-8598-fe844028161e-istio-data" pod "21d3419d-d29b-4e2b-8598-fe844028161e" (UID: "21d3419d-d29b-4e2b-8598-fe844028161e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:35.567612 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.565935    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21d3419d-d29b-4e2b-8598-fe844028161e-workload-socket" pod "21d3419d-d29b-4e2b-8598-fe844028161e" (UID: "21d3419d-d29b-4e2b-8598-fe844028161e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:35.568121 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.568102    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21d3419d-d29b-4e2b-8598-fe844028161e-istio-token\" (UniqueName: \"kubernetes.io/projected/21d3419d-d29b-4e2b-8598-fe844028161e-istio-token\") pod \"21d3419d-d29b-4e2b-8598-fe844028161e\" (UID: \"21d3419d-d29b-4e2b-8598-fe844028161e\") "
Oct 30 19:13:35.572839 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.570781    4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21d3419d-d29b-4e2b-8598-fe844028161e-workload-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:35.572839 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.570809    4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21d3419d-d29b-4e2b-8598-fe844028161e-istiod-ca-cert\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:35.572839 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.570823    4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21d3419d-d29b-4e2b-8598-fe844028161e-credential-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:35.572839 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.570836    4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21d3419d-d29b-4e2b-8598-fe844028161e-istio-data\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:35.572839 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.571712    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21d3419d-d29b-4e2b-8598-fe844028161e-workload-certs" pod "21d3419d-d29b-4e2b-8598-fe844028161e" (UID: "21d3419d-d29b-4e2b-8598-fe844028161e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:35.574905 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.574250    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-jv2tm"]
Oct 30 19:13:35.575498 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.575472    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21d3419d-d29b-4e2b-8598-fe844028161e-kube-api-access-jvpks" pod "21d3419d-d29b-4e2b-8598-fe844028161e" (UID: "21d3419d-d29b-4e2b-8598-fe844028161e"). InnerVolumeSpecName "kube-api-access-jvpks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:13:35.575810 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.575748    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21d3419d-d29b-4e2b-8598-fe844028161e-istio-envoy" pod "21d3419d-d29b-4e2b-8598-fe844028161e" (UID: "21d3419d-d29b-4e2b-8598-fe844028161e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:35.601707 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.598898    4525 cpu_manager.go:414] "containerMap: removing container" podUID="21d3419d-d29b-4e2b-8598-fe844028161e" containerName="istio-proxy"
Oct 30 19:13:35.601707 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.598931    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21d3419d-d29b-4e2b-8598-fe844028161e" containerName="istio-proxy"
Oct 30 19:13:35.601707 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.599288    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="21d3419d-d29b-4e2b-8598-fe844028161e" containerName="istio-proxy"
Oct 30 19:13:35.613255 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.612492    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-jv2tm"
Oct 30 19:13:35.620124 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.620078    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/21d3419d-d29b-4e2b-8598-fe844028161e-istio-podinfo" pod "21d3419d-d29b-4e2b-8598-fe844028161e" (UID: "21d3419d-d29b-4e2b-8598-fe844028161e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:13:35.620278 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.620172    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-jv2tm"]
Oct 30 19:13:35.622120 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.622080    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21d3419d-d29b-4e2b-8598-fe844028161e-istio-token" pod "21d3419d-d29b-4e2b-8598-fe844028161e" (UID: "21d3419d-d29b-4e2b-8598-fe844028161e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:13:35.711483 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.711428    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4071206-077a-4d0f-8f0d-a2b76604d376-workload-socket\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-jv2tm\" (UID: \"c4071206-077a-4d0f-8f0d-a2b76604d376\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-jv2tm"
Oct 30 19:13:35.711759 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.711555    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4071206-077a-4d0f-8f0d-a2b76604d376-credential-socket\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-jv2tm\" (UID: \"c4071206-077a-4d0f-8f0d-a2b76604d376\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-jv2tm"
Oct 30 19:13:35.711759 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.711617    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4071206-077a-4d0f-8f0d-a2b76604d376-workload-certs\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-jv2tm\" (UID: \"c4071206-077a-4d0f-8f0d-a2b76604d376\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-jv2tm"
Oct 30 19:13:35.711759 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.711658    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4071206-077a-4d0f-8f0d-a2b76604d376-istiod-ca-cert\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-jv2tm\" (UID: \"c4071206-077a-4d0f-8f0d-a2b76604d376\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-jv2tm"
Oct 30 19:13:35.711983 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.711937    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4071206-077a-4d0f-8f0d-a2b76604d376-istio-envoy\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-jv2tm\" (UID: \"c4071206-077a-4d0f-8f0d-a2b76604d376\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-jv2tm"
Oct 30 19:13:35.712089 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.712065    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4071206-077a-4d0f-8f0d-a2b76604d376-istio-podinfo\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-jv2tm\" (UID: \"c4071206-077a-4d0f-8f0d-a2b76604d376\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-jv2tm"
Oct 30 19:13:35.712178 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.712157    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4071206-077a-4d0f-8f0d-a2b76604d376-istio-token\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-jv2tm\" (UID: \"c4071206-077a-4d0f-8f0d-a2b76604d376\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-jv2tm"
Oct 30 19:13:35.712307 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.712286    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4071206-077a-4d0f-8f0d-a2b76604d376-istio-data\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-jv2tm\" (UID: \"c4071206-077a-4d0f-8f0d-a2b76604d376\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-jv2tm"
Oct 30 19:13:35.712411 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.712393    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqjg2\" (UniqueName: \"kubernetes.io/projected/c4071206-077a-4d0f-8f0d-a2b76604d376-kube-api-access-mqjg2\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-jv2tm\" (UID: \"c4071206-077a-4d0f-8f0d-a2b76604d376\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-jv2tm"
Oct 30 19:13:35.713176 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.712529    4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21d3419d-d29b-4e2b-8598-fe844028161e-istio-podinfo\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:35.713176 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.712552    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvpks\" (UniqueName: \"kubernetes.io/projected/21d3419d-d29b-4e2b-8598-fe844028161e-kube-api-access-jvpks\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:35.713176 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.712571    4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21d3419d-d29b-4e2b-8598-fe844028161e-workload-certs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:35.713176 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.712586    4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21d3419d-d29b-4e2b-8598-fe844028161e-istio-envoy\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:35.713176 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.712600    4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21d3419d-d29b-4e2b-8598-fe844028161e-istio-token\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:35.814041 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.814004    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqjg2\" (UniqueName: \"kubernetes.io/projected/c4071206-077a-4d0f-8f0d-a2b76604d376-kube-api-access-mqjg2\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-jv2tm\" (UID: \"c4071206-077a-4d0f-8f0d-a2b76604d376\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-jv2tm"
Oct 30 19:13:35.814901 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.814063    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4071206-077a-4d0f-8f0d-a2b76604d376-workload-socket\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-jv2tm\" (UID: \"c4071206-077a-4d0f-8f0d-a2b76604d376\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-jv2tm"
Oct 30 19:13:35.814901 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.814116    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4071206-077a-4d0f-8f0d-a2b76604d376-credential-socket\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-jv2tm\" (UID: \"c4071206-077a-4d0f-8f0d-a2b76604d376\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-jv2tm"
Oct 30 19:13:35.814901 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.814242    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4071206-077a-4d0f-8f0d-a2b76604d376-workload-certs\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-jv2tm\" (UID: \"c4071206-077a-4d0f-8f0d-a2b76604d376\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-jv2tm"
Oct 30 19:13:35.814901 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.814288    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4071206-077a-4d0f-8f0d-a2b76604d376-istiod-ca-cert\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-jv2tm\" (UID: \"c4071206-077a-4d0f-8f0d-a2b76604d376\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-jv2tm"
Oct 30 19:13:35.814901 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.814411    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4071206-077a-4d0f-8f0d-a2b76604d376-istio-envoy\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-jv2tm\" (UID: \"c4071206-077a-4d0f-8f0d-a2b76604d376\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-jv2tm"
Oct 30 19:13:35.814901 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.814536    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4071206-077a-4d0f-8f0d-a2b76604d376-istio-podinfo\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-jv2tm\" (UID: \"c4071206-077a-4d0f-8f0d-a2b76604d376\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-jv2tm"
Oct 30 19:13:35.814901 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.814582    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4071206-077a-4d0f-8f0d-a2b76604d376-workload-socket\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-jv2tm\" (UID: \"c4071206-077a-4d0f-8f0d-a2b76604d376\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-jv2tm"
Oct 30 19:13:35.814901 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.814612    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4071206-077a-4d0f-8f0d-a2b76604d376-credential-socket\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-jv2tm\" (UID: \"c4071206-077a-4d0f-8f0d-a2b76604d376\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-jv2tm"
Oct 30 19:13:35.814901 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.814618    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4071206-077a-4d0f-8f0d-a2b76604d376-istio-token\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-jv2tm\" (UID: \"c4071206-077a-4d0f-8f0d-a2b76604d376\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-jv2tm"
Oct 30 19:13:35.814901 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.814665    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4071206-077a-4d0f-8f0d-a2b76604d376-workload-certs\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-jv2tm\" (UID: \"c4071206-077a-4d0f-8f0d-a2b76604d376\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-jv2tm"
Oct 30 19:13:35.814901 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.814885    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4071206-077a-4d0f-8f0d-a2b76604d376-istio-data\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-jv2tm\" (UID: \"c4071206-077a-4d0f-8f0d-a2b76604d376\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-jv2tm"
Oct 30 19:13:35.815868 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.815064    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4071206-077a-4d0f-8f0d-a2b76604d376-istiod-ca-cert\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-jv2tm\" (UID: \"c4071206-077a-4d0f-8f0d-a2b76604d376\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-jv2tm"
Oct 30 19:13:35.815868 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.815145    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4071206-077a-4d0f-8f0d-a2b76604d376-istio-data\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-jv2tm\" (UID: \"c4071206-077a-4d0f-8f0d-a2b76604d376\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-jv2tm"
Oct 30 19:13:35.816951 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.816915    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4071206-077a-4d0f-8f0d-a2b76604d376-istio-envoy\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-jv2tm\" (UID: \"c4071206-077a-4d0f-8f0d-a2b76604d376\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-jv2tm"
Oct 30 19:13:35.817318 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.817282    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4071206-077a-4d0f-8f0d-a2b76604d376-istio-podinfo\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-jv2tm\" (UID: \"c4071206-077a-4d0f-8f0d-a2b76604d376\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-jv2tm"
Oct 30 19:13:35.822189 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.822167    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4071206-077a-4d0f-8f0d-a2b76604d376-istio-token\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-jv2tm\" (UID: \"c4071206-077a-4d0f-8f0d-a2b76604d376\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-jv2tm"
Oct 30 19:13:35.822455 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.822432    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqjg2\" (UniqueName: \"kubernetes.io/projected/c4071206-077a-4d0f-8f0d-a2b76604d376-kube-api-access-mqjg2\") pod \"e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-jv2tm\" (UID: \"c4071206-077a-4d0f-8f0d-a2b76604d376\") " pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-jv2tm"
Oct 30 19:13:35.852786 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:13:35.852752    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd0977b0_c75d_41ee_b911_4e71bd8c9b3a.slice/crio-1b1e851e5e6a96a533427100bf6f1d08619bdca2b328a0240371857132596b24 WatchSource:0}: Error finding container 1b1e851e5e6a96a533427100bf6f1d08619bdca2b328a0240371857132596b24: Status 404 returned error can't find the container with id 1b1e851e5e6a96a533427100bf6f1d08619bdca2b328a0240371857132596b24
Oct 30 19:13:35.855006 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.854978    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw"]
Oct 30 19:13:35.856458 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.856427    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:13:35.856523 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.856493    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:13:35.856564 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.856524    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:13:35.946726 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:35.944655    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-jv2tm"
Oct 30 19:13:36.236385 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:36.235881    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-9v7v9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body=
Oct 30 19:13:36.236385 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:36.235954    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-9v7v9" podUID="a0856440-337e-4507-ad1a-bdce910e1989" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused"
Oct 30 19:13:36.250960 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:36.249545    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw" event={"ID":"bd0977b0-c75d-41ee-b911-4e71bd8c9b3a","Type":"ContainerStarted","Data":"279e860cf6912f80fd5d65c29c25ee1b35f5b7218d7bafd583abb950d7bcf2f7"}
Oct 30 19:13:36.250960 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:36.249610    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw" event={"ID":"bd0977b0-c75d-41ee-b911-4e71bd8c9b3a","Type":"ContainerStarted","Data":"1b1e851e5e6a96a533427100bf6f1d08619bdca2b328a0240371857132596b24"}
Oct 30 19:13:36.262043 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:36.260358    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-6pb9d" event={"ID":"21d3419d-d29b-4e2b-8598-fe844028161e","Type":"ContainerDied","Data":"c1de833ef5efcb0650d72119599f9ca9c4363e2c900cf1db9932b1f1941105b2"}
Oct 30 19:13:36.262043 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:36.260427    4525 scope.go:122] "RemoveContainer" containerID="a4836020c1f29976f62b485d0adb521ab0a7fddc503f4cbb0259bae8cdd1da30"
Oct 30 19:13:36.262043 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:36.260539    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-6pb9d"
Oct 30 19:13:36.300631 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:36.295502    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw" podStartSLOduration=107.295480728 podStartE2EDuration="1m47.295480728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:11:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:13:36.272867219 +0000 UTC m=+2238.898537184" watchObservedRunningTime="2026-10-30 19:13:36.295480728 +0000 UTC m=+2238.921150693"
Oct 30 19:13:36.351044 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:36.350999    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-f3d6a5/payload-processing-external-model-e2e-extmod-b-f3d6a5-f7d7npjtw"]
Oct 30 19:13:36.351202 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:36.351079    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-6pb9d"]
Oct 30 19:13:36.370441 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:36.370399    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-6pb9d"]
Oct 30 19:13:36.427795 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:36.426782    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw"
Oct 30 19:13:36.436422 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:36.436058    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused" start-of-body=
Oct 30 19:13:36.436422 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:36.436123    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw" podUID="bd0977b0-c75d-41ee-b911-4e71bd8c9b3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused"
Oct 30 19:13:36.454974 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:36.454946    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-jv2tm"]
Oct 30 19:13:36.457761 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:36.457358    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:13:36.457761 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:36.457429    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:13:36.457761 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:36.457473    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:13:37.262173 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:37.242740    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-9v7v9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body=
Oct 30 19:13:37.265720 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:37.265660    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-9v7v9" podUID="a0856440-337e-4507-ad1a-bdce910e1989" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused"
Oct 30 19:13:37.360004 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:37.359528    4525 generic.go:356] "Generic (PLEG): container finished" podID="5d9a2656-6bae-4277-a7f3-b43e8143dbd7" containerID="7396b8db18338af05d47281d1cfb35fe280b6afbe7bd7a122a987ac114fa04f7" exitCode=0
Oct 30 19:13:37.360004 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:37.359910    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-229cp" event={"ID":"5d9a2656-6bae-4277-a7f3-b43e8143dbd7","Type":"ContainerDied","Data":"7396b8db18338af05d47281d1cfb35fe280b6afbe7bd7a122a987ac114fa04f7"}
Oct 30 19:13:37.363316 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:37.363290    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-jv2tm" event={"ID":"c4071206-077a-4d0f-8f0d-a2b76604d376","Type":"ContainerStarted","Data":"a8bd98c7596fb9d80d0c2a8f1903ef9f06b92586f171513e4b3aaa88e8922a8c"}
Oct 30 19:13:37.414183 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:37.413891    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused" start-of-body=
Oct 30 19:13:37.414183 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:37.413943    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw" podUID="bd0977b0-c75d-41ee-b911-4e71bd8c9b3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused"
Oct 30 19:13:37.438028 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:37.438003    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-229cp"
Oct 30 19:13:37.568644 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:37.568563    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21d3419d-d29b-4e2b-8598-fe844028161e" path="/var/lib/kubelet/pods/21d3419d-d29b-4e2b-8598-fe844028161e/volumes"
Oct 30 19:13:37.611287 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:37.611248    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d9a2656-6bae-4277-a7f3-b43e8143dbd7-kube-api-access-2bf66\" (UniqueName: \"kubernetes.io/projected/5d9a2656-6bae-4277-a7f3-b43e8143dbd7-kube-api-access-2bf66\") pod \"5d9a2656-6bae-4277-a7f3-b43e8143dbd7\" (UID: \"5d9a2656-6bae-4277-a7f3-b43e8143dbd7\") "
Oct 30 19:13:37.611486 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:37.611305    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d9a2656-6bae-4277-a7f3-b43e8143dbd7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d9a2656-6bae-4277-a7f3-b43e8143dbd7-istio-data\") pod \"5d9a2656-6bae-4277-a7f3-b43e8143dbd7\" (UID: \"5d9a2656-6bae-4277-a7f3-b43e8143dbd7\") "
Oct 30 19:13:37.611486 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:37.611376    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d9a2656-6bae-4277-a7f3-b43e8143dbd7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d9a2656-6bae-4277-a7f3-b43e8143dbd7-workload-certs\") pod \"5d9a2656-6bae-4277-a7f3-b43e8143dbd7\" (UID: \"5d9a2656-6bae-4277-a7f3-b43e8143dbd7\") "
Oct 30 19:13:37.611486 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:37.611458    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5d9a2656-6bae-4277-a7f3-b43e8143dbd7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d9a2656-6bae-4277-a7f3-b43e8143dbd7-istio-podinfo\") pod \"5d9a2656-6bae-4277-a7f3-b43e8143dbd7\" (UID: \"5d9a2656-6bae-4277-a7f3-b43e8143dbd7\") "
Oct 30 19:13:37.611651 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:37.611500    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d9a2656-6bae-4277-a7f3-b43e8143dbd7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d9a2656-6bae-4277-a7f3-b43e8143dbd7-istiod-ca-cert\") pod \"5d9a2656-6bae-4277-a7f3-b43e8143dbd7\" (UID: \"5d9a2656-6bae-4277-a7f3-b43e8143dbd7\") "
Oct 30 19:13:37.611651 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:37.611565    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d9a2656-6bae-4277-a7f3-b43e8143dbd7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d9a2656-6bae-4277-a7f3-b43e8143dbd7-istio-envoy\") pod \"5d9a2656-6bae-4277-a7f3-b43e8143dbd7\" (UID: \"5d9a2656-6bae-4277-a7f3-b43e8143dbd7\") "
Oct 30 19:13:37.611651 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:37.611636    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d9a2656-6bae-4277-a7f3-b43e8143dbd7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d9a2656-6bae-4277-a7f3-b43e8143dbd7-workload-socket\") pod \"5d9a2656-6bae-4277-a7f3-b43e8143dbd7\" (UID: \"5d9a2656-6bae-4277-a7f3-b43e8143dbd7\") "
Oct 30 19:13:37.611854 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:37.611805    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d9a2656-6bae-4277-a7f3-b43e8143dbd7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d9a2656-6bae-4277-a7f3-b43e8143dbd7-credential-socket\") pod \"5d9a2656-6bae-4277-a7f3-b43e8143dbd7\" (UID: \"5d9a2656-6bae-4277-a7f3-b43e8143dbd7\") "
Oct 30 19:13:37.612005 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:37.611984    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d9a2656-6bae-4277-a7f3-b43e8143dbd7-istio-token\" (UniqueName: \"kubernetes.io/projected/5d9a2656-6bae-4277-a7f3-b43e8143dbd7-istio-token\") pod \"5d9a2656-6bae-4277-a7f3-b43e8143dbd7\" (UID: \"5d9a2656-6bae-4277-a7f3-b43e8143dbd7\") "
Oct 30 19:13:37.612072 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:37.612030    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d9a2656-6bae-4277-a7f3-b43e8143dbd7-workload-certs" pod "5d9a2656-6bae-4277-a7f3-b43e8143dbd7" (UID: "5d9a2656-6bae-4277-a7f3-b43e8143dbd7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:37.612860 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:37.612836    4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d9a2656-6bae-4277-a7f3-b43e8143dbd7-workload-certs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:37.613037 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:37.612988    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d9a2656-6bae-4277-a7f3-b43e8143dbd7-istio-data" pod "5d9a2656-6bae-4277-a7f3-b43e8143dbd7" (UID: "5d9a2656-6bae-4277-a7f3-b43e8143dbd7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:37.613442 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:37.613412    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d9a2656-6bae-4277-a7f3-b43e8143dbd7-credential-socket" pod "5d9a2656-6bae-4277-a7f3-b43e8143dbd7" (UID: "5d9a2656-6bae-4277-a7f3-b43e8143dbd7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:37.613556 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:37.613454    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d9a2656-6bae-4277-a7f3-b43e8143dbd7-workload-socket" pod "5d9a2656-6bae-4277-a7f3-b43e8143dbd7" (UID: "5d9a2656-6bae-4277-a7f3-b43e8143dbd7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:37.613812 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:37.613787    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d9a2656-6bae-4277-a7f3-b43e8143dbd7-istiod-ca-cert" pod "5d9a2656-6bae-4277-a7f3-b43e8143dbd7" (UID: "5d9a2656-6bae-4277-a7f3-b43e8143dbd7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:13:37.615660 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:37.615631    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d9a2656-6bae-4277-a7f3-b43e8143dbd7-istio-envoy" pod "5d9a2656-6bae-4277-a7f3-b43e8143dbd7" (UID: "5d9a2656-6bae-4277-a7f3-b43e8143dbd7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:37.615780 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:37.615663    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d9a2656-6bae-4277-a7f3-b43e8143dbd7-istio-token" pod "5d9a2656-6bae-4277-a7f3-b43e8143dbd7" (UID: "5d9a2656-6bae-4277-a7f3-b43e8143dbd7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:13:37.615780 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:37.615743    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d9a2656-6bae-4277-a7f3-b43e8143dbd7-kube-api-access-2bf66" pod "5d9a2656-6bae-4277-a7f3-b43e8143dbd7" (UID: "5d9a2656-6bae-4277-a7f3-b43e8143dbd7"). InnerVolumeSpecName "kube-api-access-2bf66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:13:37.616271 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:37.616253    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d9a2656-6bae-4277-a7f3-b43e8143dbd7-istio-podinfo" pod "5d9a2656-6bae-4277-a7f3-b43e8143dbd7" (UID: "5d9a2656-6bae-4277-a7f3-b43e8143dbd7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:13:37.761702 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:37.758891    4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d9a2656-6bae-4277-a7f3-b43e8143dbd7-workload-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:37.761702 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:37.758927    4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d9a2656-6bae-4277-a7f3-b43e8143dbd7-credential-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:37.761702 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:37.758942    4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d9a2656-6bae-4277-a7f3-b43e8143dbd7-istio-token\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:37.761702 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:37.758957    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bf66\" (UniqueName: \"kubernetes.io/projected/5d9a2656-6bae-4277-a7f3-b43e8143dbd7-kube-api-access-2bf66\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:37.761702 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:37.758972    4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d9a2656-6bae-4277-a7f3-b43e8143dbd7-istio-data\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:37.761702 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:37.758985    4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d9a2656-6bae-4277-a7f3-b43e8143dbd7-istio-podinfo\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:37.761702 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:37.758998    4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d9a2656-6bae-4277-a7f3-b43e8143dbd7-istiod-ca-cert\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:37.761702 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:37.759014    4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d9a2656-6bae-4277-a7f3-b43e8143dbd7-istio-envoy\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:38.065475 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:38.065434    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-37f2b8-qznrj"]
Oct 30 19:13:38.066525 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:38.066505    4525 cpu_manager.go:414] "containerMap: removing container" podUID="5d9a2656-6bae-4277-a7f3-b43e8143dbd7" containerName="istio-proxy"
Oct 30 19:13:38.066575 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:38.066528    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d9a2656-6bae-4277-a7f3-b43e8143dbd7" containerName="istio-proxy"
Oct 30 19:13:38.066710 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:38.066651    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d9a2656-6bae-4277-a7f3-b43e8143dbd7" containerName="istio-proxy"
Oct 30 19:13:38.074597 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:38.074571    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-37f2b8-qznrj"]
Oct 30 19:13:38.074766 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:38.074698    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-37f2b8-qznrj"
Oct 30 19:13:38.096705 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:38.087107    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94hcn\""
Oct 30 19:13:38.096705 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:38.087182    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:13:38.168905 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:38.168872    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/139e7b9b-49a4-4dff-b2cf-434f9a6cb3bf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-37f2b8-qznrj\" (UID: \"139e7b9b-49a4-4dff-b2cf-434f9a6cb3bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-37f2b8-qznrj"
Oct 30 19:13:38.169075 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:38.168992    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/139e7b9b-49a4-4dff-b2cf-434f9a6cb3bf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-37f2b8-qznrj\" (UID: \"139e7b9b-49a4-4dff-b2cf-434f9a6cb3bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-37f2b8-qznrj"
Oct 30 19:13:38.169075 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:38.169028    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mc9zr\" (UniqueName: \"kubernetes.io/projected/139e7b9b-49a4-4dff-b2cf-434f9a6cb3bf-kube-api-access-mc9zr\") pod \"maas-api-revoke-keys-e2e-extmod-a-37f2b8-qznrj\" (UID: \"139e7b9b-49a4-4dff-b2cf-434f9a6cb3bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-37f2b8-qznrj"
Oct 30 19:13:38.236777 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:38.236735    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-9v7v9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body=
Oct 30 19:13:38.236958 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:38.236802    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-9v7v9" podUID="a0856440-337e-4507-ad1a-bdce910e1989" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused"
Oct 30 19:13:38.271256 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:38.271223    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/139e7b9b-49a4-4dff-b2cf-434f9a6cb3bf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-37f2b8-qznrj\" (UID: \"139e7b9b-49a4-4dff-b2cf-434f9a6cb3bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-37f2b8-qznrj"
Oct 30 19:13:38.271764 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:38.271266    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc9zr\" (UniqueName: \"kubernetes.io/projected/139e7b9b-49a4-4dff-b2cf-434f9a6cb3bf-kube-api-access-mc9zr\") pod \"maas-api-revoke-keys-e2e-extmod-a-37f2b8-qznrj\" (UID: \"139e7b9b-49a4-4dff-b2cf-434f9a6cb3bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-37f2b8-qznrj"
Oct 30 19:13:38.271764 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:38.271600    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/139e7b9b-49a4-4dff-b2cf-434f9a6cb3bf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-37f2b8-qznrj\" (UID: \"139e7b9b-49a4-4dff-b2cf-434f9a6cb3bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-37f2b8-qznrj"
Oct 30 19:13:38.272284 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:38.272264    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/139e7b9b-49a4-4dff-b2cf-434f9a6cb3bf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-37f2b8-qznrj\" (UID: \"139e7b9b-49a4-4dff-b2cf-434f9a6cb3bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-37f2b8-qznrj"
Oct 30 19:13:38.273807 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:38.273789    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/139e7b9b-49a4-4dff-b2cf-434f9a6cb3bf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-37f2b8-qznrj\" (UID: \"139e7b9b-49a4-4dff-b2cf-434f9a6cb3bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-37f2b8-qznrj"
Oct 30 19:13:38.306907 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:38.306873    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mc9zr\" (UniqueName: \"kubernetes.io/projected/139e7b9b-49a4-4dff-b2cf-434f9a6cb3bf-kube-api-access-mc9zr\") pod \"maas-api-revoke-keys-e2e-extmod-a-37f2b8-qznrj\" (UID: \"139e7b9b-49a4-4dff-b2cf-434f9a6cb3bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-37f2b8-qznrj"
Oct 30 19:13:38.374647 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:38.374587    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-jv2tm" event={"ID":"c4071206-077a-4d0f-8f0d-a2b76604d376","Type":"ContainerStarted","Data":"c0a01bdb2075f569db79a3abd753564e2b87ed479f0ce90b585890680b4404cb"}
Oct 30 19:13:38.397515 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:38.395777    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-229cp" event={"ID":"5d9a2656-6bae-4277-a7f3-b43e8143dbd7","Type":"ContainerDied","Data":"be3c9abead7112b6b27a4ebe0645812ce49b73d1ac27f6fdbed82fe2754470ba"}
Oct 30 19:13:38.397515 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:38.395870    4525 scope.go:122] "RemoveContainer" containerID="7396b8db18338af05d47281d1cfb35fe280b6afbe7bd7a122a987ac114fa04f7"
Oct 30 19:13:38.398007 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:38.397984    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-229cp"
Oct 30 19:13:38.422711 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:38.417048    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused" start-of-body=
Oct 30 19:13:38.422711 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:38.417109    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw" podUID="bd0977b0-c75d-41ee-b911-4e71bd8c9b3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused"
Oct 30 19:13:38.441337 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:38.441261    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-jv2tm" podStartSLOduration=109.441239948 podStartE2EDuration="1m49.441239948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:11:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:13:38.438779123 +0000 UTC m=+2241.064449088" watchObservedRunningTime="2026-10-30 19:13:38.441239948 +0000 UTC m=+2241.066909914"
Oct 30 19:13:38.461244 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:38.461209    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-37f2b8-qznrj"
Oct 30 19:13:38.570554 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:38.550925    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-229cp"]
Oct 30 19:13:38.573412 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:38.573384    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-229cp"]
Oct 30 19:13:38.999126 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:38.948994    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-jv2tm"
Oct 30 19:13:38.999126 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:38.981831    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-jv2tm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused" start-of-body=
Oct 30 19:13:38.999126 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:38.981895    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-jv2tm" podUID="c4071206-077a-4d0f-8f0d-a2b76604d376" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused"
Oct 30 19:13:39.250336 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:39.250223    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-9v7v9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body=
Oct 30 19:13:39.250336 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:39.250292    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-9v7v9" podUID="a0856440-337e-4507-ad1a-bdce910e1989" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused"
Oct 30 19:13:39.352186 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:39.352154    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-37f2b8-qznrj"]
Oct 30 19:13:39.358980 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:13:39.358952    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod139e7b9b_49a4_4dff_b2cf_434f9a6cb3bf.slice/crio-a47dbee543b9cb74d70fb0fb3a36276b7147d9c933f8a3ba556555c214a5fc00 WatchSource:0}: Error finding container a47dbee543b9cb74d70fb0fb3a36276b7147d9c933f8a3ba556555c214a5fc00: Status 404 returned error can't find the container with id a47dbee543b9cb74d70fb0fb3a36276b7147d9c933f8a3ba556555c214a5fc00
Oct 30 19:13:39.434538 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:39.434495    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused" start-of-body=
Oct 30 19:13:39.434880 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:39.434839    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw" podUID="bd0977b0-c75d-41ee-b911-4e71bd8c9b3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused"
Oct 30 19:13:39.447404 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:39.447355    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-37f2b8-qznrj" event={"ID":"139e7b9b-49a4-4dff-b2cf-434f9a6cb3bf","Type":"ContainerStarted","Data":"a47dbee543b9cb74d70fb0fb3a36276b7147d9c933f8a3ba556555c214a5fc00"}
Oct 30 19:13:39.558166 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:39.558089    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d9a2656-6bae-4277-a7f3-b43e8143dbd7" path="/var/lib/kubelet/pods/5d9a2656-6bae-4277-a7f3-b43e8143dbd7/volumes"
Oct 30 19:13:39.946549 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:39.946508    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-jv2tm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused" start-of-body=
Oct 30 19:13:39.946754 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:39.946567    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-jv2tm" podUID="c4071206-077a-4d0f-8f0d-a2b76604d376" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused"
Oct 30 19:13:40.171371 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:40.171346    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ecc266-559cbb854f-8wvqg"
Oct 30 19:13:40.245348 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:40.244368    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-9v7v9"
Oct 30 19:13:40.245912 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:40.245888    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-9v7v9"
Oct 30 19:13:40.246436 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:40.246415    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-9v7v9"
Oct 30 19:13:40.443766 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:40.437752    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused" start-of-body=
Oct 30 19:13:40.443766 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:40.437816    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw" podUID="bd0977b0-c75d-41ee-b911-4e71bd8c9b3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused"
Oct 30 19:13:40.534780 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:40.534661    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-37f2b8-qznrj" event={"ID":"139e7b9b-49a4-4dff-b2cf-434f9a6cb3bf","Type":"ContainerStarted","Data":"2a48cb8e22cc1fa596d13d0ce61b9a684cbdfa87d5df0b9aba2ebd99141814e9"}
Oct 30 19:13:40.555518 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:40.555471    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-37f2b8-qznrj" podStartSLOduration=2.555457649 podStartE2EDuration="2.555457649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:13:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:13:40.551100912 +0000 UTC m=+2243.176770878" watchObservedRunningTime="2026-10-30 19:13:40.555457649 +0000 UTC m=+2243.181127613"
Oct 30 19:13:40.946693 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:40.946644    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-jv2tm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused" start-of-body=
Oct 30 19:13:40.946875 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:40.946732    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-jv2tm" podUID="c4071206-077a-4d0f-8f0d-a2b76604d376" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused"
Oct 30 19:13:41.353019 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:41.351732    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-f3d6a5/payload-processing-external-model-e2e-extmod-b-f3d6a5-f7d7npjtw" podUID="c52512ff-3354-4aba-9d39-99a3084e02af" containerName="payload-processing" containerID="cri-o://d15f22c40ccf75d8c2265fe3d023ccc0401bf6843bf5a0a4a3d5b37d0ebc0dff" gracePeriod=25
Oct 30 19:13:41.418705 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:41.416124    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused" start-of-body=
Oct 30 19:13:41.418705 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:41.416189    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw" podUID="bd0977b0-c75d-41ee-b911-4e71bd8c9b3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused"
Oct 30 19:13:41.574544 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:41.570694    4525 generic.go:356] "Generic (PLEG): container finished" podID="c52512ff-3354-4aba-9d39-99a3084e02af" containerID="d15f22c40ccf75d8c2265fe3d023ccc0401bf6843bf5a0a4a3d5b37d0ebc0dff" exitCode=0
Oct 30 19:13:41.574544 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:41.570925    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f3d6a5/payload-processing-external-model-e2e-extmod-b-f3d6a5-f7d7npjtw" event={"ID":"c52512ff-3354-4aba-9d39-99a3084e02af","Type":"ContainerDied","Data":"d15f22c40ccf75d8c2265fe3d023ccc0401bf6843bf5a0a4a3d5b37d0ebc0dff"}
Oct 30 19:13:41.574544 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:41.573331    4525 generic.go:356] "Generic (PLEG): container finished" podID="139e7b9b-49a4-4dff-b2cf-434f9a6cb3bf" containerID="2a48cb8e22cc1fa596d13d0ce61b9a684cbdfa87d5df0b9aba2ebd99141814e9" exitCode=0
Oct 30 19:13:41.574544 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:41.573408    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-37f2b8-qznrj" event={"ID":"139e7b9b-49a4-4dff-b2cf-434f9a6cb3bf","Type":"ContainerDied","Data":"2a48cb8e22cc1fa596d13d0ce61b9a684cbdfa87d5df0b9aba2ebd99141814e9"}
Oct 30 19:13:41.951366 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:41.951327    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-jv2tm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused" start-of-body=
Oct 30 19:13:41.952281 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:41.951589    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-jv2tm" podUID="c4071206-077a-4d0f-8f0d-a2b76604d376" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused"
Oct 30 19:13:42.072001 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:42.071978    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f3d6a5/payload-processing-external-model-e2e-extmod-b-f3d6a5-f7d7npjtw"
Oct 30 19:13:42.258721 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:42.257425    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c52512ff-3354-4aba-9d39-99a3084e02af-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c52512ff-3354-4aba-9d39-99a3084e02af-routing-overlay-volume\") pod \"c52512ff-3354-4aba-9d39-99a3084e02af\" (UID: \"c52512ff-3354-4aba-9d39-99a3084e02af\") "
Oct 30 19:13:42.258721 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:42.257726    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c52512ff-3354-4aba-9d39-99a3084e02af-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c52512ff-3354-4aba-9d39-99a3084e02af-provider-credentials\") pod \"c52512ff-3354-4aba-9d39-99a3084e02af\" (UID: \"c52512ff-3354-4aba-9d39-99a3084e02af\") "
Oct 30 19:13:42.258721 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:42.257905    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c52512ff-3354-4aba-9d39-99a3084e02af-kube-api-access-lpbd2\" (UniqueName: \"kubernetes.io/projected/c52512ff-3354-4aba-9d39-99a3084e02af-kube-api-access-lpbd2\") pod \"c52512ff-3354-4aba-9d39-99a3084e02af\" (UID: \"c52512ff-3354-4aba-9d39-99a3084e02af\") "
Oct 30 19:13:42.258721 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:42.257950    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c52512ff-3354-4aba-9d39-99a3084e02af-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c52512ff-3354-4aba-9d39-99a3084e02af-plugins-config-volume\") pod \"c52512ff-3354-4aba-9d39-99a3084e02af\" (UID: \"c52512ff-3354-4aba-9d39-99a3084e02af\") "
Oct 30 19:13:42.326056 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:42.325958    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c52512ff-3354-4aba-9d39-99a3084e02af-plugins-config-volume" pod "c52512ff-3354-4aba-9d39-99a3084e02af" (UID: "c52512ff-3354-4aba-9d39-99a3084e02af"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:13:42.356597 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:42.356555    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c52512ff-3354-4aba-9d39-99a3084e02af-routing-overlay-volume" pod "c52512ff-3354-4aba-9d39-99a3084e02af" (UID: "c52512ff-3354-4aba-9d39-99a3084e02af"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:13:42.395384 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:42.395345    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c52512ff-3354-4aba-9d39-99a3084e02af-kube-api-access-lpbd2" pod "c52512ff-3354-4aba-9d39-99a3084e02af" (UID: "c52512ff-3354-4aba-9d39-99a3084e02af"). InnerVolumeSpecName "kube-api-access-lpbd2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:13:42.416056 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:42.416020    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused" start-of-body=
Oct 30 19:13:42.424845 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:42.416246    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw" podUID="bd0977b0-c75d-41ee-b911-4e71bd8c9b3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused"
Oct 30 19:13:42.456810 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:42.426717    4525 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c52512ff-3354-4aba-9d39-99a3084e02af-routing-overlay-volume\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:42.466707 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:42.457716    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpbd2\" (UniqueName: \"kubernetes.io/projected/c52512ff-3354-4aba-9d39-99a3084e02af-kube-api-access-lpbd2\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:42.466952 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:42.466928    4525 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c52512ff-3354-4aba-9d39-99a3084e02af-plugins-config-volume\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:42.522706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:42.512257    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c52512ff-3354-4aba-9d39-99a3084e02af-provider-credentials" pod "c52512ff-3354-4aba-9d39-99a3084e02af" (UID: "c52512ff-3354-4aba-9d39-99a3084e02af"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:13:42.652501 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:42.652468    4525 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c52512ff-3354-4aba-9d39-99a3084e02af-provider-credentials\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:42.702706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:42.694209    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f3d6a5/payload-processing-external-model-e2e-extmod-b-f3d6a5-f7d7npjtw"
Oct 30 19:13:42.702706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:42.698009    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f3d6a5/payload-processing-external-model-e2e-extmod-b-f3d6a5-f7d7npjtw" event={"ID":"c52512ff-3354-4aba-9d39-99a3084e02af","Type":"ContainerDied","Data":"68c8e680874ec1dd0928db7da95af8dfd6efaf763b3c7c3744da85d3307a5282"}
Oct 30 19:13:42.702706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:42.698105    4525 scope.go:122] "RemoveContainer" containerID="d15f22c40ccf75d8c2265fe3d023ccc0401bf6843bf5a0a4a3d5b37d0ebc0dff"
Oct 30 19:13:42.858755 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:42.852880    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-f3d6a5/payload-processing-external-model-e2e-extmod-b-f3d6a5-f7d7npjtw"]
Oct 30 19:13:42.894707 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:42.892781    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-f3d6a5/payload-processing-external-model-e2e-extmod-b-f3d6a5-f7d7npjtw"]
Oct 30 19:13:42.950773 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:42.948120    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-jv2tm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused" start-of-body=
Oct 30 19:13:42.950773 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:42.948181    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-jv2tm" podUID="c4071206-077a-4d0f-8f0d-a2b76604d376" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused"
Oct 30 19:13:43.425376 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:43.425337    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused" start-of-body=
Oct 30 19:13:43.425667 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:43.425627    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw" podUID="bd0977b0-c75d-41ee-b911-4e71bd8c9b3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused"
Oct 30 19:13:43.445831 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:43.445798    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-37f2b8-qznrj"
Oct 30 19:13:43.535513 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:43.535468    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/139e7b9b-49a4-4dff-b2cf-434f9a6cb3bf-kube-api-access-mc9zr\" (UniqueName: \"kubernetes.io/projected/139e7b9b-49a4-4dff-b2cf-434f9a6cb3bf-kube-api-access-mc9zr\") pod \"139e7b9b-49a4-4dff-b2cf-434f9a6cb3bf\" (UID: \"139e7b9b-49a4-4dff-b2cf-434f9a6cb3bf\") "
Oct 30 19:13:43.537918 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:43.537893    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/139e7b9b-49a4-4dff-b2cf-434f9a6cb3bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/139e7b9b-49a4-4dff-b2cf-434f9a6cb3bf-maas-api-xks-ca\") pod \"139e7b9b-49a4-4dff-b2cf-434f9a6cb3bf\" (UID: \"139e7b9b-49a4-4dff-b2cf-434f9a6cb3bf\") "
Oct 30 19:13:43.539141 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:43.539121    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/139e7b9b-49a4-4dff-b2cf-434f9a6cb3bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/139e7b9b-49a4-4dff-b2cf-434f9a6cb3bf-maas-api-openshift-ca\") pod \"139e7b9b-49a4-4dff-b2cf-434f9a6cb3bf\" (UID: \"139e7b9b-49a4-4dff-b2cf-434f9a6cb3bf\") "
Oct 30 19:13:43.539735 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:43.539632    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/139e7b9b-49a4-4dff-b2cf-434f9a6cb3bf-maas-api-openshift-ca" pod "139e7b9b-49a4-4dff-b2cf-434f9a6cb3bf" (UID: "139e7b9b-49a4-4dff-b2cf-434f9a6cb3bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:13:43.541947 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:43.541908    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/139e7b9b-49a4-4dff-b2cf-434f9a6cb3bf-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:43.548726 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:43.548634    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/139e7b9b-49a4-4dff-b2cf-434f9a6cb3bf-kube-api-access-mc9zr" pod "139e7b9b-49a4-4dff-b2cf-434f9a6cb3bf" (UID: "139e7b9b-49a4-4dff-b2cf-434f9a6cb3bf"). InnerVolumeSpecName "kube-api-access-mc9zr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:13:43.565326 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:43.562150    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/139e7b9b-49a4-4dff-b2cf-434f9a6cb3bf-maas-api-xks-ca" pod "139e7b9b-49a4-4dff-b2cf-434f9a6cb3bf" (UID: "139e7b9b-49a4-4dff-b2cf-434f9a6cb3bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:13:43.566553 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:43.566527    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c52512ff-3354-4aba-9d39-99a3084e02af" path="/var/lib/kubelet/pods/c52512ff-3354-4aba-9d39-99a3084e02af/volumes"
Oct 30 19:13:43.647812 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:43.647777    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mc9zr\" (UniqueName: \"kubernetes.io/projected/139e7b9b-49a4-4dff-b2cf-434f9a6cb3bf-kube-api-access-mc9zr\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:43.647812 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:43.647815    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/139e7b9b-49a4-4dff-b2cf-434f9a6cb3bf-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:43.744446 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:43.744419    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-37f2b8-qznrj"
Oct 30 19:13:43.745080 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:43.744422    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-37f2b8-qznrj" event={"ID":"139e7b9b-49a4-4dff-b2cf-434f9a6cb3bf","Type":"ContainerDied","Data":"a47dbee543b9cb74d70fb0fb3a36276b7147d9c933f8a3ba556555c214a5fc00"}
Oct 30 19:13:43.745186 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:43.745169    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a47dbee543b9cb74d70fb0fb3a36276b7147d9c933f8a3ba556555c214a5fc00"
Oct 30 19:13:43.950156 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:43.950113    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-jv2tm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused" start-of-body=
Oct 30 19:13:43.950456 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:43.950429    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-jv2tm" podUID="c4071206-077a-4d0f-8f0d-a2b76604d376" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused"
Oct 30 19:13:44.418491 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:44.418455    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused" start-of-body=
Oct 30 19:13:44.418814 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:44.418783    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw" podUID="bd0977b0-c75d-41ee-b911-4e71bd8c9b3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused"
Oct 30 19:13:44.949702 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:44.948768    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-jv2tm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused" start-of-body=
Oct 30 19:13:44.949702 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:44.948856    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-jv2tm" podUID="c4071206-077a-4d0f-8f0d-a2b76604d376" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused"
Oct 30 19:13:45.418560 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:45.418530    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw"
Oct 30 19:13:45.418937 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:45.418906    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw"
Oct 30 19:13:45.419787 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:45.419768    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw"
Oct 30 19:13:45.951886 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:45.951855    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-jv2tm"
Oct 30 19:13:45.952298 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:45.952278    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-jv2tm"
Oct 30 19:13:45.952953 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:13:45.952931    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-jv2tm"
Oct 30 19:13:47.401521 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:13:47.401480    4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d49l" containerName="extractor"
Oct 30 19:14:01.578474 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:01.578440    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-6crc4"]
Oct 30 19:14:01.579566 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:01.579541    4525 cpu_manager.go:414] "containerMap: removing container" podUID="139e7b9b-49a4-4dff-b2cf-434f9a6cb3bf" containerName="revoke-keys"
Oct 30 19:14:01.579638 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:01.579571    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="139e7b9b-49a4-4dff-b2cf-434f9a6cb3bf" containerName="revoke-keys"
Oct 30 19:14:01.579638 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:01.579603    4525 cpu_manager.go:414] "containerMap: removing container" podUID="c52512ff-3354-4aba-9d39-99a3084e02af" containerName="payload-processing"
Oct 30 19:14:01.579638 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:01.579612    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c52512ff-3354-4aba-9d39-99a3084e02af" containerName="payload-processing"
Oct 30 19:14:01.579889 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:01.579872    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="139e7b9b-49a4-4dff-b2cf-434f9a6cb3bf" containerName="revoke-keys"
Oct 30 19:14:01.579928 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:01.579897    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="c52512ff-3354-4aba-9d39-99a3084e02af" containerName="payload-processing"
Oct 30 19:14:01.585854 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:01.585830    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-6crc4"
Oct 30 19:14:01.609520 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:01.609489    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-6crc4"]
Oct 30 19:14:01.647638 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:01.647605    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcr65\" (UniqueName: \"kubernetes.io/projected/f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419-kube-api-access-zcr65\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-6crc4\" (UID: \"f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-6crc4"
Oct 30 19:14:01.647865 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:01.647778    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419-workload-certs\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-6crc4\" (UID: \"f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-6crc4"
Oct 30 19:14:01.648185 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:01.648159    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419-istio-token\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-6crc4\" (UID: \"f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-6crc4"
Oct 30 19:14:01.648386 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:01.648363    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419-workload-socket\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-6crc4\" (UID: \"f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-6crc4"
Oct 30 19:14:01.648465 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:01.648416    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419-credential-socket\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-6crc4\" (UID: \"f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-6crc4"
Oct 30 19:14:01.648465 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:01.648434    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419-istio-data\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-6crc4\" (UID: \"f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-6crc4"
Oct 30 19:14:01.648769 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:01.648742    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419-istio-podinfo\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-6crc4\" (UID: \"f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-6crc4"
Oct 30 19:14:01.648881 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:01.648870    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419-istio-envoy\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-6crc4\" (UID: \"f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-6crc4"
Oct 30 19:14:01.649052 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:01.649027    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419-istiod-ca-cert\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-6crc4\" (UID: \"f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-6crc4"
Oct 30 19:14:01.751187 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:01.751154    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcr65\" (UniqueName: \"kubernetes.io/projected/f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419-kube-api-access-zcr65\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-6crc4\" (UID: \"f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-6crc4"
Oct 30 19:14:01.751389 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:01.751227    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419-workload-certs\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-6crc4\" (UID: \"f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-6crc4"
Oct 30 19:14:01.751389 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:01.751363    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419-istio-token\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-6crc4\" (UID: \"f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-6crc4"
Oct 30 19:14:01.751530 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:01.751410    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419-workload-socket\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-6crc4\" (UID: \"f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-6crc4"
Oct 30 19:14:01.751530 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:01.751448    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419-credential-socket\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-6crc4\" (UID: \"f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-6crc4"
Oct 30 19:14:01.751530 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:01.751469    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419-istio-data\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-6crc4\" (UID: \"f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-6crc4"
Oct 30 19:14:01.751728 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:01.751530    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419-istio-podinfo\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-6crc4\" (UID: \"f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-6crc4"
Oct 30 19:14:01.751728 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:01.751635    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419-istio-envoy\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-6crc4\" (UID: \"f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-6crc4"
Oct 30 19:14:01.751728 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:01.751716    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419-workload-certs\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-6crc4\" (UID: \"f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-6crc4"
Oct 30 19:14:01.751904 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:01.751782    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419-workload-socket\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-6crc4\" (UID: \"f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-6crc4"
Oct 30 19:14:01.751904 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:01.751787    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419-istiod-ca-cert\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-6crc4\" (UID: \"f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-6crc4"
Oct 30 19:14:01.751904 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:01.751852    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419-credential-socket\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-6crc4\" (UID: \"f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-6crc4"
Oct 30 19:14:01.752084 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:01.751971    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419-istio-data\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-6crc4\" (UID: \"f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-6crc4"
Oct 30 19:14:01.752416 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:01.752394    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419-istiod-ca-cert\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-6crc4\" (UID: \"f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-6crc4"
Oct 30 19:14:01.754162 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:01.754135    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419-istio-envoy\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-6crc4\" (UID: \"f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-6crc4"
Oct 30 19:14:01.754381 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:01.754362    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419-istio-podinfo\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-6crc4\" (UID: \"f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-6crc4"
Oct 30 19:14:01.759582 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:01.759560    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419-istio-token\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-6crc4\" (UID: \"f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-6crc4"
Oct 30 19:14:01.759813 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:01.759792    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcr65\" (UniqueName: \"kubernetes.io/projected/f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419-kube-api-access-zcr65\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-6crc4\" (UID: \"f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-6crc4"
Oct 30 19:14:01.933271 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:01.933235    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-6crc4"
Oct 30 19:14:02.268566 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:02.268532    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-6crc4"]
Oct 30 19:14:02.271244 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:14:02.271216    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8bb77e5_3e3a_4bc7_ae77_06b5eb01d419.slice/crio-31b202142edd53065be9f6c0dfa3de2b7ef07bb363b0252d672140080691cbe0 WatchSource:0}: Error finding container 31b202142edd53065be9f6c0dfa3de2b7ef07bb363b0252d672140080691cbe0: Status 404 returned error can't find the container with id 31b202142edd53065be9f6c0dfa3de2b7ef07bb363b0252d672140080691cbe0
Oct 30 19:14:02.275055 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:02.275024    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:14:02.275157 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:02.275091    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:14:02.275157 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:02.275121    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:14:03.289706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:03.289593    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-6crc4" event={"ID":"f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419","Type":"ContainerStarted","Data":"8ee5451b361b14e17fa190056169c7ac412528760bf7e91138bce835ef7d6ccb"}
Oct 30 19:14:03.301704 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:03.289671    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-6crc4" event={"ID":"f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419","Type":"ContainerStarted","Data":"31b202142edd53065be9f6c0dfa3de2b7ef07bb363b0252d672140080691cbe0"}
Oct 30 19:14:03.321190 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:03.321139    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-6crc4" podStartSLOduration=2.321125251 podStartE2EDuration="2.321125251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:14:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:14:03.316455523 +0000 UTC m=+2265.942125487" watchObservedRunningTime="2026-10-30 19:14:03.321125251 +0000 UTC m=+2265.946795356"
Oct 30 19:14:03.933873 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:03.933793    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-6crc4"
Oct 30 19:14:03.935311 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:03.934935    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-d6f956-openshift-default-56888545df-6crc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused" start-of-body=
Oct 30 19:14:03.935311 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:03.934989    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-6crc4" podUID="f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused"
Oct 30 19:14:04.934221 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:04.934190    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-d6f956-openshift-default-56888545df-6crc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused" start-of-body=
Oct 30 19:14:04.934718 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:04.934248    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-6crc4" podUID="f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused"
Oct 30 19:14:05.933993 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:05.933950    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-d6f956-openshift-default-56888545df-6crc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused" start-of-body=
Oct 30 19:14:05.934179 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:05.934025    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-6crc4" podUID="f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused"
Oct 30 19:14:06.211403 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:06.211311    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-37f2b8-546dfdf475-q9nfs"]
Oct 30 19:14:06.211942 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:06.211594    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-37f2b8-546dfdf475-q9nfs" podUID="81d638fc-4ff9-4cd3-9b13-cb71154b4e9a" containerName="maas-api" containerID="cri-o://cf6e6ddcc3db9c349c46bf3e241ee24fff7502865e20516c7b17f6ae9363d022" gracePeriod=30
Oct 30 19:14:06.941951 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:06.941908    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-d6f956-openshift-default-56888545df-6crc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused" start-of-body=
Oct 30 19:14:06.942290 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:06.942243    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-6crc4" podUID="f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused"
Oct 30 19:14:07.012757 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:07.008169    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-37f2b8-qznrj"]
Oct 30 19:14:07.018148 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:07.018101    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-37f2b8-qznrj"]
Oct 30 19:14:07.218116 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:07.218089    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-37f2b8-546dfdf475-q9nfs"
Oct 30 19:14:07.336232 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:07.336145    4525 generic.go:356] "Generic (PLEG): container finished" podID="81d638fc-4ff9-4cd3-9b13-cb71154b4e9a" containerID="cf6e6ddcc3db9c349c46bf3e241ee24fff7502865e20516c7b17f6ae9363d022" exitCode=0
Oct 30 19:14:07.336232 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:07.336193    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-37f2b8-546dfdf475-q9nfs" event={"ID":"81d638fc-4ff9-4cd3-9b13-cb71154b4e9a","Type":"ContainerDied","Data":"cf6e6ddcc3db9c349c46bf3e241ee24fff7502865e20516c7b17f6ae9363d022"}
Oct 30 19:14:07.336478 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:07.336270    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-37f2b8-546dfdf475-q9nfs" event={"ID":"81d638fc-4ff9-4cd3-9b13-cb71154b4e9a","Type":"ContainerDied","Data":"4ee2f567d1a16e71378ca6d5d55224a30221f30b1966f0ce600409d8ae7b7e0d"}
Oct 30 19:14:07.336478 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:07.336301    4525 scope.go:122] "RemoveContainer" containerID="cf6e6ddcc3db9c349c46bf3e241ee24fff7502865e20516c7b17f6ae9363d022"
Oct 30 19:14:07.336478 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:07.336202    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-37f2b8-546dfdf475-q9nfs"
Oct 30 19:14:07.371479 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:07.368140    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81d638fc-4ff9-4cd3-9b13-cb71154b4e9a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81d638fc-4ff9-4cd3-9b13-cb71154b4e9a-maas-api-tls\") pod \"81d638fc-4ff9-4cd3-9b13-cb71154b4e9a\" (UID: \"81d638fc-4ff9-4cd3-9b13-cb71154b4e9a\") "
Oct 30 19:14:07.372037 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:07.372013    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81d638fc-4ff9-4cd3-9b13-cb71154b4e9a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/81d638fc-4ff9-4cd3-9b13-cb71154b4e9a-metrics-certs\") pod \"81d638fc-4ff9-4cd3-9b13-cb71154b4e9a\" (UID: \"81d638fc-4ff9-4cd3-9b13-cb71154b4e9a\") "
Oct 30 19:14:07.372283 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:07.372262    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81d638fc-4ff9-4cd3-9b13-cb71154b4e9a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81d638fc-4ff9-4cd3-9b13-cb71154b4e9a-openshift-service-ca\") pod \"81d638fc-4ff9-4cd3-9b13-cb71154b4e9a\" (UID: \"81d638fc-4ff9-4cd3-9b13-cb71154b4e9a\") "
Oct 30 19:14:07.372463 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:07.372449    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81d638fc-4ff9-4cd3-9b13-cb71154b4e9a-kube-api-access-vfcj9\" (UniqueName: \"kubernetes.io/projected/81d638fc-4ff9-4cd3-9b13-cb71154b4e9a-kube-api-access-vfcj9\") pod \"81d638fc-4ff9-4cd3-9b13-cb71154b4e9a\" (UID: \"81d638fc-4ff9-4cd3-9b13-cb71154b4e9a\") "
Oct 30 19:14:07.373568 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:07.373313    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81d638fc-4ff9-4cd3-9b13-cb71154b4e9a-openshift-service-ca" pod "81d638fc-4ff9-4cd3-9b13-cb71154b4e9a" (UID: "81d638fc-4ff9-4cd3-9b13-cb71154b4e9a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:14:07.375031 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:07.375011    4525 scope.go:122] "RemoveContainer" containerID="cf6e6ddcc3db9c349c46bf3e241ee24fff7502865e20516c7b17f6ae9363d022"
Oct 30 19:14:07.375220 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:07.374995    4525 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81d638fc-4ff9-4cd3-9b13-cb71154b4e9a-openshift-service-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:07.413708 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:07.413011    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81d638fc-4ff9-4cd3-9b13-cb71154b4e9a-kube-api-access-vfcj9" pod "81d638fc-4ff9-4cd3-9b13-cb71154b4e9a" (UID: "81d638fc-4ff9-4cd3-9b13-cb71154b4e9a"). InnerVolumeSpecName "kube-api-access-vfcj9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:14:07.413708 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:07.413514    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81d638fc-4ff9-4cd3-9b13-cb71154b4e9a-metrics-certs" pod "81d638fc-4ff9-4cd3-9b13-cb71154b4e9a" (UID: "81d638fc-4ff9-4cd3-9b13-cb71154b4e9a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:14:07.413708 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:07.413611    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81d638fc-4ff9-4cd3-9b13-cb71154b4e9a-maas-api-tls" pod "81d638fc-4ff9-4cd3-9b13-cb71154b4e9a" (UID: "81d638fc-4ff9-4cd3-9b13-cb71154b4e9a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:14:07.413997 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:14:07.413922    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf6e6ddcc3db9c349c46bf3e241ee24fff7502865e20516c7b17f6ae9363d022\": container with ID starting with cf6e6ddcc3db9c349c46bf3e241ee24fff7502865e20516c7b17f6ae9363d022 not found: ID does not exist" containerID="cf6e6ddcc3db9c349c46bf3e241ee24fff7502865e20516c7b17f6ae9363d022"
Oct 30 19:14:07.414056 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:07.413990    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf6e6ddcc3db9c349c46bf3e241ee24fff7502865e20516c7b17f6ae9363d022"} err="failed to get container status \"cf6e6ddcc3db9c349c46bf3e241ee24fff7502865e20516c7b17f6ae9363d022\": rpc error: code = NotFound desc = could not find container \"cf6e6ddcc3db9c349c46bf3e241ee24fff7502865e20516c7b17f6ae9363d022\": container with ID starting with cf6e6ddcc3db9c349c46bf3e241ee24fff7502865e20516c7b17f6ae9363d022 not found: ID does not exist"
Oct 30 19:14:07.494706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:07.482380    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfcj9\" (UniqueName: \"kubernetes.io/projected/81d638fc-4ff9-4cd3-9b13-cb71154b4e9a-kube-api-access-vfcj9\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:07.494706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:07.482422    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81d638fc-4ff9-4cd3-9b13-cb71154b4e9a-maas-api-tls\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:07.494706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:07.482438    4525 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81d638fc-4ff9-4cd3-9b13-cb71154b4e9a-metrics-certs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:07.547877 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:07.547845    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="139e7b9b-49a4-4dff-b2cf-434f9a6cb3bf" path="/var/lib/kubelet/pods/139e7b9b-49a4-4dff-b2cf-434f9a6cb3bf/volumes"
Oct 30 19:14:07.657283 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:07.657218    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-37f2b8-546dfdf475-q9nfs"]
Oct 30 19:14:07.663784 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:07.663738    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-37f2b8-546dfdf475-q9nfs"]
Oct 30 19:14:07.962756 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:07.961988    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-d6f956-openshift-default-56888545df-6crc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused" start-of-body=
Oct 30 19:14:07.962756 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:07.962069    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-6crc4" podUID="f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused"
Oct 30 19:14:08.875713 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:08.842801    4525 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t" podUID="424fca14-6d8a-4663-86ec-0c91165f34ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:14:08.962711 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:08.962522    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-d6f956-openshift-default-56888545df-6crc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused" start-of-body=
Oct 30 19:14:08.962711 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:08.962584    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-6crc4" podUID="f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused"
Oct 30 19:14:09.410709 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:09.403830    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t" podUID="424fca14-6d8a-4663-86ec-0c91165f34ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 19:14:09.410951 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:09.410770    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr" podUID="82bbbf85-ed93-48d2-9f17-49ac1f31c0a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:14:09.668764 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:09.656518    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81d638fc-4ff9-4cd3-9b13-cb71154b4e9a" path="/var/lib/kubelet/pods/81d638fc-4ff9-4cd3-9b13-cb71154b4e9a/volumes"
Oct 30 19:14:09.738879 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:09.738780    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb" podUID="12ebc8b9-c794-47b5-b87b-a63c4bd993e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:14:09.933848 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:09.933760    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-d6f956-openshift-default-56888545df-6crc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused" start-of-body=
Oct 30 19:14:09.933848 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:09.933821    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-6crc4" podUID="f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused"
Oct 30 19:14:11.016722 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:11.012333    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-6crc4"
Oct 30 19:14:11.016722 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:11.014435    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-6crc4"
Oct 30 19:14:11.016722 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:11.014965    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-6crc4"
Oct 30 19:14:11.300626 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:11.300541    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8684c15-vf9cs"]
Oct 30 19:14:11.301412 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:11.301388    4525 cpu_manager.go:414] "containerMap: removing container" podUID="81d638fc-4ff9-4cd3-9b13-cb71154b4e9a" containerName="maas-api"
Oct 30 19:14:11.301539 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:11.301412    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81d638fc-4ff9-4cd3-9b13-cb71154b4e9a" containerName="maas-api"
Oct 30 19:14:11.301694 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:11.301660    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="81d638fc-4ff9-4cd3-9b13-cb71154b4e9a" containerName="maas-api"
Oct 30 19:14:11.307120 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:11.307094    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8684c15-vf9cs"
Oct 30 19:14:11.312727 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:11.312700    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8684c15-vf9cs"]
Oct 30 19:14:11.337263 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:11.337225    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16cc85e9-28d0-45d4-ac48-6f15f18399a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8684c15-vf9cs\" (UID: \"16cc85e9-28d0-45d4-ac48-6f15f18399a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8684c15-vf9cs"
Oct 30 19:14:11.337401 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:11.337378    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cl7lw\" (UniqueName: \"kubernetes.io/projected/16cc85e9-28d0-45d4-ac48-6f15f18399a3-kube-api-access-cl7lw\") pod \"maas-api-revoke-sub-c8684c15-vf9cs\" (UID: \"16cc85e9-28d0-45d4-ac48-6f15f18399a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8684c15-vf9cs"
Oct 30 19:14:11.337481 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:11.337465    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16cc85e9-28d0-45d4-ac48-6f15f18399a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8684c15-vf9cs\" (UID: \"16cc85e9-28d0-45d4-ac48-6f15f18399a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8684c15-vf9cs"
Oct 30 19:14:11.387298 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:11.387241    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:14:11.387454 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:11.387425    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94hcn\""
Oct 30 19:14:11.439199 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:11.439164    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16cc85e9-28d0-45d4-ac48-6f15f18399a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8684c15-vf9cs\" (UID: \"16cc85e9-28d0-45d4-ac48-6f15f18399a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8684c15-vf9cs"
Oct 30 19:14:11.439403 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:11.439261    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl7lw\" (UniqueName: \"kubernetes.io/projected/16cc85e9-28d0-45d4-ac48-6f15f18399a3-kube-api-access-cl7lw\") pod \"maas-api-revoke-sub-c8684c15-vf9cs\" (UID: \"16cc85e9-28d0-45d4-ac48-6f15f18399a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8684c15-vf9cs"
Oct 30 19:14:11.439403 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:11.439327    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16cc85e9-28d0-45d4-ac48-6f15f18399a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8684c15-vf9cs\" (UID: \"16cc85e9-28d0-45d4-ac48-6f15f18399a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8684c15-vf9cs"
Oct 30 19:14:11.440025 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:11.440006    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16cc85e9-28d0-45d4-ac48-6f15f18399a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8684c15-vf9cs\" (UID: \"16cc85e9-28d0-45d4-ac48-6f15f18399a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8684c15-vf9cs"
Oct 30 19:14:11.441819 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:11.441794    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16cc85e9-28d0-45d4-ac48-6f15f18399a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8684c15-vf9cs\" (UID: \"16cc85e9-28d0-45d4-ac48-6f15f18399a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8684c15-vf9cs"
Oct 30 19:14:11.447628 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:11.447605    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl7lw\" (UniqueName: \"kubernetes.io/projected/16cc85e9-28d0-45d4-ac48-6f15f18399a3-kube-api-access-cl7lw\") pod \"maas-api-revoke-sub-c8684c15-vf9cs\" (UID: \"16cc85e9-28d0-45d4-ac48-6f15f18399a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8684c15-vf9cs"
Oct 30 19:14:11.627089 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:11.627050    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8684c15-vf9cs"
Oct 30 19:14:12.008977 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:14:12.008944    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16cc85e9_28d0_45d4_ac48_6f15f18399a3.slice/crio-6e02b0219cd5388f8038054b10f3ad2f0e22d35a4b2c48b87e381dd43310d5c9 WatchSource:0}: Error finding container 6e02b0219cd5388f8038054b10f3ad2f0e22d35a4b2c48b87e381dd43310d5c9: Status 404 returned error can't find the container with id 6e02b0219cd5388f8038054b10f3ad2f0e22d35a4b2c48b87e381dd43310d5c9
Oct 30 19:14:12.010413 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:12.010389    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8684c15-vf9cs"]
Oct 30 19:14:12.021081 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:12.017750    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14c2b681-cvbkd"]
Oct 30 19:14:12.026915 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:12.026878    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14c2b681-cvbkd"]
Oct 30 19:14:12.027054 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:12.027011    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14c2b681-cvbkd"
Oct 30 19:14:12.076923 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:12.076890    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb17e9a7-c339-431a-bac6-f5d21ae854a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14c2b681-cvbkd\" (UID: \"bb17e9a7-c339-431a-bac6-f5d21ae854a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14c2b681-cvbkd"
Oct 30 19:14:12.077165 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:12.077150    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22kh9\" (UniqueName: \"kubernetes.io/projected/bb17e9a7-c339-431a-bac6-f5d21ae854a9-kube-api-access-22kh9\") pod \"maas-api-revoke-sub-14c2b681-cvbkd\" (UID: \"bb17e9a7-c339-431a-bac6-f5d21ae854a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14c2b681-cvbkd"
Oct 30 19:14:12.077208 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:12.077190    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb17e9a7-c339-431a-bac6-f5d21ae854a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14c2b681-cvbkd\" (UID: \"bb17e9a7-c339-431a-bac6-f5d21ae854a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14c2b681-cvbkd"
Oct 30 19:14:12.179149 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:12.179100    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb17e9a7-c339-431a-bac6-f5d21ae854a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14c2b681-cvbkd\" (UID: \"bb17e9a7-c339-431a-bac6-f5d21ae854a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14c2b681-cvbkd"
Oct 30 19:14:12.179335 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:12.179317    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22kh9\" (UniqueName: \"kubernetes.io/projected/bb17e9a7-c339-431a-bac6-f5d21ae854a9-kube-api-access-22kh9\") pod \"maas-api-revoke-sub-14c2b681-cvbkd\" (UID: \"bb17e9a7-c339-431a-bac6-f5d21ae854a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14c2b681-cvbkd"
Oct 30 19:14:12.179399 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:12.179358    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb17e9a7-c339-431a-bac6-f5d21ae854a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14c2b681-cvbkd\" (UID: \"bb17e9a7-c339-431a-bac6-f5d21ae854a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14c2b681-cvbkd"
Oct 30 19:14:12.180082 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:12.180001    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb17e9a7-c339-431a-bac6-f5d21ae854a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14c2b681-cvbkd\" (UID: \"bb17e9a7-c339-431a-bac6-f5d21ae854a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14c2b681-cvbkd"
Oct 30 19:14:12.181798 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:12.181774    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb17e9a7-c339-431a-bac6-f5d21ae854a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14c2b681-cvbkd\" (UID: \"bb17e9a7-c339-431a-bac6-f5d21ae854a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14c2b681-cvbkd"
Oct 30 19:14:12.188005 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:12.187983    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22kh9\" (UniqueName: \"kubernetes.io/projected/bb17e9a7-c339-431a-bac6-f5d21ae854a9-kube-api-access-22kh9\") pod \"maas-api-revoke-sub-14c2b681-cvbkd\" (UID: \"bb17e9a7-c339-431a-bac6-f5d21ae854a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14c2b681-cvbkd"
Oct 30 19:14:12.354658 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:12.354001    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14c2b681-cvbkd"
Oct 30 19:14:12.549231 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:12.549131    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8684c15-vf9cs" event={"ID":"16cc85e9-28d0-45d4-ac48-6f15f18399a3","Type":"ContainerStarted","Data":"a2011a75ad980ed201f912617d47de2641f17123bfff27ef0660c0998b7d18b7"}
Oct 30 19:14:12.549231 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:12.549193    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8684c15-vf9cs" event={"ID":"16cc85e9-28d0-45d4-ac48-6f15f18399a3","Type":"ContainerStarted","Data":"6e02b0219cd5388f8038054b10f3ad2f0e22d35a4b2c48b87e381dd43310d5c9"}
Oct 30 19:14:12.583571 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:12.583520    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8684c15-vf9cs" podStartSLOduration=1.583503729 podStartE2EDuration="1.583503729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:14:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:14:12.576921387 +0000 UTC m=+2275.202591347" watchObservedRunningTime="2026-10-30 19:14:12.583503729 +0000 UTC m=+2275.209173693"
Oct 30 19:14:12.716870 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:14:12.716836    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb17e9a7_c339_431a_bac6_f5d21ae854a9.slice/crio-dfba4bbb4312523aa73e96f379df2cf9b0bfaae1edbbac3ebe281c6a50a305f0 WatchSource:0}: Error finding container dfba4bbb4312523aa73e96f379df2cf9b0bfaae1edbbac3ebe281c6a50a305f0: Status 404 returned error can't find the container with id dfba4bbb4312523aa73e96f379df2cf9b0bfaae1edbbac3ebe281c6a50a305f0
Oct 30 19:14:12.717895 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:12.717842    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14c2b681-cvbkd"]
Oct 30 19:14:13.562464 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:13.562362    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14c2b681-cvbkd" event={"ID":"bb17e9a7-c339-431a-bac6-f5d21ae854a9","Type":"ContainerStarted","Data":"2a711a7926861033413f3c30e2d85010fa8dff3f6c4ec4d4111107f1a97b0084"}
Oct 30 19:14:13.562464 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:13.562424    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14c2b681-cvbkd" event={"ID":"bb17e9a7-c339-431a-bac6-f5d21ae854a9","Type":"ContainerStarted","Data":"dfba4bbb4312523aa73e96f379df2cf9b0bfaae1edbbac3ebe281c6a50a305f0"}
Oct 30 19:14:13.564258 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:13.564229    4525 generic.go:356] "Generic (PLEG): container finished" podID="16cc85e9-28d0-45d4-ac48-6f15f18399a3" containerID="a2011a75ad980ed201f912617d47de2641f17123bfff27ef0660c0998b7d18b7" exitCode=0
Oct 30 19:14:13.564368 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:13.564294    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8684c15-vf9cs" event={"ID":"16cc85e9-28d0-45d4-ac48-6f15f18399a3","Type":"ContainerDied","Data":"a2011a75ad980ed201f912617d47de2641f17123bfff27ef0660c0998b7d18b7"}
Oct 30 19:14:13.583415 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:13.583351    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14c2b681-cvbkd" podStartSLOduration=2.583335142 podStartE2EDuration="2.583335142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:14:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:14:13.58184843 +0000 UTC m=+2276.207518412" watchObservedRunningTime="2026-10-30 19:14:13.583335142 +0000 UTC m=+2276.209005105"
Oct 30 19:14:14.731443 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:14.731407    4525 generic.go:356] "Generic (PLEG): container finished" podID="bb17e9a7-c339-431a-bac6-f5d21ae854a9" containerID="2a711a7926861033413f3c30e2d85010fa8dff3f6c4ec4d4111107f1a97b0084" exitCode=0
Oct 30 19:14:14.732751 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:14.732727    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14c2b681-cvbkd" event={"ID":"bb17e9a7-c339-431a-bac6-f5d21ae854a9","Type":"ContainerDied","Data":"2a711a7926861033413f3c30e2d85010fa8dff3f6c4ec4d4111107f1a97b0084"}
Oct 30 19:14:14.805343 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:14.805306    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ecc266-2spmx"]
Oct 30 19:14:14.810115 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:14.810094    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ecc266-2spmx"
Oct 30 19:14:14.835619 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:14.834270    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ecc266-2spmx"]
Oct 30 19:14:14.954785 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:14.954758    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8684c15-vf9cs"
Oct 30 19:14:15.021752 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:15.021312    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16cc85e9-28d0-45d4-ac48-6f15f18399a3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16cc85e9-28d0-45d4-ac48-6f15f18399a3-maas-api-xks-ca\") pod \"16cc85e9-28d0-45d4-ac48-6f15f18399a3\" (UID: \"16cc85e9-28d0-45d4-ac48-6f15f18399a3\") "
Oct 30 19:14:15.021752 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:15.021386    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16cc85e9-28d0-45d4-ac48-6f15f18399a3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16cc85e9-28d0-45d4-ac48-6f15f18399a3-maas-api-openshift-ca\") pod \"16cc85e9-28d0-45d4-ac48-6f15f18399a3\" (UID: \"16cc85e9-28d0-45d4-ac48-6f15f18399a3\") "
Oct 30 19:14:15.021752 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:15.021519    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16cc85e9-28d0-45d4-ac48-6f15f18399a3-kube-api-access-cl7lw\" (UniqueName: \"kubernetes.io/projected/16cc85e9-28d0-45d4-ac48-6f15f18399a3-kube-api-access-cl7lw\") pod \"16cc85e9-28d0-45d4-ac48-6f15f18399a3\" (UID: \"16cc85e9-28d0-45d4-ac48-6f15f18399a3\") "
Oct 30 19:14:15.025704 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:15.022185    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/446c488b-eaeb-49c5-b99e-117d7aae32a4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-ecc266-2spmx\" (UID: \"446c488b-eaeb-49c5-b99e-117d7aae32a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ecc266-2spmx"
Oct 30 19:14:15.025704 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:15.022563    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/446c488b-eaeb-49c5-b99e-117d7aae32a4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-ecc266-2spmx\" (UID: \"446c488b-eaeb-49c5-b99e-117d7aae32a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ecc266-2spmx"
Oct 30 19:14:15.025704 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:15.022656    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spszq\" (UniqueName: \"kubernetes.io/projected/446c488b-eaeb-49c5-b99e-117d7aae32a4-kube-api-access-spszq\") pod \"maas-api-revoke-keys-e2e-extmod-b-ecc266-2spmx\" (UID: \"446c488b-eaeb-49c5-b99e-117d7aae32a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ecc266-2spmx"
Oct 30 19:14:15.025704 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:15.024112    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16cc85e9-28d0-45d4-ac48-6f15f18399a3-maas-api-openshift-ca" pod "16cc85e9-28d0-45d4-ac48-6f15f18399a3" (UID: "16cc85e9-28d0-45d4-ac48-6f15f18399a3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:14:15.053548 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:15.050479    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16cc85e9-28d0-45d4-ac48-6f15f18399a3-maas-api-xks-ca" pod "16cc85e9-28d0-45d4-ac48-6f15f18399a3" (UID: "16cc85e9-28d0-45d4-ac48-6f15f18399a3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:14:15.057707 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:15.057342    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16cc85e9-28d0-45d4-ac48-6f15f18399a3-kube-api-access-cl7lw" pod "16cc85e9-28d0-45d4-ac48-6f15f18399a3" (UID: "16cc85e9-28d0-45d4-ac48-6f15f18399a3"). InnerVolumeSpecName "kube-api-access-cl7lw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:14:15.125342 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:15.125286    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/446c488b-eaeb-49c5-b99e-117d7aae32a4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-ecc266-2spmx\" (UID: \"446c488b-eaeb-49c5-b99e-117d7aae32a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ecc266-2spmx"
Oct 30 19:14:15.125644 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:15.125433    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spszq\" (UniqueName: \"kubernetes.io/projected/446c488b-eaeb-49c5-b99e-117d7aae32a4-kube-api-access-spszq\") pod \"maas-api-revoke-keys-e2e-extmod-b-ecc266-2spmx\" (UID: \"446c488b-eaeb-49c5-b99e-117d7aae32a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ecc266-2spmx"
Oct 30 19:14:15.125644 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:15.125590    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/446c488b-eaeb-49c5-b99e-117d7aae32a4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-ecc266-2spmx\" (UID: \"446c488b-eaeb-49c5-b99e-117d7aae32a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ecc266-2spmx"
Oct 30 19:14:15.125879 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:15.125863    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cl7lw\" (UniqueName: \"kubernetes.io/projected/16cc85e9-28d0-45d4-ac48-6f15f18399a3-kube-api-access-cl7lw\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:15.125953 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:15.125883    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16cc85e9-28d0-45d4-ac48-6f15f18399a3-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:15.125953 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:15.125893    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16cc85e9-28d0-45d4-ac48-6f15f18399a3-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:15.127001 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:15.126971    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/446c488b-eaeb-49c5-b99e-117d7aae32a4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-ecc266-2spmx\" (UID: \"446c488b-eaeb-49c5-b99e-117d7aae32a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ecc266-2spmx"
Oct 30 19:14:15.130553 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:15.130524    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/446c488b-eaeb-49c5-b99e-117d7aae32a4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-ecc266-2spmx\" (UID: \"446c488b-eaeb-49c5-b99e-117d7aae32a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ecc266-2spmx"
Oct 30 19:14:15.135645 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:15.135617    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spszq\" (UniqueName: \"kubernetes.io/projected/446c488b-eaeb-49c5-b99e-117d7aae32a4-kube-api-access-spszq\") pod \"maas-api-revoke-keys-e2e-extmod-b-ecc266-2spmx\" (UID: \"446c488b-eaeb-49c5-b99e-117d7aae32a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ecc266-2spmx"
Oct 30 19:14:15.155759 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:15.155736    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ecc266-2spmx"
Oct 30 19:14:15.441144 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:14:15.441113    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod446c488b_eaeb_49c5_b99e_117d7aae32a4.slice/crio-009be92b1e1a4589f9ed9949f8773fcf3e0784c1ac58c84c5136ed71548cd1cd WatchSource:0}: Error finding container 009be92b1e1a4589f9ed9949f8773fcf3e0784c1ac58c84c5136ed71548cd1cd: Status 404 returned error can't find the container with id 009be92b1e1a4589f9ed9949f8773fcf3e0784c1ac58c84c5136ed71548cd1cd
Oct 30 19:14:15.441607 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:15.441579    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ecc266-2spmx"]
Oct 30 19:14:15.743345 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:15.743261    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8684c15-vf9cs"
Oct 30 19:14:15.743345 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:15.743303    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8684c15-vf9cs" event={"ID":"16cc85e9-28d0-45d4-ac48-6f15f18399a3","Type":"ContainerDied","Data":"6e02b0219cd5388f8038054b10f3ad2f0e22d35a4b2c48b87e381dd43310d5c9"}
Oct 30 19:14:15.743803 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:15.743351    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e02b0219cd5388f8038054b10f3ad2f0e22d35a4b2c48b87e381dd43310d5c9"
Oct 30 19:14:15.745028 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:15.745001    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ecc266-2spmx" event={"ID":"446c488b-eaeb-49c5-b99e-117d7aae32a4","Type":"ContainerStarted","Data":"009be92b1e1a4589f9ed9949f8773fcf3e0784c1ac58c84c5136ed71548cd1cd"}
Oct 30 19:14:16.037374 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:16.037345    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14c2b681-cvbkd"
Oct 30 19:14:16.050128 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:16.050104    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb17e9a7-c339-431a-bac6-f5d21ae854a9-kube-api-access-22kh9\" (UniqueName: \"kubernetes.io/projected/bb17e9a7-c339-431a-bac6-f5d21ae854a9-kube-api-access-22kh9\") pod \"bb17e9a7-c339-431a-bac6-f5d21ae854a9\" (UID: \"bb17e9a7-c339-431a-bac6-f5d21ae854a9\") "
Oct 30 19:14:16.050254 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:16.050222    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb17e9a7-c339-431a-bac6-f5d21ae854a9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb17e9a7-c339-431a-bac6-f5d21ae854a9-maas-api-xks-ca\") pod \"bb17e9a7-c339-431a-bac6-f5d21ae854a9\" (UID: \"bb17e9a7-c339-431a-bac6-f5d21ae854a9\") "
Oct 30 19:14:16.050415 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:16.050395    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb17e9a7-c339-431a-bac6-f5d21ae854a9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb17e9a7-c339-431a-bac6-f5d21ae854a9-maas-api-openshift-ca\") pod \"bb17e9a7-c339-431a-bac6-f5d21ae854a9\" (UID: \"bb17e9a7-c339-431a-bac6-f5d21ae854a9\") "
Oct 30 19:14:16.050785 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:16.050758    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb17e9a7-c339-431a-bac6-f5d21ae854a9-maas-api-openshift-ca" pod "bb17e9a7-c339-431a-bac6-f5d21ae854a9" (UID: "bb17e9a7-c339-431a-bac6-f5d21ae854a9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:14:16.051268 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:16.051243    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb17e9a7-c339-431a-bac6-f5d21ae854a9-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:16.052542 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:16.052519    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb17e9a7-c339-431a-bac6-f5d21ae854a9-kube-api-access-22kh9" pod "bb17e9a7-c339-431a-bac6-f5d21ae854a9" (UID: "bb17e9a7-c339-431a-bac6-f5d21ae854a9"). InnerVolumeSpecName "kube-api-access-22kh9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:14:16.052542 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:16.052526    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb17e9a7-c339-431a-bac6-f5d21ae854a9-maas-api-xks-ca" pod "bb17e9a7-c339-431a-bac6-f5d21ae854a9" (UID: "bb17e9a7-c339-431a-bac6-f5d21ae854a9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:14:16.152762 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:16.152727    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22kh9\" (UniqueName: \"kubernetes.io/projected/bb17e9a7-c339-431a-bac6-f5d21ae854a9-kube-api-access-22kh9\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:16.152762 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:16.152758    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb17e9a7-c339-431a-bac6-f5d21ae854a9-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:16.576197 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:16.576162    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2cf46"]
Oct 30 19:14:16.576984 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:16.576963    4525 cpu_manager.go:414] "containerMap: removing container" podUID="16cc85e9-28d0-45d4-ac48-6f15f18399a3" containerName="revoke-keys"
Oct 30 19:14:16.577084 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:16.576987    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16cc85e9-28d0-45d4-ac48-6f15f18399a3" containerName="revoke-keys"
Oct 30 19:14:16.577084 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:16.577068    4525 cpu_manager.go:414] "containerMap: removing container" podUID="bb17e9a7-c339-431a-bac6-f5d21ae854a9" containerName="revoke-keys"
Oct 30 19:14:16.577084 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:16.577075    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb17e9a7-c339-431a-bac6-f5d21ae854a9" containerName="revoke-keys"
Oct 30 19:14:16.577285 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:16.577271    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb17e9a7-c339-431a-bac6-f5d21ae854a9" containerName="revoke-keys"
Oct 30 19:14:16.577327 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:16.577297    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="16cc85e9-28d0-45d4-ac48-6f15f18399a3" containerName="revoke-keys"
Oct 30 19:14:16.581639 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:16.581613    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2cf46"
Oct 30 19:14:16.593078 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:16.593049    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2cf46"]
Oct 30 19:14:16.698710 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:16.698664    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52a25631-2cfc-4155-a3c3-4a381f96bf8b-workload-socket\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-2cf46\" (UID: \"52a25631-2cfc-4155-a3c3-4a381f96bf8b\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2cf46"
Oct 30 19:14:16.698896 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:16.698740    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52a25631-2cfc-4155-a3c3-4a381f96bf8b-istio-podinfo\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-2cf46\" (UID: \"52a25631-2cfc-4155-a3c3-4a381f96bf8b\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2cf46"
Oct 30 19:14:16.698896 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:16.698793    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52a25631-2cfc-4155-a3c3-4a381f96bf8b-istio-data\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-2cf46\" (UID: \"52a25631-2cfc-4155-a3c3-4a381f96bf8b\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2cf46"
Oct 30 19:14:16.699028 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:16.698927    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52a25631-2cfc-4155-a3c3-4a381f96bf8b-istio-token\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-2cf46\" (UID: \"52a25631-2cfc-4155-a3c3-4a381f96bf8b\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2cf46"
Oct 30 19:14:16.699084 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:16.699060    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnvc4\" (UniqueName: \"kubernetes.io/projected/52a25631-2cfc-4155-a3c3-4a381f96bf8b-kube-api-access-hnvc4\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-2cf46\" (UID: \"52a25631-2cfc-4155-a3c3-4a381f96bf8b\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2cf46"
Oct 30 19:14:16.699131 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:16.699119    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52a25631-2cfc-4155-a3c3-4a381f96bf8b-credential-socket\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-2cf46\" (UID: \"52a25631-2cfc-4155-a3c3-4a381f96bf8b\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2cf46"
Oct 30 19:14:16.699228 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:16.699180    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52a25631-2cfc-4155-a3c3-4a381f96bf8b-workload-certs\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-2cf46\" (UID: \"52a25631-2cfc-4155-a3c3-4a381f96bf8b\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2cf46"
Oct 30 19:14:16.699353 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:16.699334    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52a25631-2cfc-4155-a3c3-4a381f96bf8b-istiod-ca-cert\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-2cf46\" (UID: \"52a25631-2cfc-4155-a3c3-4a381f96bf8b\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2cf46"
Oct 30 19:14:16.699421 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:16.699369    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52a25631-2cfc-4155-a3c3-4a381f96bf8b-istio-envoy\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-2cf46\" (UID: \"52a25631-2cfc-4155-a3c3-4a381f96bf8b\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2cf46"
Oct 30 19:14:16.755614 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:16.755575    4525 generic.go:356] "Generic (PLEG): container finished" podID="446c488b-eaeb-49c5-b99e-117d7aae32a4" containerID="e30fcf7acdd45ade7d2349627eb490c742250a83a0385dcb8a81d04978dd821f" exitCode=0
Oct 30 19:14:16.756054 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:16.755656    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ecc266-2spmx" event={"ID":"446c488b-eaeb-49c5-b99e-117d7aae32a4","Type":"ContainerDied","Data":"e30fcf7acdd45ade7d2349627eb490c742250a83a0385dcb8a81d04978dd821f"}
Oct 30 19:14:16.757327 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:16.757308    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14c2b681-cvbkd"
Oct 30 19:14:16.757444 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:16.757330    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14c2b681-cvbkd" event={"ID":"bb17e9a7-c339-431a-bac6-f5d21ae854a9","Type":"ContainerDied","Data":"dfba4bbb4312523aa73e96f379df2cf9b0bfaae1edbbac3ebe281c6a50a305f0"}
Oct 30 19:14:16.757444 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:16.757368    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dfba4bbb4312523aa73e96f379df2cf9b0bfaae1edbbac3ebe281c6a50a305f0"
Oct 30 19:14:16.800571 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:16.800536    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52a25631-2cfc-4155-a3c3-4a381f96bf8b-workload-socket\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-2cf46\" (UID: \"52a25631-2cfc-4155-a3c3-4a381f96bf8b\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2cf46"
Oct 30 19:14:16.800571 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:16.800575    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52a25631-2cfc-4155-a3c3-4a381f96bf8b-istio-podinfo\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-2cf46\" (UID: \"52a25631-2cfc-4155-a3c3-4a381f96bf8b\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2cf46"
Oct 30 19:14:16.800819 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:16.800610    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52a25631-2cfc-4155-a3c3-4a381f96bf8b-istio-data\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-2cf46\" (UID: \"52a25631-2cfc-4155-a3c3-4a381f96bf8b\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2cf46"
Oct 30 19:14:16.800819 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:16.800662    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52a25631-2cfc-4155-a3c3-4a381f96bf8b-istio-token\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-2cf46\" (UID: \"52a25631-2cfc-4155-a3c3-4a381f96bf8b\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2cf46"
Oct 30 19:14:16.800819 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:16.800787    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnvc4\" (UniqueName: \"kubernetes.io/projected/52a25631-2cfc-4155-a3c3-4a381f96bf8b-kube-api-access-hnvc4\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-2cf46\" (UID: \"52a25631-2cfc-4155-a3c3-4a381f96bf8b\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2cf46"
Oct 30 19:14:16.800975 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:16.800857    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52a25631-2cfc-4155-a3c3-4a381f96bf8b-credential-socket\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-2cf46\" (UID: \"52a25631-2cfc-4155-a3c3-4a381f96bf8b\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2cf46"
Oct 30 19:14:16.800975 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:16.800896    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52a25631-2cfc-4155-a3c3-4a381f96bf8b-workload-certs\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-2cf46\" (UID: \"52a25631-2cfc-4155-a3c3-4a381f96bf8b\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2cf46"
Oct 30 19:14:16.801090 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:16.801002    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52a25631-2cfc-4155-a3c3-4a381f96bf8b-workload-socket\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-2cf46\" (UID: \"52a25631-2cfc-4155-a3c3-4a381f96bf8b\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2cf46"
Oct 30 19:14:16.801090 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:16.801045    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52a25631-2cfc-4155-a3c3-4a381f96bf8b-istio-data\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-2cf46\" (UID: \"52a25631-2cfc-4155-a3c3-4a381f96bf8b\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2cf46"
Oct 30 19:14:16.801209 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:16.801137    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52a25631-2cfc-4155-a3c3-4a381f96bf8b-istiod-ca-cert\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-2cf46\" (UID: \"52a25631-2cfc-4155-a3c3-4a381f96bf8b\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2cf46"
Oct 30 19:14:16.801209 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:16.801187    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52a25631-2cfc-4155-a3c3-4a381f96bf8b-istio-envoy\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-2cf46\" (UID: \"52a25631-2cfc-4155-a3c3-4a381f96bf8b\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2cf46"
Oct 30 19:14:16.801327 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:16.801300    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52a25631-2cfc-4155-a3c3-4a381f96bf8b-credential-socket\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-2cf46\" (UID: \"52a25631-2cfc-4155-a3c3-4a381f96bf8b\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2cf46"
Oct 30 19:14:16.801402 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:16.801379    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52a25631-2cfc-4155-a3c3-4a381f96bf8b-workload-certs\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-2cf46\" (UID: \"52a25631-2cfc-4155-a3c3-4a381f96bf8b\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2cf46"
Oct 30 19:14:16.801739 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:16.801716    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52a25631-2cfc-4155-a3c3-4a381f96bf8b-istiod-ca-cert\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-2cf46\" (UID: \"52a25631-2cfc-4155-a3c3-4a381f96bf8b\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2cf46"
Oct 30 19:14:16.803309 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:16.803287    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52a25631-2cfc-4155-a3c3-4a381f96bf8b-istio-envoy\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-2cf46\" (UID: \"52a25631-2cfc-4155-a3c3-4a381f96bf8b\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2cf46"
Oct 30 19:14:16.803405 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:16.803359    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52a25631-2cfc-4155-a3c3-4a381f96bf8b-istio-podinfo\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-2cf46\" (UID: \"52a25631-2cfc-4155-a3c3-4a381f96bf8b\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2cf46"
Oct 30 19:14:16.811509 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:16.811485    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52a25631-2cfc-4155-a3c3-4a381f96bf8b-istio-token\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-2cf46\" (UID: \"52a25631-2cfc-4155-a3c3-4a381f96bf8b\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2cf46"
Oct 30 19:14:16.812581 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:16.812549    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnvc4\" (UniqueName: \"kubernetes.io/projected/52a25631-2cfc-4155-a3c3-4a381f96bf8b-kube-api-access-hnvc4\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-2cf46\" (UID: \"52a25631-2cfc-4155-a3c3-4a381f96bf8b\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2cf46"
Oct 30 19:14:16.938606 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:16.938577    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2cf46"
Oct 30 19:14:17.288574 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:14:17.288540    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52a25631_2cfc_4155_a3c3_4a381f96bf8b.slice/crio-da703547836a8016d2af8bbc05dd14b2720547bf506c0364bf5d6e5e6631be73 WatchSource:0}: Error finding container da703547836a8016d2af8bbc05dd14b2720547bf506c0364bf5d6e5e6631be73: Status 404 returned error can't find the container with id da703547836a8016d2af8bbc05dd14b2720547bf506c0364bf5d6e5e6631be73
Oct 30 19:14:17.290758 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:17.290659    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2cf46"]
Oct 30 19:14:17.293133 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:17.293100    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:14:17.293254 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:17.293166    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:14:17.293254 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:17.293210    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:14:17.403297 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:14:17.403258    4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d49l" containerName="extractor"
Oct 30 19:14:17.769397 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:17.769350    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2cf46" event={"ID":"52a25631-2cfc-4155-a3c3-4a381f96bf8b","Type":"ContainerStarted","Data":"b520c9f45cc994c8fe9071082fad3019e873bfaac64681f87b690ca7d3c8714d"}
Oct 30 19:14:17.769876 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:17.769430    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2cf46" event={"ID":"52a25631-2cfc-4155-a3c3-4a381f96bf8b","Type":"ContainerStarted","Data":"da703547836a8016d2af8bbc05dd14b2720547bf506c0364bf5d6e5e6631be73"}
Oct 30 19:14:17.792989 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:17.792936    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2cf46" podStartSLOduration=1.792920311 podStartE2EDuration="1.792920311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:14:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:14:17.790465273 +0000 UTC m=+2280.416135272" watchObservedRunningTime="2026-10-30 19:14:17.792920311 +0000 UTC m=+2280.418590274"
Oct 30 19:14:17.952523 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:17.945456    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2cf46"
Oct 30 19:14:18.073919 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:18.073890    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ecc266-2spmx"
Oct 30 19:14:18.165256 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:18.165222    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/446c488b-eaeb-49c5-b99e-117d7aae32a4-kube-api-access-spszq\" (UniqueName: \"kubernetes.io/projected/446c488b-eaeb-49c5-b99e-117d7aae32a4-kube-api-access-spszq\") pod \"446c488b-eaeb-49c5-b99e-117d7aae32a4\" (UID: \"446c488b-eaeb-49c5-b99e-117d7aae32a4\") "
Oct 30 19:14:18.165458 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:18.165352    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/446c488b-eaeb-49c5-b99e-117d7aae32a4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/446c488b-eaeb-49c5-b99e-117d7aae32a4-maas-api-openshift-ca\") pod \"446c488b-eaeb-49c5-b99e-117d7aae32a4\" (UID: \"446c488b-eaeb-49c5-b99e-117d7aae32a4\") "
Oct 30 19:14:18.165535 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:18.165463    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/446c488b-eaeb-49c5-b99e-117d7aae32a4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/446c488b-eaeb-49c5-b99e-117d7aae32a4-maas-api-xks-ca\") pod \"446c488b-eaeb-49c5-b99e-117d7aae32a4\" (UID: \"446c488b-eaeb-49c5-b99e-117d7aae32a4\") "
Oct 30 19:14:18.165791 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:18.165757    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/446c488b-eaeb-49c5-b99e-117d7aae32a4-maas-api-openshift-ca" pod "446c488b-eaeb-49c5-b99e-117d7aae32a4" (UID: "446c488b-eaeb-49c5-b99e-117d7aae32a4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:14:18.166187 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:18.166125    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/446c488b-eaeb-49c5-b99e-117d7aae32a4-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:18.168031 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:18.168004    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/446c488b-eaeb-49c5-b99e-117d7aae32a4-maas-api-xks-ca" pod "446c488b-eaeb-49c5-b99e-117d7aae32a4" (UID: "446c488b-eaeb-49c5-b99e-117d7aae32a4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:14:18.168260 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:18.168236    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/446c488b-eaeb-49c5-b99e-117d7aae32a4-kube-api-access-spszq" pod "446c488b-eaeb-49c5-b99e-117d7aae32a4" (UID: "446c488b-eaeb-49c5-b99e-117d7aae32a4"). InnerVolumeSpecName "kube-api-access-spszq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:14:18.267653 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:18.267617    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spszq\" (UniqueName: \"kubernetes.io/projected/446c488b-eaeb-49c5-b99e-117d7aae32a4-kube-api-access-spszq\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:18.267653 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:18.267648    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/446c488b-eaeb-49c5-b99e-117d7aae32a4-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:18.810640 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:18.810609    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ecc266-2spmx"
Oct 30 19:14:18.811919 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:18.811829    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ecc266-2spmx" event={"ID":"446c488b-eaeb-49c5-b99e-117d7aae32a4","Type":"ContainerDied","Data":"009be92b1e1a4589f9ed9949f8773fcf3e0784c1ac58c84c5136ed71548cd1cd"}
Oct 30 19:14:18.811919 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:18.811892    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="009be92b1e1a4589f9ed9949f8773fcf3e0784c1ac58c84c5136ed71548cd1cd"
Oct 30 19:14:18.940788 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:18.940756    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-d6f956-openshift-default-56888545df-2cf46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused" start-of-body=
Oct 30 19:14:18.940933 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:18.940809    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2cf46" podUID="52a25631-2cfc-4155-a3c3-4a381f96bf8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused"
Oct 30 19:14:19.951701 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:19.944157    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-d6f956-openshift-default-56888545df-2cf46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused" start-of-body=
Oct 30 19:14:19.951701 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:19.944220    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2cf46" podUID="52a25631-2cfc-4155-a3c3-4a381f96bf8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused"
Oct 30 19:14:20.939473 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:20.939437    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-d6f956-openshift-default-56888545df-2cf46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused" start-of-body=
Oct 30 19:14:20.939673 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:20.939498    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2cf46" podUID="52a25631-2cfc-4155-a3c3-4a381f96bf8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused"
Oct 30 19:14:21.943622 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:21.943556    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-d6f956-openshift-default-56888545df-2cf46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused" start-of-body=
Oct 30 19:14:21.944257 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:21.944221    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2cf46" podUID="52a25631-2cfc-4155-a3c3-4a381f96bf8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused"
Oct 30 19:14:22.335263 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:22.335183    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-1dd09d/payload-processing-external-model-e2e-extmod-a-1dd09d-5445l9g92"]
Oct 30 19:14:22.940708 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:22.940118    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-d6f956-openshift-default-56888545df-2cf46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused" start-of-body=
Oct 30 19:14:22.940708 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:22.940187    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2cf46" podUID="52a25631-2cfc-4155-a3c3-4a381f96bf8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused"
Oct 30 19:14:23.135749 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:23.135197    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1dd09d-5n2dz"]
Oct 30 19:14:23.145964 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:23.136443    4525 cpu_manager.go:414] "containerMap: removing container" podUID="446c488b-eaeb-49c5-b99e-117d7aae32a4" containerName="revoke-keys"
Oct 30 19:14:23.145964 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:23.136470    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="446c488b-eaeb-49c5-b99e-117d7aae32a4" containerName="revoke-keys"
Oct 30 19:14:23.145964 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:23.136705    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="446c488b-eaeb-49c5-b99e-117d7aae32a4" containerName="revoke-keys"
Oct 30 19:14:23.257570 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:23.257446    4525 scope.go:122] "RemoveContainer" containerID="f92e360403e1fbcf91c9c7b5a9948880a6617e8b5d1571c517ec4df81624912a"
Oct 30 19:14:23.344504 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:23.344471    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1dd09d-5n2dz"
Oct 30 19:14:23.349849 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:23.349816    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:14:23.356515 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:23.356481    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1dd09d-5n2dz"]
Oct 30 19:14:23.542758 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:23.541873    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94hcn\""
Oct 30 19:14:23.670543 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:23.635650    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88947136-4eba-457a-b60a-42ec0301efd8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-1dd09d-5n2dz\" (UID: \"88947136-4eba-457a-b60a-42ec0301efd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1dd09d-5n2dz"
Oct 30 19:14:23.670543 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:23.635763    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2w5b\" (UniqueName: \"kubernetes.io/projected/88947136-4eba-457a-b60a-42ec0301efd8-kube-api-access-c2w5b\") pod \"maas-api-revoke-keys-e2e-extmod-a-1dd09d-5n2dz\" (UID: \"88947136-4eba-457a-b60a-42ec0301efd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1dd09d-5n2dz"
Oct 30 19:14:23.670543 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:23.635955    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88947136-4eba-457a-b60a-42ec0301efd8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-1dd09d-5n2dz\" (UID: \"88947136-4eba-457a-b60a-42ec0301efd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1dd09d-5n2dz"
Oct 30 19:14:23.683951 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:23.674102    4525 scope.go:122] "RemoveContainer" containerID="fe25cbc30a6ea4051a77fd6b9acd67aaccee9d58d48e9b3cb7226e971f822317"
Oct 30 19:14:23.868890 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:23.868850    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88947136-4eba-457a-b60a-42ec0301efd8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-1dd09d-5n2dz\" (UID: \"88947136-4eba-457a-b60a-42ec0301efd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1dd09d-5n2dz"
Oct 30 19:14:23.869238 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:23.869217    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2w5b\" (UniqueName: \"kubernetes.io/projected/88947136-4eba-457a-b60a-42ec0301efd8-kube-api-access-c2w5b\") pod \"maas-api-revoke-keys-e2e-extmod-a-1dd09d-5n2dz\" (UID: \"88947136-4eba-457a-b60a-42ec0301efd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1dd09d-5n2dz"
Oct 30 19:14:23.869592 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:23.869569    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88947136-4eba-457a-b60a-42ec0301efd8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-1dd09d-5n2dz\" (UID: \"88947136-4eba-457a-b60a-42ec0301efd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1dd09d-5n2dz"
Oct 30 19:14:23.889718 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:23.871297    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88947136-4eba-457a-b60a-42ec0301efd8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-1dd09d-5n2dz\" (UID: \"88947136-4eba-457a-b60a-42ec0301efd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1dd09d-5n2dz"
Oct 30 19:14:23.948273 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:23.948237    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-d6f956-openshift-default-56888545df-2cf46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused" start-of-body=
Oct 30 19:14:24.207224 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:24.200150    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2cf46" podUID="52a25631-2cfc-4155-a3c3-4a381f96bf8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused"
Oct 30 19:14:24.207224 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:24.012135    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88947136-4eba-457a-b60a-42ec0301efd8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-1dd09d-5n2dz\" (UID: \"88947136-4eba-457a-b60a-42ec0301efd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1dd09d-5n2dz"
Oct 30 19:14:24.207224 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:24.201425    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2w5b\" (UniqueName: \"kubernetes.io/projected/88947136-4eba-457a-b60a-42ec0301efd8-kube-api-access-c2w5b\") pod \"maas-api-revoke-keys-e2e-extmod-a-1dd09d-5n2dz\" (UID: \"88947136-4eba-457a-b60a-42ec0301efd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1dd09d-5n2dz"
Oct 30 19:14:24.293537 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:24.293504    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1dd09d-5n2dz"
Oct 30 19:14:24.958211 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:24.958161    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-d6f956-openshift-default-56888545df-2cf46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused" start-of-body=
Oct 30 19:14:24.958558 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:24.958524    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2cf46" podUID="52a25631-2cfc-4155-a3c3-4a381f96bf8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused"
Oct 30 19:14:25.228376 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:25.227620    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-ecc266-68f9ddc76c-2whjr"]
Oct 30 19:14:25.237319 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:14:25.236947    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88947136_4eba_457a_b60a_42ec0301efd8.slice/crio-9d4939b3bc56f090beb0d641d25b9c3c9bfc926d27510f6ea11b1fc78e7180f6 WatchSource:0}: Error finding container 9d4939b3bc56f090beb0d641d25b9c3c9bfc926d27510f6ea11b1fc78e7180f6: Status 404 returned error can't find the container with id 9d4939b3bc56f090beb0d641d25b9c3c9bfc926d27510f6ea11b1fc78e7180f6
Oct 30 19:14:25.238667 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:25.238616    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1dd09d-5n2dz"]
Oct 30 19:14:25.321963 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:25.321927    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1dd09d-5n2dz" event={"ID":"88947136-4eba-457a-b60a-42ec0301efd8","Type":"ContainerStarted","Data":"9d4939b3bc56f090beb0d641d25b9c3c9bfc926d27510f6ea11b1fc78e7180f6"}
Oct 30 19:14:26.146556 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:26.146519    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2cf46"
Oct 30 19:14:26.147155 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:26.147127    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2cf46"
Oct 30 19:14:26.148346 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:26.148321    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2cf46"
Oct 30 19:14:26.335361 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:26.335261    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1dd09d-5n2dz" event={"ID":"88947136-4eba-457a-b60a-42ec0301efd8","Type":"ContainerStarted","Data":"f3dbc284034692b0eec16725be2b98ea915e756adba04966daa5814c3fb21e16"}
Oct 30 19:14:26.354934 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:26.354884    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1dd09d-5n2dz" podStartSLOduration=3.354869282 podStartE2EDuration="3.354869282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:14:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:14:26.350635696 +0000 UTC m=+2288.976305661" watchObservedRunningTime="2026-10-30 19:14:26.354869282 +0000 UTC m=+2288.980539245"
Oct 30 19:14:27.336192 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:27.336128    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-1dd09d/payload-processing-external-model-e2e-extmod-a-1dd09d-5445l9g92" podUID="0da000be-c37d-4fe0-87b5-9e15525341c8" containerName="payload-processing" containerID="cri-o://0abbdc2b2a07258ea865019f035bfafe47ff17ab8836f5a9679790bde1285e74" gracePeriod=25
Oct 30 19:14:27.347068 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:27.347037    4525 generic.go:356] "Generic (PLEG): container finished" podID="88947136-4eba-457a-b60a-42ec0301efd8" containerID="f3dbc284034692b0eec16725be2b98ea915e756adba04966daa5814c3fb21e16" exitCode=0
Oct 30 19:14:27.347206 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:27.347119    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1dd09d-5n2dz" event={"ID":"88947136-4eba-457a-b60a-42ec0301efd8","Type":"ContainerDied","Data":"f3dbc284034692b0eec16725be2b98ea915e756adba04966daa5814c3fb21e16"}
Oct 30 19:14:27.687754 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:27.687720    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-1dd09d/payload-processing-external-model-e2e-extmod-a-1dd09d-5445l9g92"
Oct 30 19:14:27.719970 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:27.719938    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-5hdmh"]
Oct 30 19:14:27.720724 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:27.720669    4525 cpu_manager.go:414] "containerMap: removing container" podUID="0da000be-c37d-4fe0-87b5-9e15525341c8" containerName="payload-processing"
Oct 30 19:14:27.720724 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:27.720706    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0da000be-c37d-4fe0-87b5-9e15525341c8" containerName="payload-processing"
Oct 30 19:14:27.720846 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:27.720837    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="0da000be-c37d-4fe0-87b5-9e15525341c8" containerName="payload-processing"
Oct 30 19:14:27.724537 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:27.724519    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-5hdmh"
Oct 30 19:14:27.744459 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:27.744429    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-5hdmh"]
Oct 30 19:14:27.837320 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:27.837286    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0da000be-c37d-4fe0-87b5-9e15525341c8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0da000be-c37d-4fe0-87b5-9e15525341c8-plugins-config-volume\") pod \"0da000be-c37d-4fe0-87b5-9e15525341c8\" (UID: \"0da000be-c37d-4fe0-87b5-9e15525341c8\") "
Oct 30 19:14:27.837533 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:27.837519    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0da000be-c37d-4fe0-87b5-9e15525341c8-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0da000be-c37d-4fe0-87b5-9e15525341c8-routing-overlay-volume\") pod \"0da000be-c37d-4fe0-87b5-9e15525341c8\" (UID: \"0da000be-c37d-4fe0-87b5-9e15525341c8\") "
Oct 30 19:14:27.837600 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:27.837586    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0da000be-c37d-4fe0-87b5-9e15525341c8-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0da000be-c37d-4fe0-87b5-9e15525341c8-provider-credentials\") pod \"0da000be-c37d-4fe0-87b5-9e15525341c8\" (UID: \"0da000be-c37d-4fe0-87b5-9e15525341c8\") "
Oct 30 19:14:27.837663 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:27.837619    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0da000be-c37d-4fe0-87b5-9e15525341c8-kube-api-access-wqs7b\" (UniqueName: \"kubernetes.io/projected/0da000be-c37d-4fe0-87b5-9e15525341c8-kube-api-access-wqs7b\") pod \"0da000be-c37d-4fe0-87b5-9e15525341c8\" (UID: \"0da000be-c37d-4fe0-87b5-9e15525341c8\") "
Oct 30 19:14:27.837906 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:27.837875    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0da000be-c37d-4fe0-87b5-9e15525341c8-plugins-config-volume" pod "0da000be-c37d-4fe0-87b5-9e15525341c8" (UID: "0da000be-c37d-4fe0-87b5-9e15525341c8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:14:27.837906 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:27.837888    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0da000be-c37d-4fe0-87b5-9e15525341c8-routing-overlay-volume" pod "0da000be-c37d-4fe0-87b5-9e15525341c8" (UID: "0da000be-c37d-4fe0-87b5-9e15525341c8"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:14:27.838290 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:27.838263    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7e914c3-d828-407a-bafb-f71a07d872c1-workload-certs\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-5hdmh\" (UID: \"f7e914c3-d828-407a-bafb-f71a07d872c1\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-5hdmh"
Oct 30 19:14:27.838386 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:27.838305    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7e914c3-d828-407a-bafb-f71a07d872c1-istio-podinfo\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-5hdmh\" (UID: \"f7e914c3-d828-407a-bafb-f71a07d872c1\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-5hdmh"
Oct 30 19:14:27.838454 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:27.838420    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7e914c3-d828-407a-bafb-f71a07d872c1-istio-data\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-5hdmh\" (UID: \"f7e914c3-d828-407a-bafb-f71a07d872c1\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-5hdmh"
Oct 30 19:14:27.838651 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:27.838630    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7e914c3-d828-407a-bafb-f71a07d872c1-istiod-ca-cert\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-5hdmh\" (UID: \"f7e914c3-d828-407a-bafb-f71a07d872c1\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-5hdmh"
Oct 30 19:14:27.838868 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:27.838842    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7e914c3-d828-407a-bafb-f71a07d872c1-workload-socket\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-5hdmh\" (UID: \"f7e914c3-d828-407a-bafb-f71a07d872c1\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-5hdmh"
Oct 30 19:14:27.838957 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:27.838939    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7e914c3-d828-407a-bafb-f71a07d872c1-istio-envoy\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-5hdmh\" (UID: \"f7e914c3-d828-407a-bafb-f71a07d872c1\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-5hdmh"
Oct 30 19:14:27.839010 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:27.838997    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7e914c3-d828-407a-bafb-f71a07d872c1-credential-socket\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-5hdmh\" (UID: \"f7e914c3-d828-407a-bafb-f71a07d872c1\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-5hdmh"
Oct 30 19:14:27.839143 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:27.839119    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8klf6\" (UniqueName: \"kubernetes.io/projected/f7e914c3-d828-407a-bafb-f71a07d872c1-kube-api-access-8klf6\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-5hdmh\" (UID: \"f7e914c3-d828-407a-bafb-f71a07d872c1\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-5hdmh"
Oct 30 19:14:27.839258 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:27.839241    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7e914c3-d828-407a-bafb-f71a07d872c1-istio-token\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-5hdmh\" (UID: \"f7e914c3-d828-407a-bafb-f71a07d872c1\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-5hdmh"
Oct 30 19:14:27.839451 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:27.839434    4525 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0da000be-c37d-4fe0-87b5-9e15525341c8-routing-overlay-volume\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:27.839506 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:27.839460    4525 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0da000be-c37d-4fe0-87b5-9e15525341c8-plugins-config-volume\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:27.840291 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:27.840267    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0da000be-c37d-4fe0-87b5-9e15525341c8-provider-credentials" pod "0da000be-c37d-4fe0-87b5-9e15525341c8" (UID: "0da000be-c37d-4fe0-87b5-9e15525341c8"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:14:27.840413 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:27.840399    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0da000be-c37d-4fe0-87b5-9e15525341c8-kube-api-access-wqs7b" pod "0da000be-c37d-4fe0-87b5-9e15525341c8" (UID: "0da000be-c37d-4fe0-87b5-9e15525341c8"). InnerVolumeSpecName "kube-api-access-wqs7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:14:27.941044 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:27.940958    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7e914c3-d828-407a-bafb-f71a07d872c1-credential-socket\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-5hdmh\" (UID: \"f7e914c3-d828-407a-bafb-f71a07d872c1\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-5hdmh"
Oct 30 19:14:27.941044 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:27.941011    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8klf6\" (UniqueName: \"kubernetes.io/projected/f7e914c3-d828-407a-bafb-f71a07d872c1-kube-api-access-8klf6\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-5hdmh\" (UID: \"f7e914c3-d828-407a-bafb-f71a07d872c1\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-5hdmh"
Oct 30 19:14:27.941277 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:27.941067    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7e914c3-d828-407a-bafb-f71a07d872c1-istio-token\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-5hdmh\" (UID: \"f7e914c3-d828-407a-bafb-f71a07d872c1\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-5hdmh"
Oct 30 19:14:27.941277 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:27.941157    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7e914c3-d828-407a-bafb-f71a07d872c1-workload-certs\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-5hdmh\" (UID: \"f7e914c3-d828-407a-bafb-f71a07d872c1\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-5hdmh"
Oct 30 19:14:27.941277 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:27.941177    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7e914c3-d828-407a-bafb-f71a07d872c1-istio-podinfo\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-5hdmh\" (UID: \"f7e914c3-d828-407a-bafb-f71a07d872c1\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-5hdmh"
Oct 30 19:14:27.941277 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:27.941236    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7e914c3-d828-407a-bafb-f71a07d872c1-istio-data\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-5hdmh\" (UID: \"f7e914c3-d828-407a-bafb-f71a07d872c1\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-5hdmh"
Oct 30 19:14:27.941498 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:27.941299    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7e914c3-d828-407a-bafb-f71a07d872c1-istiod-ca-cert\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-5hdmh\" (UID: \"f7e914c3-d828-407a-bafb-f71a07d872c1\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-5hdmh"
Oct 30 19:14:27.941498 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:27.941374    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7e914c3-d828-407a-bafb-f71a07d872c1-credential-socket\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-5hdmh\" (UID: \"f7e914c3-d828-407a-bafb-f71a07d872c1\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-5hdmh"
Oct 30 19:14:27.941498 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:27.941388    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7e914c3-d828-407a-bafb-f71a07d872c1-workload-socket\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-5hdmh\" (UID: \"f7e914c3-d828-407a-bafb-f71a07d872c1\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-5hdmh"
Oct 30 19:14:27.941498 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:27.941429    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7e914c3-d828-407a-bafb-f71a07d872c1-workload-certs\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-5hdmh\" (UID: \"f7e914c3-d828-407a-bafb-f71a07d872c1\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-5hdmh"
Oct 30 19:14:27.941498 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:27.941482    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7e914c3-d828-407a-bafb-f71a07d872c1-istio-envoy\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-5hdmh\" (UID: \"f7e914c3-d828-407a-bafb-f71a07d872c1\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-5hdmh"
Oct 30 19:14:27.941813 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:27.941620    4525 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0da000be-c37d-4fe0-87b5-9e15525341c8-provider-credentials\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:27.941813 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:27.941638    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqs7b\" (UniqueName: \"kubernetes.io/projected/0da000be-c37d-4fe0-87b5-9e15525341c8-kube-api-access-wqs7b\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:27.941813 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:27.941696    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7e914c3-d828-407a-bafb-f71a07d872c1-istio-data\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-5hdmh\" (UID: \"f7e914c3-d828-407a-bafb-f71a07d872c1\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-5hdmh"
Oct 30 19:14:27.941813 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:27.941779    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7e914c3-d828-407a-bafb-f71a07d872c1-workload-socket\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-5hdmh\" (UID: \"f7e914c3-d828-407a-bafb-f71a07d872c1\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-5hdmh"
Oct 30 19:14:27.942072 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:27.942053    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7e914c3-d828-407a-bafb-f71a07d872c1-istiod-ca-cert\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-5hdmh\" (UID: \"f7e914c3-d828-407a-bafb-f71a07d872c1\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-5hdmh"
Oct 30 19:14:27.943843 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:27.943821    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7e914c3-d828-407a-bafb-f71a07d872c1-istio-envoy\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-5hdmh\" (UID: \"f7e914c3-d828-407a-bafb-f71a07d872c1\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-5hdmh"
Oct 30 19:14:27.943955 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:27.943909    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7e914c3-d828-407a-bafb-f71a07d872c1-istio-podinfo\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-5hdmh\" (UID: \"f7e914c3-d828-407a-bafb-f71a07d872c1\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-5hdmh"
Oct 30 19:14:27.949360 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:27.949335    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7e914c3-d828-407a-bafb-f71a07d872c1-istio-token\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-5hdmh\" (UID: \"f7e914c3-d828-407a-bafb-f71a07d872c1\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-5hdmh"
Oct 30 19:14:27.949457 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:27.949366    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8klf6\" (UniqueName: \"kubernetes.io/projected/f7e914c3-d828-407a-bafb-f71a07d872c1-kube-api-access-8klf6\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-5hdmh\" (UID: \"f7e914c3-d828-407a-bafb-f71a07d872c1\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-5hdmh"
Oct 30 19:14:28.064710 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:28.063646    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-5hdmh"
Oct 30 19:14:28.357409 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:28.357376    4525 generic.go:356] "Generic (PLEG): container finished" podID="0da000be-c37d-4fe0-87b5-9e15525341c8" containerID="0abbdc2b2a07258ea865019f035bfafe47ff17ab8836f5a9679790bde1285e74" exitCode=0
Oct 30 19:14:28.357860 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:28.357437    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-1dd09d/payload-processing-external-model-e2e-extmod-a-1dd09d-5445l9g92"
Oct 30 19:14:28.357860 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:28.357464    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-1dd09d/payload-processing-external-model-e2e-extmod-a-1dd09d-5445l9g92" event={"ID":"0da000be-c37d-4fe0-87b5-9e15525341c8","Type":"ContainerDied","Data":"0abbdc2b2a07258ea865019f035bfafe47ff17ab8836f5a9679790bde1285e74"}
Oct 30 19:14:28.357860 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:28.357542    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-1dd09d/payload-processing-external-model-e2e-extmod-a-1dd09d-5445l9g92" event={"ID":"0da000be-c37d-4fe0-87b5-9e15525341c8","Type":"ContainerDied","Data":"7f9ce19ca9bdb416428d8996176dcac24e343e3a387106b282b501d9e7e08a09"}
Oct 30 19:14:28.357860 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:28.357564    4525 scope.go:122] "RemoveContainer" containerID="0abbdc2b2a07258ea865019f035bfafe47ff17ab8836f5a9679790bde1285e74"
Oct 30 19:14:28.371849 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:28.371829    4525 scope.go:122] "RemoveContainer" containerID="0abbdc2b2a07258ea865019f035bfafe47ff17ab8836f5a9679790bde1285e74"
Oct 30 19:14:28.372548 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:14:28.372527    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0abbdc2b2a07258ea865019f035bfafe47ff17ab8836f5a9679790bde1285e74\": container with ID starting with 0abbdc2b2a07258ea865019f035bfafe47ff17ab8836f5a9679790bde1285e74 not found: ID does not exist" containerID="0abbdc2b2a07258ea865019f035bfafe47ff17ab8836f5a9679790bde1285e74"
Oct 30 19:14:28.372548 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:28.372555    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0abbdc2b2a07258ea865019f035bfafe47ff17ab8836f5a9679790bde1285e74"} err="failed to get container status \"0abbdc2b2a07258ea865019f035bfafe47ff17ab8836f5a9679790bde1285e74\": rpc error: code = NotFound desc = could not find container \"0abbdc2b2a07258ea865019f035bfafe47ff17ab8836f5a9679790bde1285e74\": container with ID starting with 0abbdc2b2a07258ea865019f035bfafe47ff17ab8836f5a9679790bde1285e74 not found: ID does not exist"
Oct 30 19:14:28.386616 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:28.386573    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-1dd09d/payload-processing-external-model-e2e-extmod-a-1dd09d-5445l9g92"]
Oct 30 19:14:28.393341 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:28.393312    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-1dd09d/payload-processing-external-model-e2e-extmod-a-1dd09d-5445l9g92"]
Oct 30 19:14:28.416839 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:28.416814    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-5hdmh"]
Oct 30 19:14:28.417107 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:14:28.417083    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7e914c3_d828_407a_bafb_f71a07d872c1.slice/crio-43144da64b6a555285cc325fad5d4d5bfb434ddd04144d8545afc3477cdd802e WatchSource:0}: Error finding container 43144da64b6a555285cc325fad5d4d5bfb434ddd04144d8545afc3477cdd802e: Status 404 returned error can't find the container with id 43144da64b6a555285cc325fad5d4d5bfb434ddd04144d8545afc3477cdd802e
Oct 30 19:14:28.420698 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:28.420648    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:14:28.420784 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:28.420746    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:14:28.420784 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:28.420779    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:14:28.537386 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:28.537363    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1dd09d-5n2dz"
Oct 30 19:14:28.662221 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:28.662188    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88947136-4eba-457a-b60a-42ec0301efd8-kube-api-access-c2w5b\" (UniqueName: \"kubernetes.io/projected/88947136-4eba-457a-b60a-42ec0301efd8-kube-api-access-c2w5b\") pod \"88947136-4eba-457a-b60a-42ec0301efd8\" (UID: \"88947136-4eba-457a-b60a-42ec0301efd8\") "
Oct 30 19:14:28.662415 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:28.662306    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88947136-4eba-457a-b60a-42ec0301efd8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88947136-4eba-457a-b60a-42ec0301efd8-maas-api-xks-ca\") pod \"88947136-4eba-457a-b60a-42ec0301efd8\" (UID: \"88947136-4eba-457a-b60a-42ec0301efd8\") "
Oct 30 19:14:28.662538 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:28.662516    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88947136-4eba-457a-b60a-42ec0301efd8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88947136-4eba-457a-b60a-42ec0301efd8-maas-api-openshift-ca\") pod \"88947136-4eba-457a-b60a-42ec0301efd8\" (UID: \"88947136-4eba-457a-b60a-42ec0301efd8\") "
Oct 30 19:14:28.663003 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:28.662973    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88947136-4eba-457a-b60a-42ec0301efd8-maas-api-openshift-ca" pod "88947136-4eba-457a-b60a-42ec0301efd8" (UID: "88947136-4eba-457a-b60a-42ec0301efd8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:14:28.663450 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:28.663427    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88947136-4eba-457a-b60a-42ec0301efd8-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:28.678087 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:28.676759    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88947136-4eba-457a-b60a-42ec0301efd8-kube-api-access-c2w5b" pod "88947136-4eba-457a-b60a-42ec0301efd8" (UID: "88947136-4eba-457a-b60a-42ec0301efd8"). InnerVolumeSpecName "kube-api-access-c2w5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:14:28.678087 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:28.677504    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88947136-4eba-457a-b60a-42ec0301efd8-maas-api-xks-ca" pod "88947136-4eba-457a-b60a-42ec0301efd8" (UID: "88947136-4eba-457a-b60a-42ec0301efd8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:14:28.822717 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:28.786772    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2w5b\" (UniqueName: \"kubernetes.io/projected/88947136-4eba-457a-b60a-42ec0301efd8-kube-api-access-c2w5b\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:28.822717 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:28.787950    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88947136-4eba-457a-b60a-42ec0301efd8-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:29.440706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:29.439660    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-5hdmh" event={"ID":"f7e914c3-d828-407a-bafb-f71a07d872c1","Type":"ContainerStarted","Data":"6c6e31cf364ed20bf0047898b173cf5e5f6e37039f70c57f5d094d2449541a4e"}
Oct 30 19:14:29.440706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:29.439760    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-5hdmh" event={"ID":"f7e914c3-d828-407a-bafb-f71a07d872c1","Type":"ContainerStarted","Data":"43144da64b6a555285cc325fad5d4d5bfb434ddd04144d8545afc3477cdd802e"}
Oct 30 19:14:29.442852 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:29.442826    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1dd09d-5n2dz"
Oct 30 19:14:29.443152 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:29.442866    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1dd09d-5n2dz" event={"ID":"88947136-4eba-457a-b60a-42ec0301efd8","Type":"ContainerDied","Data":"9d4939b3bc56f090beb0d641d25b9c3c9bfc926d27510f6ea11b1fc78e7180f6"}
Oct 30 19:14:29.443152 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:29.443131    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d4939b3bc56f090beb0d641d25b9c3c9bfc926d27510f6ea11b1fc78e7180f6"
Oct 30 19:14:29.469638 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:29.469580    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-5hdmh" podStartSLOduration=13.469562118 podStartE2EDuration="13.469562118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:14:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:14:29.462175066 +0000 UTC m=+2292.087845030" watchObservedRunningTime="2026-10-30 19:14:29.469562118 +0000 UTC m=+2292.095232083"
Oct 30 19:14:29.546564 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:29.546527    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0da000be-c37d-4fe0-87b5-9e15525341c8" path="/var/lib/kubelet/pods/0da000be-c37d-4fe0-87b5-9e15525341c8/volumes"
Oct 30 19:14:30.078796 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:30.069135    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-d6f956-openshift-default-56888545df-5hdmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused" start-of-body=
Oct 30 19:14:30.078796 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:30.069301    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-5hdmh" podUID="f7e914c3-d828-407a-bafb-f71a07d872c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused"
Oct 30 19:14:30.078796 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:30.069741    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-5hdmh"
Oct 30 19:14:30.228879 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:30.228827    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ecc266-68f9ddc76c-2whjr" podUID="1eb4c854-2d00-4016-a851-33b9908bc34d" containerName="payload-pre-processing" containerID="cri-o://2e0795455b95992a78b03ce1175876ce8eaeb573ca5133c6a04440604cd09345" gracePeriod=25
Oct 30 19:14:30.459848 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:30.459812    4525 generic.go:356] "Generic (PLEG): container finished" podID="1eb4c854-2d00-4016-a851-33b9908bc34d" containerID="2e0795455b95992a78b03ce1175876ce8eaeb573ca5133c6a04440604cd09345" exitCode=0
Oct 30 19:14:30.461356 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:30.461323    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ecc266-68f9ddc76c-2whjr" event={"ID":"1eb4c854-2d00-4016-a851-33b9908bc34d","Type":"ContainerDied","Data":"2e0795455b95992a78b03ce1175876ce8eaeb573ca5133c6a04440604cd09345"}
Oct 30 19:14:30.632206 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:30.632182    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ecc266-68f9ddc76c-2whjr"
Oct 30 19:14:30.728964 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:30.728500    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1eb4c854-2d00-4016-a851-33b9908bc34d-kube-api-access-qgcpj\" (UniqueName: \"kubernetes.io/projected/1eb4c854-2d00-4016-a851-33b9908bc34d-kube-api-access-qgcpj\") pod \"1eb4c854-2d00-4016-a851-33b9908bc34d\" (UID: \"1eb4c854-2d00-4016-a851-33b9908bc34d\") "
Oct 30 19:14:30.728964 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:30.728721    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1eb4c854-2d00-4016-a851-33b9908bc34d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1eb4c854-2d00-4016-a851-33b9908bc34d-plugins-config-volume\") pod \"1eb4c854-2d00-4016-a851-33b9908bc34d\" (UID: \"1eb4c854-2d00-4016-a851-33b9908bc34d\") "
Oct 30 19:14:30.729769 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:30.729739    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1eb4c854-2d00-4016-a851-33b9908bc34d-plugins-config-volume" pod "1eb4c854-2d00-4016-a851-33b9908bc34d" (UID: "1eb4c854-2d00-4016-a851-33b9908bc34d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:14:30.733112 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:30.733087    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1eb4c854-2d00-4016-a851-33b9908bc34d-kube-api-access-qgcpj" pod "1eb4c854-2d00-4016-a851-33b9908bc34d" (UID: "1eb4c854-2d00-4016-a851-33b9908bc34d"). InnerVolumeSpecName "kube-api-access-qgcpj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:14:30.898857 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:30.893381    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgcpj\" (UniqueName: \"kubernetes.io/projected/1eb4c854-2d00-4016-a851-33b9908bc34d-kube-api-access-qgcpj\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:30.898857 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:30.893419    4525 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1eb4c854-2d00-4016-a851-33b9908bc34d-plugins-config-volume\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:31.082218 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:31.078808    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-d6f956-openshift-default-56888545df-5hdmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused" start-of-body=
Oct 30 19:14:31.082218 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:31.078888    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-5hdmh" podUID="f7e914c3-d828-407a-bafb-f71a07d872c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused"
Oct 30 19:14:31.470948 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:31.470910    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ecc266-68f9ddc76c-2whjr"
Oct 30 19:14:31.471462 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:31.470946    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ecc266-68f9ddc76c-2whjr" event={"ID":"1eb4c854-2d00-4016-a851-33b9908bc34d","Type":"ContainerDied","Data":"3eafd822163d8e767937032d8262a930baaaed7ef6646c320aed07d4da731517"}
Oct 30 19:14:31.471462 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:31.471016    4525 scope.go:122] "RemoveContainer" containerID="2e0795455b95992a78b03ce1175876ce8eaeb573ca5133c6a04440604cd09345"
Oct 30 19:14:31.523428 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:31.523383    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-ecc266-68f9ddc76c-2whjr"]
Oct 30 19:14:31.636747 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:31.634842    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-ecc266-68f9ddc76c-2whjr"]
Oct 30 19:14:32.064389 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:32.064353    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-d6f956-openshift-default-56888545df-5hdmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused" start-of-body=
Oct 30 19:14:32.064570 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:32.064419    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-5hdmh" podUID="f7e914c3-d828-407a-bafb-f71a07d872c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused"
Oct 30 19:14:33.066827 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:33.065140    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-d6f956-openshift-default-56888545df-5hdmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused" start-of-body=
Oct 30 19:14:33.066827 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:33.065217    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-5hdmh" podUID="f7e914c3-d828-407a-bafb-f71a07d872c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused"
Oct 30 19:14:33.707728 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:33.707625    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1eb4c854-2d00-4016-a851-33b9908bc34d" path="/var/lib/kubelet/pods/1eb4c854-2d00-4016-a851-33b9908bc34d/volumes"
Oct 30 19:14:33.975780 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:33.964471    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-ecc266-559cbb854f-8wvqg"]
Oct 30 19:14:34.175249 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:34.066297    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-d6f956-openshift-default-56888545df-5hdmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused" start-of-body=
Oct 30 19:14:34.175249 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:34.153951    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-5hdmh" podUID="f7e914c3-d828-407a-bafb-f71a07d872c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused"
Oct 30 19:14:34.175249 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:34.154053    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ecc266-559cbb854f-8wvqg" podUID="bc021748-e220-42d9-9e39-029115dade4b" containerName="maas-api" containerID="cri-o://255e3f0fb10a28ee3a6022a6d455dde70711138dccc92715cedec8617113d664" gracePeriod=30
Oct 30 19:14:34.542817 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:34.542773    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts" podUID="be6d4969-285b-4da0-883e-4847f2cbf3af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:14:34.600970 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:34.591424    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj" podUID="84c99c49-1d60-4176-bda4-cad29c7d801c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:14:34.662032 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:34.661972    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw" podUID="ac6852fd-01be-4d69-8c9b-0c591e50b889" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:14:34.759717 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:34.757964    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb" podUID="12ebc8b9-c794-47b5-b87b-a63c4bd993e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:14:34.847199 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:34.847138    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv" podUID="595230e3-55f3-43eb-9a7b-6d89250af9b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 19:14:34.871821 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:34.871792    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ecc266-2spmx"]
Oct 30 19:14:34.898704 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:34.881744    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ecc266-2spmx"]
Oct 30 19:14:34.927299 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:14:34.927266    4525 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc021748_e220_42d9_9e39_029115dade4b.slice/crio-255e3f0fb10a28ee3a6022a6d455dde70711138dccc92715cedec8617113d664.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 19:14:35.068591 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:35.068558    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-d6f956-openshift-default-56888545df-5hdmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused" start-of-body=
Oct 30 19:14:35.068878 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:35.068621    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-5hdmh" podUID="f7e914c3-d828-407a-bafb-f71a07d872c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused"
Oct 30 19:14:35.159219 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:35.159181    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ecc266-559cbb854f-8wvqg"
Oct 30 19:14:35.242316 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:35.242279    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-ftq7b"]
Oct 30 19:14:35.243048 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:35.243029    4525 cpu_manager.go:414] "containerMap: removing container" podUID="88947136-4eba-457a-b60a-42ec0301efd8" containerName="revoke-keys"
Oct 30 19:14:35.243159 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:35.243050    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88947136-4eba-457a-b60a-42ec0301efd8" containerName="revoke-keys"
Oct 30 19:14:35.243159 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:35.243070    4525 cpu_manager.go:414] "containerMap: removing container" podUID="1eb4c854-2d00-4016-a851-33b9908bc34d" containerName="payload-pre-processing"
Oct 30 19:14:35.243159 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:35.243076    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1eb4c854-2d00-4016-a851-33b9908bc34d" containerName="payload-pre-processing"
Oct 30 19:14:35.243159 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:35.243096    4525 cpu_manager.go:414] "containerMap: removing container" podUID="bc021748-e220-42d9-9e39-029115dade4b" containerName="maas-api"
Oct 30 19:14:35.243159 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:35.243101    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc021748-e220-42d9-9e39-029115dade4b" containerName="maas-api"
Oct 30 19:14:35.243427 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:35.243204    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="88947136-4eba-457a-b60a-42ec0301efd8" containerName="revoke-keys"
Oct 30 19:14:35.243427 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:35.243217    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="1eb4c854-2d00-4016-a851-33b9908bc34d" containerName="payload-pre-processing"
Oct 30 19:14:35.243427 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:35.243227    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc021748-e220-42d9-9e39-029115dade4b" containerName="maas-api"
Oct 30 19:14:35.247457 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:35.247432    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-ftq7b"
Oct 30 19:14:35.250263 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:35.250236    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-ftq7b"]
Oct 30 19:14:35.265624 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:35.265588    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc021748-e220-42d9-9e39-029115dade4b-kube-api-access-c6szt\" (UniqueName: \"kubernetes.io/projected/bc021748-e220-42d9-9e39-029115dade4b-kube-api-access-c6szt\") pod \"bc021748-e220-42d9-9e39-029115dade4b\" (UID: \"bc021748-e220-42d9-9e39-029115dade4b\") "
Oct 30 19:14:35.265847 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:35.265831    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc021748-e220-42d9-9e39-029115dade4b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc021748-e220-42d9-9e39-029115dade4b-maas-api-tls\") pod \"bc021748-e220-42d9-9e39-029115dade4b\" (UID: \"bc021748-e220-42d9-9e39-029115dade4b\") "
Oct 30 19:14:35.265956 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:35.265931    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc021748-e220-42d9-9e39-029115dade4b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc021748-e220-42d9-9e39-029115dade4b-openshift-service-ca\") pod \"bc021748-e220-42d9-9e39-029115dade4b\" (UID: \"bc021748-e220-42d9-9e39-029115dade4b\") "
Oct 30 19:14:35.266096 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:35.265993    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc021748-e220-42d9-9e39-029115dade4b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc021748-e220-42d9-9e39-029115dade4b-metrics-certs\") pod \"bc021748-e220-42d9-9e39-029115dade4b\" (UID: \"bc021748-e220-42d9-9e39-029115dade4b\") "
Oct 30 19:14:35.266385 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:35.266345    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc021748-e220-42d9-9e39-029115dade4b-openshift-service-ca" pod "bc021748-e220-42d9-9e39-029115dade4b" (UID: "bc021748-e220-42d9-9e39-029115dade4b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:14:35.266517 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:35.266432    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4cbde58-0ec2-4f0d-af09-048853d07d63-workload-socket\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-ftq7b\" (UID: \"e4cbde58-0ec2-4f0d-af09-048853d07d63\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-ftq7b"
Oct 30 19:14:35.266609 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:35.266516    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4cbde58-0ec2-4f0d-af09-048853d07d63-istio-envoy\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-ftq7b\" (UID: \"e4cbde58-0ec2-4f0d-af09-048853d07d63\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-ftq7b"
Oct 30 19:14:35.267527 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:35.266781    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4cbde58-0ec2-4f0d-af09-048853d07d63-istio-data\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-ftq7b\" (UID: \"e4cbde58-0ec2-4f0d-af09-048853d07d63\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-ftq7b"
Oct 30 19:14:35.267527 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:35.266932    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4cbde58-0ec2-4f0d-af09-048853d07d63-workload-certs\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-ftq7b\" (UID: \"e4cbde58-0ec2-4f0d-af09-048853d07d63\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-ftq7b"
Oct 30 19:14:35.267527 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:35.266984    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4cbde58-0ec2-4f0d-af09-048853d07d63-istio-podinfo\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-ftq7b\" (UID: \"e4cbde58-0ec2-4f0d-af09-048853d07d63\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-ftq7b"
Oct 30 19:14:35.267527 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:35.267042    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks47v\" (UniqueName: \"kubernetes.io/projected/e4cbde58-0ec2-4f0d-af09-048853d07d63-kube-api-access-ks47v\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-ftq7b\" (UID: \"e4cbde58-0ec2-4f0d-af09-048853d07d63\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-ftq7b"
Oct 30 19:14:35.267527 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:35.267164    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4cbde58-0ec2-4f0d-af09-048853d07d63-istio-token\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-ftq7b\" (UID: \"e4cbde58-0ec2-4f0d-af09-048853d07d63\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-ftq7b"
Oct 30 19:14:35.267527 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:35.267195    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4cbde58-0ec2-4f0d-af09-048853d07d63-istiod-ca-cert\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-ftq7b\" (UID: \"e4cbde58-0ec2-4f0d-af09-048853d07d63\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-ftq7b"
Oct 30 19:14:35.267527 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:35.267303    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4cbde58-0ec2-4f0d-af09-048853d07d63-credential-socket\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-ftq7b\" (UID: \"e4cbde58-0ec2-4f0d-af09-048853d07d63\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-ftq7b"
Oct 30 19:14:35.267527 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:35.267437    4525 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc021748-e220-42d9-9e39-029115dade4b-openshift-service-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:35.269149 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:35.269113    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc021748-e220-42d9-9e39-029115dade4b-kube-api-access-c6szt" pod "bc021748-e220-42d9-9e39-029115dade4b" (UID: "bc021748-e220-42d9-9e39-029115dade4b"). InnerVolumeSpecName "kube-api-access-c6szt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:14:35.269251 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:35.269192    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc021748-e220-42d9-9e39-029115dade4b-maas-api-tls" pod "bc021748-e220-42d9-9e39-029115dade4b" (UID: "bc021748-e220-42d9-9e39-029115dade4b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:14:35.269866 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:35.269841    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc021748-e220-42d9-9e39-029115dade4b-metrics-certs" pod "bc021748-e220-42d9-9e39-029115dade4b" (UID: "bc021748-e220-42d9-9e39-029115dade4b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:14:35.368610 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:35.368574    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4cbde58-0ec2-4f0d-af09-048853d07d63-istio-token\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-ftq7b\" (UID: \"e4cbde58-0ec2-4f0d-af09-048853d07d63\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-ftq7b"
Oct 30 19:14:35.368610 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:35.368615    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4cbde58-0ec2-4f0d-af09-048853d07d63-istiod-ca-cert\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-ftq7b\" (UID: \"e4cbde58-0ec2-4f0d-af09-048853d07d63\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-ftq7b"
Oct 30 19:14:35.368872 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:35.368706    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4cbde58-0ec2-4f0d-af09-048853d07d63-credential-socket\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-ftq7b\" (UID: \"e4cbde58-0ec2-4f0d-af09-048853d07d63\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-ftq7b"
Oct 30 19:14:35.368872 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:35.368785    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4cbde58-0ec2-4f0d-af09-048853d07d63-workload-socket\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-ftq7b\" (UID: \"e4cbde58-0ec2-4f0d-af09-048853d07d63\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-ftq7b"
Oct 30 19:14:35.368872 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:35.368808    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4cbde58-0ec2-4f0d-af09-048853d07d63-istio-envoy\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-ftq7b\" (UID: \"e4cbde58-0ec2-4f0d-af09-048853d07d63\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-ftq7b"
Oct 30 19:14:35.369039 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:35.368941    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4cbde58-0ec2-4f0d-af09-048853d07d63-istio-data\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-ftq7b\" (UID: \"e4cbde58-0ec2-4f0d-af09-048853d07d63\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-ftq7b"
Oct 30 19:14:35.369039 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:35.369018    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4cbde58-0ec2-4f0d-af09-048853d07d63-workload-certs\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-ftq7b\" (UID: \"e4cbde58-0ec2-4f0d-af09-048853d07d63\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-ftq7b"
Oct 30 19:14:35.369233 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:35.369059    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4cbde58-0ec2-4f0d-af09-048853d07d63-istio-podinfo\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-ftq7b\" (UID: \"e4cbde58-0ec2-4f0d-af09-048853d07d63\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-ftq7b"
Oct 30 19:14:35.369233 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:35.369110    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks47v\" (UniqueName: \"kubernetes.io/projected/e4cbde58-0ec2-4f0d-af09-048853d07d63-kube-api-access-ks47v\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-ftq7b\" (UID: \"e4cbde58-0ec2-4f0d-af09-048853d07d63\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-ftq7b"
Oct 30 19:14:35.369233 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:35.369137    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4cbde58-0ec2-4f0d-af09-048853d07d63-workload-socket\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-ftq7b\" (UID: \"e4cbde58-0ec2-4f0d-af09-048853d07d63\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-ftq7b"
Oct 30 19:14:35.369425 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:35.369301    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc021748-e220-42d9-9e39-029115dade4b-maas-api-tls\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:35.369425 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:35.369325    4525 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc021748-e220-42d9-9e39-029115dade4b-metrics-certs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:35.369425 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:35.369343    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6szt\" (UniqueName: \"kubernetes.io/projected/bc021748-e220-42d9-9e39-029115dade4b-kube-api-access-c6szt\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:35.369425 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:35.369355    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4cbde58-0ec2-4f0d-af09-048853d07d63-credential-socket\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-ftq7b\" (UID: \"e4cbde58-0ec2-4f0d-af09-048853d07d63\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-ftq7b"
Oct 30 19:14:35.369425 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:35.369386    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4cbde58-0ec2-4f0d-af09-048853d07d63-istio-data\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-ftq7b\" (UID: \"e4cbde58-0ec2-4f0d-af09-048853d07d63\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-ftq7b"
Oct 30 19:14:35.369657 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:35.369487    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4cbde58-0ec2-4f0d-af09-048853d07d63-istiod-ca-cert\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-ftq7b\" (UID: \"e4cbde58-0ec2-4f0d-af09-048853d07d63\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-ftq7b"
Oct 30 19:14:35.369657 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:35.369620    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4cbde58-0ec2-4f0d-af09-048853d07d63-workload-certs\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-ftq7b\" (UID: \"e4cbde58-0ec2-4f0d-af09-048853d07d63\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-ftq7b"
Oct 30 19:14:35.371575 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:35.371550    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4cbde58-0ec2-4f0d-af09-048853d07d63-istio-envoy\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-ftq7b\" (UID: \"e4cbde58-0ec2-4f0d-af09-048853d07d63\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-ftq7b"
Oct 30 19:14:35.371807 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:35.371790    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4cbde58-0ec2-4f0d-af09-048853d07d63-istio-podinfo\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-ftq7b\" (UID: \"e4cbde58-0ec2-4f0d-af09-048853d07d63\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-ftq7b"
Oct 30 19:14:35.402781 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:35.402236    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4cbde58-0ec2-4f0d-af09-048853d07d63-istio-token\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-ftq7b\" (UID: \"e4cbde58-0ec2-4f0d-af09-048853d07d63\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-ftq7b"
Oct 30 19:14:35.402781 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:35.402422    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks47v\" (UniqueName: \"kubernetes.io/projected/e4cbde58-0ec2-4f0d-af09-048853d07d63-kube-api-access-ks47v\") pod \"e2e-extmod-a-d6f956-openshift-default-56888545df-ftq7b\" (UID: \"e4cbde58-0ec2-4f0d-af09-048853d07d63\") " pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-ftq7b"
Oct 30 19:14:35.576343 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:35.570393    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-ftq7b"
Oct 30 19:14:35.640092 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:35.622884    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="446c488b-eaeb-49c5-b99e-117d7aae32a4" path="/var/lib/kubelet/pods/446c488b-eaeb-49c5-b99e-117d7aae32a4/volumes"
Oct 30 19:14:35.718876 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:35.718845    4525 generic.go:356] "Generic (PLEG): container finished" podID="bc021748-e220-42d9-9e39-029115dade4b" containerID="255e3f0fb10a28ee3a6022a6d455dde70711138dccc92715cedec8617113d664" exitCode=0
Oct 30 19:14:35.719045 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:35.718908    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ecc266-559cbb854f-8wvqg" event={"ID":"bc021748-e220-42d9-9e39-029115dade4b","Type":"ContainerDied","Data":"255e3f0fb10a28ee3a6022a6d455dde70711138dccc92715cedec8617113d664"}
Oct 30 19:14:35.719045 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:35.718925    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ecc266-559cbb854f-8wvqg"
Oct 30 19:14:35.719045 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:35.718971    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ecc266-559cbb854f-8wvqg" event={"ID":"bc021748-e220-42d9-9e39-029115dade4b","Type":"ContainerDied","Data":"a2a761a5adef1d2dff520551fa19ebec3d04806b531ac3a7448656ee8a116709"}
Oct 30 19:14:35.719045 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:35.718995    4525 scope.go:122] "RemoveContainer" containerID="255e3f0fb10a28ee3a6022a6d455dde70711138dccc92715cedec8617113d664"
Oct 30 19:14:35.731719 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:35.731697    4525 scope.go:122] "RemoveContainer" containerID="255e3f0fb10a28ee3a6022a6d455dde70711138dccc92715cedec8617113d664"
Oct 30 19:14:35.732014 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:14:35.731996    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"255e3f0fb10a28ee3a6022a6d455dde70711138dccc92715cedec8617113d664\": container with ID starting with 255e3f0fb10a28ee3a6022a6d455dde70711138dccc92715cedec8617113d664 not found: ID does not exist" containerID="255e3f0fb10a28ee3a6022a6d455dde70711138dccc92715cedec8617113d664"
Oct 30 19:14:35.732104 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:35.732023    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"255e3f0fb10a28ee3a6022a6d455dde70711138dccc92715cedec8617113d664"} err="failed to get container status \"255e3f0fb10a28ee3a6022a6d455dde70711138dccc92715cedec8617113d664\": rpc error: code = NotFound desc = could not find container \"255e3f0fb10a28ee3a6022a6d455dde70711138dccc92715cedec8617113d664\": container with ID starting with 255e3f0fb10a28ee3a6022a6d455dde70711138dccc92715cedec8617113d664 not found: ID does not exist"
Oct 30 19:14:35.758335 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:35.758285    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-ecc266-559cbb854f-8wvqg"]
Oct 30 19:14:35.764461 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:35.764418    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-ecc266-559cbb854f-8wvqg"]
Oct 30 19:14:36.003625 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:36.003511    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-ftq7b"]
Oct 30 19:14:36.005441 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:36.005404    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:14:36.005658 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:36.005637    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:14:36.005842 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:36.005822    4525 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 19:14:36.064266 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:36.064235    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-d6f956-openshift-default-56888545df-5hdmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused" start-of-body=
Oct 30 19:14:36.064388 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:36.064294    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-5hdmh" podUID="f7e914c3-d828-407a-bafb-f71a07d872c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused"
Oct 30 19:14:36.730805 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:36.730768    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-ftq7b" event={"ID":"e4cbde58-0ec2-4f0d-af09-048853d07d63","Type":"ContainerStarted","Data":"aa10dce9b5a287001fe76c4462e484a4bd2813cb8c89327e64aa82ea0959d438"}
Oct 30 19:14:36.731298 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:36.730829    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-ftq7b" event={"ID":"e4cbde58-0ec2-4f0d-af09-048853d07d63","Type":"ContainerStarted","Data":"2be7b29c1633f3d6b149e3984606eb558ec95e9083ff506e07ca312cf972b629"}
Oct 30 19:14:36.765231 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:36.765169    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-ftq7b" podStartSLOduration=5.7651538559999995 podStartE2EDuration="5.765153856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:14:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:14:36.752509028 +0000 UTC m=+2299.378178991" watchObservedRunningTime="2026-10-30 19:14:36.765153856 +0000 UTC m=+2299.390823820"
Oct 30 19:14:37.397703 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:37.396048    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-5hdmh"
Oct 30 19:14:37.397899 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:37.397818    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-5hdmh"
Oct 30 19:14:37.454177 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:37.454147    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-5hdmh"
Oct 30 19:14:37.598959 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:37.598926    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-d6f956-openshift-default-56888545df-ftq7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" start-of-body=
Oct 30 19:14:37.599227 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:37.599201    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-ftq7b" podUID="e4cbde58-0ec2-4f0d-af09-048853d07d63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused"
Oct 30 19:14:37.614151 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:37.614121    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc021748-e220-42d9-9e39-029115dade4b" path="/var/lib/kubelet/pods/bc021748-e220-42d9-9e39-029115dade4b/volumes"
Oct 30 19:14:37.616554 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:37.616531    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-ftq7b"
Oct 30 19:14:37.827185 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:37.827099    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1dd09d-5n2dz"]
Oct 30 19:14:37.913706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:37.906032    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1dd09d-5n2dz"]
Oct 30 19:14:38.547668 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:38.547637    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e51c10-97m92"]
Oct 30 19:14:38.555872 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:38.555839    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e51c10-97m92"]
Oct 30 19:14:38.556046 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:38.555922    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e51c10-97m92"
Oct 30 19:14:38.602061 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:38.602031    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94hcn\""
Oct 30 19:14:38.602421 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:38.602403    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:14:38.610155 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:38.603879    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-d6f956-openshift-default-56888545df-ftq7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" start-of-body=
Oct 30 19:14:38.610384 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:38.610352    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-ftq7b" podUID="e4cbde58-0ec2-4f0d-af09-048853d07d63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused"
Oct 30 19:14:38.665401 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:38.665366    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e395ae73-f217-48d0-a277-6798e8221162-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-e51c10-97m92\" (UID: \"e395ae73-f217-48d0-a277-6798e8221162\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e51c10-97m92"
Oct 30 19:14:38.665750 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:38.665729    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-th84w\" (UniqueName: \"kubernetes.io/projected/e395ae73-f217-48d0-a277-6798e8221162-kube-api-access-th84w\") pod \"maas-api-revoke-keys-e2e-extmod-a-e51c10-97m92\" (UID: \"e395ae73-f217-48d0-a277-6798e8221162\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e51c10-97m92"
Oct 30 19:14:38.665810 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:38.665769    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e395ae73-f217-48d0-a277-6798e8221162-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-e51c10-97m92\" (UID: \"e395ae73-f217-48d0-a277-6798e8221162\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e51c10-97m92"
Oct 30 19:14:38.766947 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:38.766917    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e395ae73-f217-48d0-a277-6798e8221162-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-e51c10-97m92\" (UID: \"e395ae73-f217-48d0-a277-6798e8221162\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e51c10-97m92"
Oct 30 19:14:38.767165 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:38.767148    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th84w\" (UniqueName: \"kubernetes.io/projected/e395ae73-f217-48d0-a277-6798e8221162-kube-api-access-th84w\") pod \"maas-api-revoke-keys-e2e-extmod-a-e51c10-97m92\" (UID: \"e395ae73-f217-48d0-a277-6798e8221162\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e51c10-97m92"
Oct 30 19:14:38.767217 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:38.767182    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e395ae73-f217-48d0-a277-6798e8221162-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-e51c10-97m92\" (UID: \"e395ae73-f217-48d0-a277-6798e8221162\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e51c10-97m92"
Oct 30 19:14:38.767845 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:38.767815    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e395ae73-f217-48d0-a277-6798e8221162-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-e51c10-97m92\" (UID: \"e395ae73-f217-48d0-a277-6798e8221162\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e51c10-97m92"
Oct 30 19:14:38.769858 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:38.769834    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e395ae73-f217-48d0-a277-6798e8221162-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-e51c10-97m92\" (UID: \"e395ae73-f217-48d0-a277-6798e8221162\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e51c10-97m92"
Oct 30 19:14:38.774863 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:38.774836    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-th84w\" (UniqueName: \"kubernetes.io/projected/e395ae73-f217-48d0-a277-6798e8221162-kube-api-access-th84w\") pod \"maas-api-revoke-keys-e2e-extmod-a-e51c10-97m92\" (UID: \"e395ae73-f217-48d0-a277-6798e8221162\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e51c10-97m92"
Oct 30 19:14:38.907860 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:38.907807    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e51c10-97m92"
Oct 30 19:14:39.264810 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:14:39.264778    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode395ae73_f217_48d0_a277_6798e8221162.slice/crio-5779d3450713ecac5c41849e00a417e0f679c5a12eb23ca091ed464627cb5aa5 WatchSource:0}: Error finding container 5779d3450713ecac5c41849e00a417e0f679c5a12eb23ca091ed464627cb5aa5: Status 404 returned error can't find the container with id 5779d3450713ecac5c41849e00a417e0f679c5a12eb23ca091ed464627cb5aa5
Oct 30 19:14:39.267648 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:39.267618    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e51c10-97m92"]
Oct 30 19:14:39.553066 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:39.552985    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88947136-4eba-457a-b60a-42ec0301efd8" path="/var/lib/kubelet/pods/88947136-4eba-457a-b60a-42ec0301efd8/volumes"
Oct 30 19:14:39.598705 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:39.598075    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-d6f956-openshift-default-56888545df-ftq7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" start-of-body=
Oct 30 19:14:39.598705 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:39.598137    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-ftq7b" podUID="e4cbde58-0ec2-4f0d-af09-048853d07d63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused"
Oct 30 19:14:39.760472 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:39.760442    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e51c10-97m92" event={"ID":"e395ae73-f217-48d0-a277-6798e8221162","Type":"ContainerStarted","Data":"5779d3450713ecac5c41849e00a417e0f679c5a12eb23ca091ed464627cb5aa5"}
Oct 30 19:14:40.596708 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:40.593062    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-d6f956-openshift-default-56888545df-ftq7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" start-of-body=
Oct 30 19:14:40.596708 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:40.593139    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-ftq7b" podUID="e4cbde58-0ec2-4f0d-af09-048853d07d63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused"
Oct 30 19:14:40.849772 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:40.848386    4525 generic.go:356] "Generic (PLEG): container finished" podID="e395ae73-f217-48d0-a277-6798e8221162" containerID="83a7c9ebb36b4609ae037c541dd7122c5aec1087d3729e645fd747102f84d71b" exitCode=0
Oct 30 19:14:40.849772 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:40.848463    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e51c10-97m92" event={"ID":"e395ae73-f217-48d0-a277-6798e8221162","Type":"ContainerDied","Data":"83a7c9ebb36b4609ae037c541dd7122c5aec1087d3729e645fd747102f84d71b"}
Oct 30 19:14:41.602143 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:41.601954    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-d6f956-openshift-default-56888545df-ftq7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" start-of-body=
Oct 30 19:14:41.602143 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:41.602054    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-ftq7b" podUID="e4cbde58-0ec2-4f0d-af09-048853d07d63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused"
Oct 30 19:14:42.591154 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:42.591115    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-d6f956-openshift-default-56888545df-ftq7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" start-of-body=
Oct 30 19:14:42.607282 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:42.591434    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-ftq7b" podUID="e4cbde58-0ec2-4f0d-af09-048853d07d63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused"
Oct 30 19:14:43.409913 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:43.409880    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-baa504-hgbht"]
Oct 30 19:14:43.562053 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:43.560035    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr" podUID="82bbbf85-ed93-48d2-9f17-49ac1f31c0a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 30 19:14:43.571963 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:43.571911    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts" podUID="be6d4969-285b-4da0-883e-4847f2cbf3af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:14:43.574867 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:43.574825    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t" podUID="424fca14-6d8a-4663-86ec-0c91165f34ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:14:43.607965 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:43.606885    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-d6f956-openshift-default-56888545df-ftq7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" start-of-body=
Oct 30 19:14:43.607965 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:43.606962    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-ftq7b" podUID="e4cbde58-0ec2-4f0d-af09-048853d07d63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused"
Oct 30 19:14:43.627234 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:43.618039    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj" podUID="84c99c49-1d60-4176-bda4-cad29c7d801c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:14:43.627234 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:43.619285    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw" podUID="ac6852fd-01be-4d69-8c9b-0c591e50b889" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 19:14:43.675825 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:43.675792    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-baa504-hgbht"
Oct 30 19:14:43.841811 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:43.837881    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb" podUID="12ebc8b9-c794-47b5-b87b-a63c4bd993e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 19:14:44.091708 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:44.024808    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35138e86-4197-404b-83a3-374e5202931f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-baa504-hgbht\" (UID: \"35138e86-4197-404b-83a3-374e5202931f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-baa504-hgbht"
Oct 30 19:14:44.091708 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:44.024907    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35138e86-4197-404b-83a3-374e5202931f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-baa504-hgbht\" (UID: \"35138e86-4197-404b-83a3-374e5202931f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-baa504-hgbht"
Oct 30 19:14:44.091708 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:44.025041    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvkjq\" (UniqueName: \"kubernetes.io/projected/35138e86-4197-404b-83a3-374e5202931f-kube-api-access-xvkjq\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-baa504-hgbht\" (UID: \"35138e86-4197-404b-83a3-374e5202931f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-baa504-hgbht"
Oct 30 19:14:44.091708 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:44.069798    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e51c10-97m92"
Oct 30 19:14:44.091708 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:44.080933    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv" podUID="595230e3-55f3-43eb-9a7b-6d89250af9b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:14:44.117904 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:44.104416    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e51c10-97m92"
Oct 30 19:14:44.181707 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:44.170033    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e395ae73-f217-48d0-a277-6798e8221162-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e395ae73-f217-48d0-a277-6798e8221162-maas-api-openshift-ca\") pod \"e395ae73-f217-48d0-a277-6798e8221162\" (UID: \"e395ae73-f217-48d0-a277-6798e8221162\") "
Oct 30 19:14:44.181707 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:44.170142    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e395ae73-f217-48d0-a277-6798e8221162-kube-api-access-th84w\" (UniqueName: \"kubernetes.io/projected/e395ae73-f217-48d0-a277-6798e8221162-kube-api-access-th84w\") pod \"e395ae73-f217-48d0-a277-6798e8221162\" (UID: \"e395ae73-f217-48d0-a277-6798e8221162\") "
Oct 30 19:14:44.181707 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:44.170583    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e395ae73-f217-48d0-a277-6798e8221162-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e395ae73-f217-48d0-a277-6798e8221162-maas-api-xks-ca\") pod \"e395ae73-f217-48d0-a277-6798e8221162\" (UID: \"e395ae73-f217-48d0-a277-6798e8221162\") "
Oct 30 19:14:44.256706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:44.233282    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e395ae73-f217-48d0-a277-6798e8221162-maas-api-xks-ca" pod "e395ae73-f217-48d0-a277-6798e8221162" (UID: "e395ae73-f217-48d0-a277-6798e8221162"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:14:44.256706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:44.237883    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e395ae73-f217-48d0-a277-6798e8221162-maas-api-openshift-ca" pod "e395ae73-f217-48d0-a277-6798e8221162" (UID: "e395ae73-f217-48d0-a277-6798e8221162"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:14:44.256706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:44.243207    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e395ae73-f217-48d0-a277-6798e8221162-kube-api-access-th84w" pod "e395ae73-f217-48d0-a277-6798e8221162" (UID: "e395ae73-f217-48d0-a277-6798e8221162"). InnerVolumeSpecName "kube-api-access-th84w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:14:44.257236 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:44.257210    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35138e86-4197-404b-83a3-374e5202931f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-baa504-hgbht\" (UID: \"35138e86-4197-404b-83a3-374e5202931f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-baa504-hgbht"
Oct 30 19:14:44.257410 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:44.257394    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35138e86-4197-404b-83a3-374e5202931f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-baa504-hgbht\" (UID: \"35138e86-4197-404b-83a3-374e5202931f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-baa504-hgbht"
Oct 30 19:14:44.257637 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:44.257623    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvkjq\" (UniqueName: \"kubernetes.io/projected/35138e86-4197-404b-83a3-374e5202931f-kube-api-access-xvkjq\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-baa504-hgbht\" (UID: \"35138e86-4197-404b-83a3-374e5202931f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-baa504-hgbht"
Oct 30 19:14:44.281490 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:44.258173    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35138e86-4197-404b-83a3-374e5202931f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-baa504-hgbht\" (UID: \"35138e86-4197-404b-83a3-374e5202931f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-baa504-hgbht"
Oct 30 19:14:44.285852 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:44.285456    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e51c10-97m92" event={"ID":"e395ae73-f217-48d0-a277-6798e8221162","Type":"ContainerDied","Data":"5779d3450713ecac5c41849e00a417e0f679c5a12eb23ca091ed464627cb5aa5"}
Oct 30 19:14:44.285852 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:44.285517    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5779d3450713ecac5c41849e00a417e0f679c5a12eb23ca091ed464627cb5aa5"
Oct 30 19:14:44.285852 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:44.285538    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-baa504-hgbht"]
Oct 30 19:14:44.311032 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:44.310281    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e395ae73-f217-48d0-a277-6798e8221162-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:44.311032 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:44.310332    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-th84w\" (UniqueName: \"kubernetes.io/projected/e395ae73-f217-48d0-a277-6798e8221162-kube-api-access-th84w\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:44.311032 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:44.310352    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e395ae73-f217-48d0-a277-6798e8221162-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:44.330479 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:44.330448    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvkjq\" (UniqueName: \"kubernetes.io/projected/35138e86-4197-404b-83a3-374e5202931f-kube-api-access-xvkjq\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-baa504-hgbht\" (UID: \"35138e86-4197-404b-83a3-374e5202931f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-baa504-hgbht"
Oct 30 19:14:44.454706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:44.450836    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35138e86-4197-404b-83a3-374e5202931f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-baa504-hgbht\" (UID: \"35138e86-4197-404b-83a3-374e5202931f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-baa504-hgbht"
Oct 30 19:14:44.586270 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:44.585802    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-d6f956-openshift-default-56888545df-ftq7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" start-of-body=
Oct 30 19:14:44.586270 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:44.585865    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-ftq7b" podUID="e4cbde58-0ec2-4f0d-af09-048853d07d63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused"
Oct 30 19:14:44.627962 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:44.627922    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-baa504-hgbht"
Oct 30 19:14:44.906816 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:44.906788    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-96zzd"]
Oct 30 19:14:44.907213 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:44.907138    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-d7mth"]
Oct 30 19:14:44.907672 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:44.907559    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-d7mth" podUID="adb63756-9989-4246-84d1-5fc1d1c3ebd2" containerName="istio-proxy" containerID="cri-o://231338f743ae0e3cf6392e485d1590fc563014f25840d830674da9e82b25c850" gracePeriod=30
Oct 30 19:14:44.908982 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:44.908939    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-96zzd" podUID="33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a" containerName="istio-proxy" containerID="cri-o://de89b1a37146d36976ac5ab7bc6418454b59b538499c9cbdbb20a2f768a8caec" gracePeriod=30
Oct 30 19:14:44.915479 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:44.915457    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-jv2tm"]
Oct 30 19:14:44.916186 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:44.916157    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-jv2tm" podUID="c4071206-077a-4d0f-8f0d-a2b76604d376" containerName="istio-proxy" containerID="cri-o://c0a01bdb2075f569db79a3abd753564e2b87ed479f0ce90b585890680b4404cb" gracePeriod=30
Oct 30 19:14:44.955977 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:44.937138    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx"]
Oct 30 19:14:44.955977 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:44.937401    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx" podUID="b2099828-c41e-43c1-b174-63e8eb826c05" containerName="istio-proxy" containerID="cri-o://d2361ac39ee640bb402fb081951a8bce097a4f38e46382ffd99fc6093953ce05" gracePeriod=30
Oct 30 19:14:44.955977 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:44.950712    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-9v7v9"]
Oct 30 19:14:44.955977 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:44.950957    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-9v7v9" podUID="a0856440-337e-4507-ad1a-bdce910e1989" containerName="istio-proxy" containerID="cri-o://d5d74098339e199a238c446030a53edb63529d268797ed9df5a630493f0ebcd0" gracePeriod=30
Oct 30 19:14:44.967213 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:44.963826    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw"]
Oct 30 19:14:44.967213 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:44.964234    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw" podUID="bd0977b0-c75d-41ee-b911-4e71bd8c9b3a" containerName="istio-proxy" containerID="cri-o://279e860cf6912f80fd5d65c29c25ee1b35f5b7218d7bafd583abb950d7bcf2f7" gracePeriod=30
Oct 30 19:14:45.556635 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:45.556424    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:14:45.556937 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:45.556909    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw" podUID="bd0977b0-c75d-41ee-b911-4e71bd8c9b3a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:14:45.621869 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:45.621192    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-d6f956-openshift-default-56888545df-ftq7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" start-of-body=
Oct 30 19:14:45.621869 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:45.621252    4525 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-ftq7b" podUID="e4cbde58-0ec2-4f0d-af09-048853d07d63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused"
Oct 30 19:14:45.716141 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:14:45.716101    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35138e86_4197_404b_83a3_374e5202931f.slice/crio-28edec4822379090805404baf41585e507388fb3a71e073d81b535f9030c17b8 WatchSource:0}: Error finding container 28edec4822379090805404baf41585e507388fb3a71e073d81b535f9030c17b8: Status 404 returned error can't find the container with id 28edec4822379090805404baf41585e507388fb3a71e073d81b535f9030c17b8
Oct 30 19:14:45.734726 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:45.733313    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e67d95-28v72"]
Oct 30 19:14:45.734726 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:45.734551    4525 cpu_manager.go:414] "containerMap: removing container" podUID="e395ae73-f217-48d0-a277-6798e8221162" containerName="revoke-keys"
Oct 30 19:14:45.734726 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:45.734573    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e395ae73-f217-48d0-a277-6798e8221162" containerName="revoke-keys"
Oct 30 19:14:45.735260 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:45.735147    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="e395ae73-f217-48d0-a277-6798e8221162" containerName="revoke-keys"
Oct 30 19:14:45.741147 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:45.740731    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e67d95-28v72"
Oct 30 19:14:45.742660 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:45.742639    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e67d95-28v72"]
Oct 30 19:14:45.753262 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:45.753238    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-baa504-hgbht"]
Oct 30 19:14:45.771171 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:45.771128    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d8d8931-62df-4486-b331-f5c5013366e8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-e67d95-28v72\" (UID: \"3d8d8931-62df-4486-b331-f5c5013366e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e67d95-28v72"
Oct 30 19:14:45.771331 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:45.771281    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d8d8931-62df-4486-b331-f5c5013366e8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-e67d95-28v72\" (UID: \"3d8d8931-62df-4486-b331-f5c5013366e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e67d95-28v72"
Oct 30 19:14:45.771389 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:45.771334    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkg56\" (UniqueName: \"kubernetes.io/projected/3d8d8931-62df-4486-b331-f5c5013366e8-kube-api-access-wkg56\") pod \"maas-api-revoke-keys-e2e-extmod-b-e67d95-28v72\" (UID: \"3d8d8931-62df-4486-b331-f5c5013366e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e67d95-28v72"
Oct 30 19:14:45.873379 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:45.873349    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d8d8931-62df-4486-b331-f5c5013366e8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-e67d95-28v72\" (UID: \"3d8d8931-62df-4486-b331-f5c5013366e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e67d95-28v72"
Oct 30 19:14:45.873617 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:45.873433    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d8d8931-62df-4486-b331-f5c5013366e8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-e67d95-28v72\" (UID: \"3d8d8931-62df-4486-b331-f5c5013366e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e67d95-28v72"
Oct 30 19:14:45.873617 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:45.873458    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkg56\" (UniqueName: \"kubernetes.io/projected/3d8d8931-62df-4486-b331-f5c5013366e8-kube-api-access-wkg56\") pod \"maas-api-revoke-keys-e2e-extmod-b-e67d95-28v72\" (UID: \"3d8d8931-62df-4486-b331-f5c5013366e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e67d95-28v72"
Oct 30 19:14:45.874243 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:45.874221    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d8d8931-62df-4486-b331-f5c5013366e8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-e67d95-28v72\" (UID: \"3d8d8931-62df-4486-b331-f5c5013366e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e67d95-28v72"
Oct 30 19:14:45.876240 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:45.876214    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d8d8931-62df-4486-b331-f5c5013366e8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-e67d95-28v72\" (UID: \"3d8d8931-62df-4486-b331-f5c5013366e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e67d95-28v72"
Oct 30 19:14:45.907590 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:45.907563    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkg56\" (UniqueName: \"kubernetes.io/projected/3d8d8931-62df-4486-b331-f5c5013366e8-kube-api-access-wkg56\") pod \"maas-api-revoke-keys-e2e-extmod-b-e67d95-28v72\" (UID: \"3d8d8931-62df-4486-b331-f5c5013366e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e67d95-28v72"
Oct 30 19:14:45.970887 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:45.970849    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-jv2tm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Oct 30 19:14:45.971192 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:45.971159    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-jv2tm" podUID="c4071206-077a-4d0f-8f0d-a2b76604d376" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:14:46.058980 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:46.058943    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-d7mth container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:14:46.059239 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:46.059208    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-d7mth" podUID="adb63756-9989-4246-84d1-5fc1d1c3ebd2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:14:46.115534 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:46.115504    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e67d95-28v72"
Oct 30 19:14:46.129362 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:46.129293    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-96zzd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:14:46.129485 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:46.129362    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-96zzd" podUID="33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:14:46.412869 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:46.412839    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e67d95-28v72"]
Oct 30 19:14:46.413989 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:14:46.413950    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d8d8931_62df_4486_b331_f5c5013366e8.slice/crio-6d67a04aa57a3cffaca7d58666e190e896a0a9a03d04c6844c7b155f88324a6d WatchSource:0}: Error finding container 6d67a04aa57a3cffaca7d58666e190e896a0a9a03d04c6844c7b155f88324a6d: Status 404 returned error can't find the container with id 6d67a04aa57a3cffaca7d58666e190e896a0a9a03d04c6844c7b155f88324a6d
Oct 30 19:14:46.552959 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:46.552920    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e67d95-28v72" event={"ID":"3d8d8931-62df-4486-b331-f5c5013366e8","Type":"ContainerStarted","Data":"6d67a04aa57a3cffaca7d58666e190e896a0a9a03d04c6844c7b155f88324a6d"}
Oct 30 19:14:46.554892 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:46.554859    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-baa504-hgbht" event={"ID":"35138e86-4197-404b-83a3-374e5202931f","Type":"ContainerStarted","Data":"f86c467110a2745f7055d8b9c7d94cb59f9dbf7e7269a68708bc4b2fe2c5c6ba"}
Oct 30 19:14:46.555036 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:46.554915    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-baa504-hgbht" event={"ID":"35138e86-4197-404b-83a3-374e5202931f","Type":"ContainerStarted","Data":"28edec4822379090805404baf41585e507388fb3a71e073d81b535f9030c17b8"}
Oct 30 19:14:46.572630 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:46.572572    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-baa504-hgbht" podStartSLOduration=3.5725556899999997 podStartE2EDuration="3.57255569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:14:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:14:46.571372657 +0000 UTC m=+2309.197042625" watchObservedRunningTime="2026-10-30 19:14:46.57255569 +0000 UTC m=+2309.198225657"
Oct 30 19:14:46.811198 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:46.811165    4525 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-ftq7b"
Oct 30 19:14:46.811865 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:46.811624    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-ftq7b"
Oct 30 19:14:46.812802 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:46.812745    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-ftq7b"
Oct 30 19:14:47.405127 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:14:47.405081    4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d49l" containerName="extractor"
Oct 30 19:14:47.566222 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:47.566191    4525 generic.go:356] "Generic (PLEG): container finished" podID="35138e86-4197-404b-83a3-374e5202931f" containerID="f86c467110a2745f7055d8b9c7d94cb59f9dbf7e7269a68708bc4b2fe2c5c6ba" exitCode=0
Oct 30 19:14:47.566377 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:47.566275    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-baa504-hgbht" event={"ID":"35138e86-4197-404b-83a3-374e5202931f","Type":"ContainerDied","Data":"f86c467110a2745f7055d8b9c7d94cb59f9dbf7e7269a68708bc4b2fe2c5c6ba"}
Oct 30 19:14:47.568013 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:47.567991    4525 generic.go:356] "Generic (PLEG): container finished" podID="3d8d8931-62df-4486-b331-f5c5013366e8" containerID="a5ff5c33ec7faa9051831cdd23bc054cb95be6e1ed5159c54cd53da050a289a7" exitCode=0
Oct 30 19:14:47.568155 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:47.568022    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e67d95-28v72" event={"ID":"3d8d8931-62df-4486-b331-f5c5013366e8","Type":"ContainerDied","Data":"a5ff5c33ec7faa9051831cdd23bc054cb95be6e1ed5159c54cd53da050a289a7"}
Oct 30 19:14:48.967707 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:48.962052    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-baa504-hgbht"
Oct 30 19:14:49.211023 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:49.210032    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35138e86-4197-404b-83a3-374e5202931f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35138e86-4197-404b-83a3-374e5202931f-maas-api-openshift-ca\") pod \"35138e86-4197-404b-83a3-374e5202931f\" (UID: \"35138e86-4197-404b-83a3-374e5202931f\") "
Oct 30 19:14:49.211023 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:49.210141    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35138e86-4197-404b-83a3-374e5202931f-kube-api-access-xvkjq\" (UniqueName: \"kubernetes.io/projected/35138e86-4197-404b-83a3-374e5202931f-kube-api-access-xvkjq\") pod \"35138e86-4197-404b-83a3-374e5202931f\" (UID: \"35138e86-4197-404b-83a3-374e5202931f\") "
Oct 30 19:14:49.211023 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:49.210296    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35138e86-4197-404b-83a3-374e5202931f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35138e86-4197-404b-83a3-374e5202931f-maas-api-xks-ca\") pod \"35138e86-4197-404b-83a3-374e5202931f\" (UID: \"35138e86-4197-404b-83a3-374e5202931f\") "
Oct 30 19:14:49.218982 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:49.212468    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35138e86-4197-404b-83a3-374e5202931f-maas-api-openshift-ca" pod "35138e86-4197-404b-83a3-374e5202931f" (UID: "35138e86-4197-404b-83a3-374e5202931f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:14:49.232365 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:49.232307    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35138e86-4197-404b-83a3-374e5202931f-maas-api-xks-ca" pod "35138e86-4197-404b-83a3-374e5202931f" (UID: "35138e86-4197-404b-83a3-374e5202931f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:14:49.242642 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:49.242600    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35138e86-4197-404b-83a3-374e5202931f-kube-api-access-xvkjq" pod "35138e86-4197-404b-83a3-374e5202931f" (UID: "35138e86-4197-404b-83a3-374e5202931f"). InnerVolumeSpecName "kube-api-access-xvkjq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:14:49.307737 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:49.274594    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e67d95-28v72"
Oct 30 19:14:49.322128 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:49.322094    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35138e86-4197-404b-83a3-374e5202931f-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:49.322128 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:49.322127    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvkjq\" (UniqueName: \"kubernetes.io/projected/35138e86-4197-404b-83a3-374e5202931f-kube-api-access-xvkjq\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:49.322422 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:49.322144    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35138e86-4197-404b-83a3-374e5202931f-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:49.422918 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:49.422831    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d8d8931-62df-4486-b331-f5c5013366e8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d8d8931-62df-4486-b331-f5c5013366e8-maas-api-xks-ca\") pod \"3d8d8931-62df-4486-b331-f5c5013366e8\" (UID: \"3d8d8931-62df-4486-b331-f5c5013366e8\") "
Oct 30 19:14:49.423241 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:49.423222    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d8d8931-62df-4486-b331-f5c5013366e8-kube-api-access-wkg56\" (UniqueName: \"kubernetes.io/projected/3d8d8931-62df-4486-b331-f5c5013366e8-kube-api-access-wkg56\") pod \"3d8d8931-62df-4486-b331-f5c5013366e8\" (UID: \"3d8d8931-62df-4486-b331-f5c5013366e8\") "
Oct 30 19:14:49.423350 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:49.423291    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d8d8931-62df-4486-b331-f5c5013366e8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d8d8931-62df-4486-b331-f5c5013366e8-maas-api-openshift-ca\") pod \"3d8d8931-62df-4486-b331-f5c5013366e8\" (UID: \"3d8d8931-62df-4486-b331-f5c5013366e8\") "
Oct 30 19:14:49.423674 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:49.423651    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d8d8931-62df-4486-b331-f5c5013366e8-maas-api-openshift-ca" pod "3d8d8931-62df-4486-b331-f5c5013366e8" (UID: "3d8d8931-62df-4486-b331-f5c5013366e8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:14:49.424032 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:49.424014    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d8d8931-62df-4486-b331-f5c5013366e8-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:49.425138 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:49.425110    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d8d8931-62df-4486-b331-f5c5013366e8-maas-api-xks-ca" pod "3d8d8931-62df-4486-b331-f5c5013366e8" (UID: "3d8d8931-62df-4486-b331-f5c5013366e8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:14:49.425511 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:49.425493    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d8d8931-62df-4486-b331-f5c5013366e8-kube-api-access-wkg56" pod "3d8d8931-62df-4486-b331-f5c5013366e8" (UID: "3d8d8931-62df-4486-b331-f5c5013366e8"). InnerVolumeSpecName "kube-api-access-wkg56". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:14:49.525608 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:49.525573    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkg56\" (UniqueName: \"kubernetes.io/projected/3d8d8931-62df-4486-b331-f5c5013366e8-kube-api-access-wkg56\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:49.525608 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:49.525604    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d8d8931-62df-4486-b331-f5c5013366e8-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:49.595060 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:49.595030    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e67d95-28v72"
Oct 30 19:14:49.595213 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:49.595059    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e67d95-28v72" event={"ID":"3d8d8931-62df-4486-b331-f5c5013366e8","Type":"ContainerDied","Data":"6d67a04aa57a3cffaca7d58666e190e896a0a9a03d04c6844c7b155f88324a6d"}
Oct 30 19:14:49.595213 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:49.595105    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d67a04aa57a3cffaca7d58666e190e896a0a9a03d04c6844c7b155f88324a6d"
Oct 30 19:14:49.596746 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:49.596722    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-baa504-hgbht" event={"ID":"35138e86-4197-404b-83a3-374e5202931f","Type":"ContainerDied","Data":"28edec4822379090805404baf41585e507388fb3a71e073d81b535f9030c17b8"}
Oct 30 19:14:49.596746 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:49.596738    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-baa504-hgbht"
Oct 30 19:14:49.596886 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:49.596765    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="28edec4822379090805404baf41585e507388fb3a71e073d81b535f9030c17b8"
Oct 30 19:14:50.290963 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.290886    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-jv2tm"
Oct 30 19:14:50.294925 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.294902    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-d7mth"
Oct 30 19:14:50.350850 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.350822    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-96zzd"
Oct 30 19:14:50.443854 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.443819    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adb63756-9989-4246-84d1-5fc1d1c3ebd2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adb63756-9989-4246-84d1-5fc1d1c3ebd2-istiod-ca-cert\") pod \"adb63756-9989-4246-84d1-5fc1d1c3ebd2\" (UID: \"adb63756-9989-4246-84d1-5fc1d1c3ebd2\") "
Oct 30 19:14:50.444019 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.443902    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c4071206-077a-4d0f-8f0d-a2b76604d376-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4071206-077a-4d0f-8f0d-a2b76604d376-istio-podinfo\") pod \"c4071206-077a-4d0f-8f0d-a2b76604d376\" (UID: \"c4071206-077a-4d0f-8f0d-a2b76604d376\") "
Oct 30 19:14:50.444019 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.443956    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adb63756-9989-4246-84d1-5fc1d1c3ebd2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adb63756-9989-4246-84d1-5fc1d1c3ebd2-istio-data\") pod \"adb63756-9989-4246-84d1-5fc1d1c3ebd2\" (UID: \"adb63756-9989-4246-84d1-5fc1d1c3ebd2\") "
Oct 30 19:14:50.444019 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.443984    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4071206-077a-4d0f-8f0d-a2b76604d376-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4071206-077a-4d0f-8f0d-a2b76604d376-istio-envoy\") pod \"c4071206-077a-4d0f-8f0d-a2b76604d376\" (UID: \"c4071206-077a-4d0f-8f0d-a2b76604d376\") "
Oct 30 19:14:50.444296 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.444022    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4071206-077a-4d0f-8f0d-a2b76604d376-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4071206-077a-4d0f-8f0d-a2b76604d376-istiod-ca-cert\") pod \"c4071206-077a-4d0f-8f0d-a2b76604d376\" (UID: \"c4071206-077a-4d0f-8f0d-a2b76604d376\") "
Oct 30 19:14:50.444296 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.444080    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4071206-077a-4d0f-8f0d-a2b76604d376-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4071206-077a-4d0f-8f0d-a2b76604d376-workload-certs\") pod \"c4071206-077a-4d0f-8f0d-a2b76604d376\" (UID: \"c4071206-077a-4d0f-8f0d-a2b76604d376\") "
Oct 30 19:14:50.444296 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.444117    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4071206-077a-4d0f-8f0d-a2b76604d376-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4071206-077a-4d0f-8f0d-a2b76604d376-workload-socket\") pod \"c4071206-077a-4d0f-8f0d-a2b76604d376\" (UID: \"c4071206-077a-4d0f-8f0d-a2b76604d376\") "
Oct 30 19:14:50.444296 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.444145    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4071206-077a-4d0f-8f0d-a2b76604d376-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4071206-077a-4d0f-8f0d-a2b76604d376-istio-data\") pod \"c4071206-077a-4d0f-8f0d-a2b76604d376\" (UID: \"c4071206-077a-4d0f-8f0d-a2b76604d376\") "
Oct 30 19:14:50.444296 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.444202    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a-credential-socket\") pod \"33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a\" (UID: \"33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a\") "
Oct 30 19:14:50.444296 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.444237    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a-istio-envoy\") pod \"33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a\" (UID: \"33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a\") "
Oct 30 19:14:50.444639 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.444297    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adb63756-9989-4246-84d1-5fc1d1c3ebd2-istio-data" pod "adb63756-9989-4246-84d1-5fc1d1c3ebd2" (UID: "adb63756-9989-4246-84d1-5fc1d1c3ebd2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:50.444639 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.444311    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adb63756-9989-4246-84d1-5fc1d1c3ebd2-istiod-ca-cert" pod "adb63756-9989-4246-84d1-5fc1d1c3ebd2" (UID: "adb63756-9989-4246-84d1-5fc1d1c3ebd2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:14:50.444639 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.444335    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adb63756-9989-4246-84d1-5fc1d1c3ebd2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adb63756-9989-4246-84d1-5fc1d1c3ebd2-workload-certs\") pod \"adb63756-9989-4246-84d1-5fc1d1c3ebd2\" (UID: \"adb63756-9989-4246-84d1-5fc1d1c3ebd2\") "
Oct 30 19:14:50.444639 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.444371    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4071206-077a-4d0f-8f0d-a2b76604d376-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4071206-077a-4d0f-8f0d-a2b76604d376-credential-socket\") pod \"c4071206-077a-4d0f-8f0d-a2b76604d376\" (UID: \"c4071206-077a-4d0f-8f0d-a2b76604d376\") "
Oct 30 19:14:50.444639 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.444407    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adb63756-9989-4246-84d1-5fc1d1c3ebd2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adb63756-9989-4246-84d1-5fc1d1c3ebd2-istio-envoy\") pod \"adb63756-9989-4246-84d1-5fc1d1c3ebd2\" (UID: \"adb63756-9989-4246-84d1-5fc1d1c3ebd2\") "
Oct 30 19:14:50.444639 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.444477    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adb63756-9989-4246-84d1-5fc1d1c3ebd2-kube-api-access-5dzln\" (UniqueName: \"kubernetes.io/projected/adb63756-9989-4246-84d1-5fc1d1c3ebd2-kube-api-access-5dzln\") pod \"adb63756-9989-4246-84d1-5fc1d1c3ebd2\" (UID: \"adb63756-9989-4246-84d1-5fc1d1c3ebd2\") "
Oct 30 19:14:50.444639 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.444504    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4071206-077a-4d0f-8f0d-a2b76604d376-istio-data" pod "c4071206-077a-4d0f-8f0d-a2b76604d376" (UID: "c4071206-077a-4d0f-8f0d-a2b76604d376"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:50.444639 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.444526    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4071206-077a-4d0f-8f0d-a2b76604d376-workload-certs" pod "c4071206-077a-4d0f-8f0d-a2b76604d376" (UID: "c4071206-077a-4d0f-8f0d-a2b76604d376"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:50.444639 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.444530    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a-credential-socket" pod "33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a" (UID: "33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:50.444639 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.444527    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adb63756-9989-4246-84d1-5fc1d1c3ebd2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb63756-9989-4246-84d1-5fc1d1c3ebd2-credential-socket\") pod \"adb63756-9989-4246-84d1-5fc1d1c3ebd2\" (UID: \"adb63756-9989-4246-84d1-5fc1d1c3ebd2\") "
Oct 30 19:14:50.444639 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.444513    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4071206-077a-4d0f-8f0d-a2b76604d376-workload-socket" pod "c4071206-077a-4d0f-8f0d-a2b76604d376" (UID: "c4071206-077a-4d0f-8f0d-a2b76604d376"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:50.445525 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.444769    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adb63756-9989-4246-84d1-5fc1d1c3ebd2-credential-socket" pod "adb63756-9989-4246-84d1-5fc1d1c3ebd2" (UID: "adb63756-9989-4246-84d1-5fc1d1c3ebd2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:50.445525 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.444835    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4071206-077a-4d0f-8f0d-a2b76604d376-istiod-ca-cert" pod "c4071206-077a-4d0f-8f0d-a2b76604d376" (UID: "c4071206-077a-4d0f-8f0d-a2b76604d376"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:14:50.445525 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.444887    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adb63756-9989-4246-84d1-5fc1d1c3ebd2-istio-token\" (UniqueName: \"kubernetes.io/projected/adb63756-9989-4246-84d1-5fc1d1c3ebd2-istio-token\") pod \"adb63756-9989-4246-84d1-5fc1d1c3ebd2\" (UID: \"adb63756-9989-4246-84d1-5fc1d1c3ebd2\") "
Oct 30 19:14:50.445525 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.444940    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4071206-077a-4d0f-8f0d-a2b76604d376-kube-api-access-mqjg2\" (UniqueName: \"kubernetes.io/projected/c4071206-077a-4d0f-8f0d-a2b76604d376-kube-api-access-mqjg2\") pod \"c4071206-077a-4d0f-8f0d-a2b76604d376\" (UID: \"c4071206-077a-4d0f-8f0d-a2b76604d376\") "
Oct 30 19:14:50.445525 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.445002    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a-kube-api-access-7hhqw\" (UniqueName: \"kubernetes.io/projected/33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a-kube-api-access-7hhqw\") pod \"33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a\" (UID: \"33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a\") "
Oct 30 19:14:50.445525 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.445034    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a-istio-data\") pod \"33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a\" (UID: \"33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a\") "
Oct 30 19:14:50.445525 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.445063    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adb63756-9989-4246-84d1-5fc1d1c3ebd2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb63756-9989-4246-84d1-5fc1d1c3ebd2-workload-socket\") pod \"adb63756-9989-4246-84d1-5fc1d1c3ebd2\" (UID: \"adb63756-9989-4246-84d1-5fc1d1c3ebd2\") "
Oct 30 19:14:50.445525 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.445099    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4071206-077a-4d0f-8f0d-a2b76604d376-credential-socket" pod "c4071206-077a-4d0f-8f0d-a2b76604d376" (UID: "c4071206-077a-4d0f-8f0d-a2b76604d376"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:50.445525 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.445110    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a-istio-podinfo\") pod \"33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a\" (UID: \"33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a\") "
Oct 30 19:14:50.445525 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.445163    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a-istio-token\" (UniqueName: \"kubernetes.io/projected/33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a-istio-token\") pod \"33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a\" (UID: \"33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a\") "
Oct 30 19:14:50.445525 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.445190    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4071206-077a-4d0f-8f0d-a2b76604d376-istio-token\" (UniqueName: \"kubernetes.io/projected/c4071206-077a-4d0f-8f0d-a2b76604d376-istio-token\") pod \"c4071206-077a-4d0f-8f0d-a2b76604d376\" (UID: \"c4071206-077a-4d0f-8f0d-a2b76604d376\") "
Oct 30 19:14:50.445525 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.445267    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a-istiod-ca-cert\") pod \"33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a\" (UID: \"33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a\") "
Oct 30 19:14:50.445525 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.445297    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/adb63756-9989-4246-84d1-5fc1d1c3ebd2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adb63756-9989-4246-84d1-5fc1d1c3ebd2-istio-podinfo\") pod \"adb63756-9989-4246-84d1-5fc1d1c3ebd2\" (UID: \"adb63756-9989-4246-84d1-5fc1d1c3ebd2\") "
Oct 30 19:14:50.445525 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.445376    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a-workload-socket\") pod \"33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a\" (UID: \"33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a\") "
Oct 30 19:14:50.445525 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.445408    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a-workload-certs\") pod \"33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a\" (UID: \"33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a\") "
Oct 30 19:14:50.446370 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.445542    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adb63756-9989-4246-84d1-5fc1d1c3ebd2-workload-certs" pod "adb63756-9989-4246-84d1-5fc1d1c3ebd2" (UID: "adb63756-9989-4246-84d1-5fc1d1c3ebd2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:50.446570 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.446543    4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adb63756-9989-4246-84d1-5fc1d1c3ebd2-istio-data\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:50.446794 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.446577    4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4071206-077a-4d0f-8f0d-a2b76604d376-istiod-ca-cert\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:50.446794 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.446642    4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4071206-077a-4d0f-8f0d-a2b76604d376-workload-certs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:50.446794 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.446703    4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4071206-077a-4d0f-8f0d-a2b76604d376-workload-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:50.446794 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.446728    4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4071206-077a-4d0f-8f0d-a2b76604d376-istio-data\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:50.446794 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.446814    4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a-credential-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:50.446794 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.446828    4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adb63756-9989-4246-84d1-5fc1d1c3ebd2-workload-certs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:50.446794 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.446844    4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4071206-077a-4d0f-8f0d-a2b76604d376-credential-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:50.446794 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.446859    4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb63756-9989-4246-84d1-5fc1d1c3ebd2-credential-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:50.446794 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.446873    4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adb63756-9989-4246-84d1-5fc1d1c3ebd2-istiod-ca-cert\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:50.447504 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.447111    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a-workload-certs" pod "33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a" (UID: "33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:50.447504 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.447193    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4071206-077a-4d0f-8f0d-a2b76604d376-istio-envoy" pod "c4071206-077a-4d0f-8f0d-a2b76604d376" (UID: "c4071206-077a-4d0f-8f0d-a2b76604d376"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:50.447504 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.447314    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a-workload-socket" pod "33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a" (UID: "33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:50.447504 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.447387    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a-istio-envoy" pod "33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a" (UID: "33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:50.447765 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.447553    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a-istio-data" pod "33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a" (UID: "33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:50.447765 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.447563    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c4071206-077a-4d0f-8f0d-a2b76604d376-istio-podinfo" pod "c4071206-077a-4d0f-8f0d-a2b76604d376" (UID: "c4071206-077a-4d0f-8f0d-a2b76604d376"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:14:50.447765 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.447646    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adb63756-9989-4246-84d1-5fc1d1c3ebd2-workload-socket" pod "adb63756-9989-4246-84d1-5fc1d1c3ebd2" (UID: "adb63756-9989-4246-84d1-5fc1d1c3ebd2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:50.447949 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.447791    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a-istiod-ca-cert" pod "33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a" (UID: "33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:14:50.448305 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.448158    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adb63756-9989-4246-84d1-5fc1d1c3ebd2-kube-api-access-5dzln" pod "adb63756-9989-4246-84d1-5fc1d1c3ebd2" (UID: "adb63756-9989-4246-84d1-5fc1d1c3ebd2"). InnerVolumeSpecName "kube-api-access-5dzln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:14:50.448892 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.448866    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adb63756-9989-4246-84d1-5fc1d1c3ebd2-istio-token" pod "adb63756-9989-4246-84d1-5fc1d1c3ebd2" (UID: "adb63756-9989-4246-84d1-5fc1d1c3ebd2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:14:50.449001 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.448956    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a-kube-api-access-7hhqw" pod "33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a" (UID: "33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a"). InnerVolumeSpecName "kube-api-access-7hhqw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:14:50.449124 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.449104    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adb63756-9989-4246-84d1-5fc1d1c3ebd2-istio-envoy" pod "adb63756-9989-4246-84d1-5fc1d1c3ebd2" (UID: "adb63756-9989-4246-84d1-5fc1d1c3ebd2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:50.449423 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.449401    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/adb63756-9989-4246-84d1-5fc1d1c3ebd2-istio-podinfo" pod "adb63756-9989-4246-84d1-5fc1d1c3ebd2" (UID: "adb63756-9989-4246-84d1-5fc1d1c3ebd2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:14:50.449845 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.449815    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a-istio-token" pod "33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a" (UID: "33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:14:50.450062 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.450047    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4071206-077a-4d0f-8f0d-a2b76604d376-kube-api-access-mqjg2" pod "c4071206-077a-4d0f-8f0d-a2b76604d376" (UID: "c4071206-077a-4d0f-8f0d-a2b76604d376"). InnerVolumeSpecName "kube-api-access-mqjg2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:14:50.450343 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.450318    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a-istio-podinfo" pod "33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a" (UID: "33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:14:50.450444 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.450370    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4071206-077a-4d0f-8f0d-a2b76604d376-istio-token" pod "c4071206-077a-4d0f-8f0d-a2b76604d376" (UID: "c4071206-077a-4d0f-8f0d-a2b76604d376"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:14:50.550017 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.549922    4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a-workload-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:50.550017 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.549950    4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a-workload-certs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:50.550017 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.549961    4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4071206-077a-4d0f-8f0d-a2b76604d376-istio-podinfo\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:50.550017 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.549971    4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4071206-077a-4d0f-8f0d-a2b76604d376-istio-envoy\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:50.550017 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.549981    4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a-istio-envoy\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:50.550017 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.549990    4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adb63756-9989-4246-84d1-5fc1d1c3ebd2-istio-envoy\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:50.550017 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.549998    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dzln\" (UniqueName: \"kubernetes.io/projected/adb63756-9989-4246-84d1-5fc1d1c3ebd2-kube-api-access-5dzln\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:50.550017 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.550016    4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adb63756-9989-4246-84d1-5fc1d1c3ebd2-istio-token\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:50.550390 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.550025    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqjg2\" (UniqueName: \"kubernetes.io/projected/c4071206-077a-4d0f-8f0d-a2b76604d376-kube-api-access-mqjg2\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:50.550390 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.550034    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hhqw\" (UniqueName: \"kubernetes.io/projected/33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a-kube-api-access-7hhqw\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:50.550390 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.550042    4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a-istio-data\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:50.550390 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.550049    4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb63756-9989-4246-84d1-5fc1d1c3ebd2-workload-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:50.550390 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.550058    4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a-istio-podinfo\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:50.550390 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.550066    4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a-istio-token\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:50.550390 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.550074    4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4071206-077a-4d0f-8f0d-a2b76604d376-istio-token\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:50.550390 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.550081    4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a-istiod-ca-cert\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:50.550390 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.550089    4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adb63756-9989-4246-84d1-5fc1d1c3ebd2-istio-podinfo\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:50.612864 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.612830    4525 generic.go:356] "Generic (PLEG): container finished" podID="a0856440-337e-4507-ad1a-bdce910e1989" containerID="d5d74098339e199a238c446030a53edb63529d268797ed9df5a630493f0ebcd0" exitCode=0
Oct 30 19:14:50.613018 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.612884    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-9v7v9" event={"ID":"a0856440-337e-4507-ad1a-bdce910e1989","Type":"ContainerDied","Data":"d5d74098339e199a238c446030a53edb63529d268797ed9df5a630493f0ebcd0"}
Oct 30 19:14:50.614849 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.614819    4525 generic.go:356] "Generic (PLEG): container finished" podID="33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a" containerID="de89b1a37146d36976ac5ab7bc6418454b59b538499c9cbdbb20a2f768a8caec" exitCode=0
Oct 30 19:14:50.614974 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.614873    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-96zzd"
Oct 30 19:14:50.614974 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.614921    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-96zzd" event={"ID":"33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a","Type":"ContainerDied","Data":"de89b1a37146d36976ac5ab7bc6418454b59b538499c9cbdbb20a2f768a8caec"}
Oct 30 19:14:50.615100 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.614997    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-96zzd" event={"ID":"33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a","Type":"ContainerDied","Data":"895cd7380f11e5ccbf85c44a94f65c9592b1222cdc2e0c384c9c646962784a85"}
Oct 30 19:14:50.615100 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.615026    4525 scope.go:122] "RemoveContainer" containerID="de89b1a37146d36976ac5ab7bc6418454b59b538499c9cbdbb20a2f768a8caec"
Oct 30 19:14:50.620250 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.620225    4525 generic.go:356] "Generic (PLEG): container finished" podID="bd0977b0-c75d-41ee-b911-4e71bd8c9b3a" containerID="279e860cf6912f80fd5d65c29c25ee1b35f5b7218d7bafd583abb950d7bcf2f7" exitCode=0
Oct 30 19:14:50.620377 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.620256    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw" event={"ID":"bd0977b0-c75d-41ee-b911-4e71bd8c9b3a","Type":"ContainerDied","Data":"279e860cf6912f80fd5d65c29c25ee1b35f5b7218d7bafd583abb950d7bcf2f7"}
Oct 30 19:14:50.623256 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.623230    4525 generic.go:356] "Generic (PLEG): container finished" podID="c4071206-077a-4d0f-8f0d-a2b76604d376" containerID="c0a01bdb2075f569db79a3abd753564e2b87ed479f0ce90b585890680b4404cb" exitCode=0
Oct 30 19:14:50.623357 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.623322    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-jv2tm" event={"ID":"c4071206-077a-4d0f-8f0d-a2b76604d376","Type":"ContainerDied","Data":"c0a01bdb2075f569db79a3abd753564e2b87ed479f0ce90b585890680b4404cb"}
Oct 30 19:14:50.623357 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.623328    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-jv2tm"
Oct 30 19:14:50.623484 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.623367    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-jv2tm" event={"ID":"c4071206-077a-4d0f-8f0d-a2b76604d376","Type":"ContainerDied","Data":"a8bd98c7596fb9d80d0c2a8f1903ef9f06b92586f171513e4b3aaa88e8922a8c"}
Oct 30 19:14:50.628422 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.628395    4525 generic.go:356] "Generic (PLEG): container finished" podID="adb63756-9989-4246-84d1-5fc1d1c3ebd2" containerID="231338f743ae0e3cf6392e485d1590fc563014f25840d830674da9e82b25c850" exitCode=0
Oct 30 19:14:50.628569 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.628544    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-d7mth" event={"ID":"adb63756-9989-4246-84d1-5fc1d1c3ebd2","Type":"ContainerDied","Data":"231338f743ae0e3cf6392e485d1590fc563014f25840d830674da9e82b25c850"}
Oct 30 19:14:50.642945 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.628616    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-d7mth" event={"ID":"adb63756-9989-4246-84d1-5fc1d1c3ebd2","Type":"ContainerDied","Data":"6a0e88c728dfeea36e794a058121d6bc4d4e8db42c489d34765d00737ea66422"}
Oct 30 19:14:50.642945 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.628788    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-d7mth"
Oct 30 19:14:50.642945 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.642889    4525 scope.go:122] "RemoveContainer" containerID="de89b1a37146d36976ac5ab7bc6418454b59b538499c9cbdbb20a2f768a8caec"
Oct 30 19:14:50.643519 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:14:50.643309    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de89b1a37146d36976ac5ab7bc6418454b59b538499c9cbdbb20a2f768a8caec\": container with ID starting with de89b1a37146d36976ac5ab7bc6418454b59b538499c9cbdbb20a2f768a8caec not found: ID does not exist" containerID="de89b1a37146d36976ac5ab7bc6418454b59b538499c9cbdbb20a2f768a8caec"
Oct 30 19:14:50.643519 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.643343    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de89b1a37146d36976ac5ab7bc6418454b59b538499c9cbdbb20a2f768a8caec"} err="failed to get container status \"de89b1a37146d36976ac5ab7bc6418454b59b538499c9cbdbb20a2f768a8caec\": rpc error: code = NotFound desc = could not find container \"de89b1a37146d36976ac5ab7bc6418454b59b538499c9cbdbb20a2f768a8caec\": container with ID starting with de89b1a37146d36976ac5ab7bc6418454b59b538499c9cbdbb20a2f768a8caec not found: ID does not exist"
Oct 30 19:14:50.643519 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.643362    4525 scope.go:122] "RemoveContainer" containerID="c0a01bdb2075f569db79a3abd753564e2b87ed479f0ce90b585890680b4404cb"
Oct 30 19:14:50.658167 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.658141    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-96zzd"]
Oct 30 19:14:50.672690 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.671178    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-96zzd"]
Oct 30 19:14:50.722970 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.722939    4525 scope.go:122] "RemoveContainer" containerID="c0a01bdb2075f569db79a3abd753564e2b87ed479f0ce90b585890680b4404cb"
Oct 30 19:14:50.781388 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:14:50.773483    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0a01bdb2075f569db79a3abd753564e2b87ed479f0ce90b585890680b4404cb\": container with ID starting with c0a01bdb2075f569db79a3abd753564e2b87ed479f0ce90b585890680b4404cb not found: ID does not exist" containerID="c0a01bdb2075f569db79a3abd753564e2b87ed479f0ce90b585890680b4404cb"
Oct 30 19:14:50.781388 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.773534    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0a01bdb2075f569db79a3abd753564e2b87ed479f0ce90b585890680b4404cb"} err="failed to get container status \"c0a01bdb2075f569db79a3abd753564e2b87ed479f0ce90b585890680b4404cb\": rpc error: code = NotFound desc = could not find container \"c0a01bdb2075f569db79a3abd753564e2b87ed479f0ce90b585890680b4404cb\": container with ID starting with c0a01bdb2075f569db79a3abd753564e2b87ed479f0ce90b585890680b4404cb not found: ID does not exist"
Oct 30 19:14:50.781388 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.773561    4525 scope.go:122] "RemoveContainer" containerID="231338f743ae0e3cf6392e485d1590fc563014f25840d830674da9e82b25c850"
Oct 30 19:14:50.840457 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.840324    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-d7mth"]
Oct 30 19:14:50.845927 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.845892    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-d7mth"]
Oct 30 19:14:50.851844 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.851792    4525 scope.go:122] "RemoveContainer" containerID="231338f743ae0e3cf6392e485d1590fc563014f25840d830674da9e82b25c850"
Oct 30 19:14:50.852277 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:14:50.852230    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"231338f743ae0e3cf6392e485d1590fc563014f25840d830674da9e82b25c850\": container with ID starting with 231338f743ae0e3cf6392e485d1590fc563014f25840d830674da9e82b25c850 not found: ID does not exist" containerID="231338f743ae0e3cf6392e485d1590fc563014f25840d830674da9e82b25c850"
Oct 30 19:14:50.852391 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.852276    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"231338f743ae0e3cf6392e485d1590fc563014f25840d830674da9e82b25c850"} err="failed to get container status \"231338f743ae0e3cf6392e485d1590fc563014f25840d830674da9e82b25c850\": rpc error: code = NotFound desc = could not find container \"231338f743ae0e3cf6392e485d1590fc563014f25840d830674da9e82b25c850\": container with ID starting with 231338f743ae0e3cf6392e485d1590fc563014f25840d830674da9e82b25c850 not found: ID does not exist"
Oct 30 19:14:50.857427 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.857405    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-jv2tm"]
Oct 30 19:14:50.865579 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.865554    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-jv2tm"]
Oct 30 19:14:50.942505 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:50.942473    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-9v7v9"
Oct 30 19:14:51.002598 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.002565    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw"
Oct 30 19:14:51.029606 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.029574    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx"
Oct 30 19:14:51.089921 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.089842    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0856440-337e-4507-ad1a-bdce910e1989-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0856440-337e-4507-ad1a-bdce910e1989-workload-certs\") pod \"a0856440-337e-4507-ad1a-bdce910e1989\" (UID: \"a0856440-337e-4507-ad1a-bdce910e1989\") "
Oct 30 19:14:51.089921 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.089883    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd0977b0-c75d-41ee-b911-4e71bd8c9b3a-kube-api-access-h8psn\" (UniqueName: \"kubernetes.io/projected/bd0977b0-c75d-41ee-b911-4e71bd8c9b3a-kube-api-access-h8psn\") pod \"bd0977b0-c75d-41ee-b911-4e71bd8c9b3a\" (UID: \"bd0977b0-c75d-41ee-b911-4e71bd8c9b3a\") "
Oct 30 19:14:51.089921 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.089920    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd0977b0-c75d-41ee-b911-4e71bd8c9b3a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd0977b0-c75d-41ee-b911-4e71bd8c9b3a-credential-socket\") pod \"bd0977b0-c75d-41ee-b911-4e71bd8c9b3a\" (UID: \"bd0977b0-c75d-41ee-b911-4e71bd8c9b3a\") "
Oct 30 19:14:51.090230 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.089954    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bd0977b0-c75d-41ee-b911-4e71bd8c9b3a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd0977b0-c75d-41ee-b911-4e71bd8c9b3a-istio-podinfo\") pod \"bd0977b0-c75d-41ee-b911-4e71bd8c9b3a\" (UID: \"bd0977b0-c75d-41ee-b911-4e71bd8c9b3a\") "
Oct 30 19:14:51.090230 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.089985    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd0977b0-c75d-41ee-b911-4e71bd8c9b3a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd0977b0-c75d-41ee-b911-4e71bd8c9b3a-istiod-ca-cert\") pod \"bd0977b0-c75d-41ee-b911-4e71bd8c9b3a\" (UID: \"bd0977b0-c75d-41ee-b911-4e71bd8c9b3a\") "
Oct 30 19:14:51.090230 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.090132    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd0977b0-c75d-41ee-b911-4e71bd8c9b3a-istio-token\" (UniqueName: \"kubernetes.io/projected/bd0977b0-c75d-41ee-b911-4e71bd8c9b3a-istio-token\") pod \"bd0977b0-c75d-41ee-b911-4e71bd8c9b3a\" (UID: \"bd0977b0-c75d-41ee-b911-4e71bd8c9b3a\") "
Oct 30 19:14:51.090230 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.090142    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0856440-337e-4507-ad1a-bdce910e1989-workload-certs" pod "a0856440-337e-4507-ad1a-bdce910e1989" (UID: "a0856440-337e-4507-ad1a-bdce910e1989"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:51.090230 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.090170    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd0977b0-c75d-41ee-b911-4e71bd8c9b3a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd0977b0-c75d-41ee-b911-4e71bd8c9b3a-istio-envoy\") pod \"bd0977b0-c75d-41ee-b911-4e71bd8c9b3a\" (UID: \"bd0977b0-c75d-41ee-b911-4e71bd8c9b3a\") "
Oct 30 19:14:51.090230 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.090206    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd0977b0-c75d-41ee-b911-4e71bd8c9b3a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd0977b0-c75d-41ee-b911-4e71bd8c9b3a-workload-socket\") pod \"bd0977b0-c75d-41ee-b911-4e71bd8c9b3a\" (UID: \"bd0977b0-c75d-41ee-b911-4e71bd8c9b3a\") "
Oct 30 19:14:51.090553 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.090255    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0856440-337e-4507-ad1a-bdce910e1989-istio-token\" (UniqueName: \"kubernetes.io/projected/a0856440-337e-4507-ad1a-bdce910e1989-istio-token\") pod \"a0856440-337e-4507-ad1a-bdce910e1989\" (UID: \"a0856440-337e-4507-ad1a-bdce910e1989\") "
Oct 30 19:14:51.090553 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.090311    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd0977b0-c75d-41ee-b911-4e71bd8c9b3a-credential-socket" pod "bd0977b0-c75d-41ee-b911-4e71bd8c9b3a" (UID: "bd0977b0-c75d-41ee-b911-4e71bd8c9b3a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:51.090553 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.090335    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0856440-337e-4507-ad1a-bdce910e1989-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0856440-337e-4507-ad1a-bdce910e1989-istio-data\") pod \"a0856440-337e-4507-ad1a-bdce910e1989\" (UID: \"a0856440-337e-4507-ad1a-bdce910e1989\") "
Oct 30 19:14:51.090553 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.090508    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0856440-337e-4507-ad1a-bdce910e1989-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0856440-337e-4507-ad1a-bdce910e1989-workload-socket\") pod \"a0856440-337e-4507-ad1a-bdce910e1989\" (UID: \"a0856440-337e-4507-ad1a-bdce910e1989\") "
Oct 30 19:14:51.090553 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.090517    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd0977b0-c75d-41ee-b911-4e71bd8c9b3a-workload-socket" pod "bd0977b0-c75d-41ee-b911-4e71bd8c9b3a" (UID: "bd0977b0-c75d-41ee-b911-4e71bd8c9b3a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:51.090553 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.090532    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd0977b0-c75d-41ee-b911-4e71bd8c9b3a-istiod-ca-cert" pod "bd0977b0-c75d-41ee-b911-4e71bd8c9b3a" (UID: "bd0977b0-c75d-41ee-b911-4e71bd8c9b3a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:14:51.091001 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.090574    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a0856440-337e-4507-ad1a-bdce910e1989-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0856440-337e-4507-ad1a-bdce910e1989-istio-podinfo\") pod \"a0856440-337e-4507-ad1a-bdce910e1989\" (UID: \"a0856440-337e-4507-ad1a-bdce910e1989\") "
Oct 30 19:14:51.091001 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.090756    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0856440-337e-4507-ad1a-bdce910e1989-istio-data" pod "a0856440-337e-4507-ad1a-bdce910e1989" (UID: "a0856440-337e-4507-ad1a-bdce910e1989"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:51.091001 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.090780    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0856440-337e-4507-ad1a-bdce910e1989-workload-socket" pod "a0856440-337e-4507-ad1a-bdce910e1989" (UID: "a0856440-337e-4507-ad1a-bdce910e1989"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:51.091001 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.090873    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0856440-337e-4507-ad1a-bdce910e1989-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0856440-337e-4507-ad1a-bdce910e1989-credential-socket\") pod \"a0856440-337e-4507-ad1a-bdce910e1989\" (UID: \"a0856440-337e-4507-ad1a-bdce910e1989\") "
Oct 30 19:14:51.091001 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.090948    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0856440-337e-4507-ad1a-bdce910e1989-kube-api-access-vjgjr\" (UniqueName: \"kubernetes.io/projected/a0856440-337e-4507-ad1a-bdce910e1989-kube-api-access-vjgjr\") pod \"a0856440-337e-4507-ad1a-bdce910e1989\" (UID: \"a0856440-337e-4507-ad1a-bdce910e1989\") "
Oct 30 19:14:51.091001 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.090978    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0856440-337e-4507-ad1a-bdce910e1989-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0856440-337e-4507-ad1a-bdce910e1989-istio-envoy\") pod \"a0856440-337e-4507-ad1a-bdce910e1989\" (UID: \"a0856440-337e-4507-ad1a-bdce910e1989\") "
Oct 30 19:14:51.091400 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.091022    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd0977b0-c75d-41ee-b911-4e71bd8c9b3a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd0977b0-c75d-41ee-b911-4e71bd8c9b3a-workload-certs\") pod \"bd0977b0-c75d-41ee-b911-4e71bd8c9b3a\" (UID: \"bd0977b0-c75d-41ee-b911-4e71bd8c9b3a\") "
Oct 30 19:14:51.091400 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.091067    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd0977b0-c75d-41ee-b911-4e71bd8c9b3a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd0977b0-c75d-41ee-b911-4e71bd8c9b3a-istio-data\") pod \"bd0977b0-c75d-41ee-b911-4e71bd8c9b3a\" (UID: \"bd0977b0-c75d-41ee-b911-4e71bd8c9b3a\") "
Oct 30 19:14:51.091400 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.091133    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0856440-337e-4507-ad1a-bdce910e1989-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0856440-337e-4507-ad1a-bdce910e1989-istiod-ca-cert\") pod \"a0856440-337e-4507-ad1a-bdce910e1989\" (UID: \"a0856440-337e-4507-ad1a-bdce910e1989\") "
Oct 30 19:14:51.092454 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.092163    4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd0977b0-c75d-41ee-b911-4e71bd8c9b3a-workload-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:51.092454 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.092195    4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0856440-337e-4507-ad1a-bdce910e1989-istio-data\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:51.092454 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.092212    4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0856440-337e-4507-ad1a-bdce910e1989-workload-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:51.092454 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.092227    4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0856440-337e-4507-ad1a-bdce910e1989-workload-certs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:51.092454 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.092242    4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd0977b0-c75d-41ee-b911-4e71bd8c9b3a-credential-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:51.092454 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.092257    4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd0977b0-c75d-41ee-b911-4e71bd8c9b3a-istiod-ca-cert\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:51.093435 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.091067    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0856440-337e-4507-ad1a-bdce910e1989-credential-socket" pod "a0856440-337e-4507-ad1a-bdce910e1989" (UID: "a0856440-337e-4507-ad1a-bdce910e1989"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:51.093435 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.093035    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd0977b0-c75d-41ee-b911-4e71bd8c9b3a-kube-api-access-h8psn" pod "bd0977b0-c75d-41ee-b911-4e71bd8c9b3a" (UID: "bd0977b0-c75d-41ee-b911-4e71bd8c9b3a"). InnerVolumeSpecName "kube-api-access-h8psn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:14:51.093435 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.093215    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd0977b0-c75d-41ee-b911-4e71bd8c9b3a-istio-token" pod "bd0977b0-c75d-41ee-b911-4e71bd8c9b3a" (UID: "bd0977b0-c75d-41ee-b911-4e71bd8c9b3a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:14:51.093435 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.093250    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0856440-337e-4507-ad1a-bdce910e1989-istiod-ca-cert" pod "a0856440-337e-4507-ad1a-bdce910e1989" (UID: "a0856440-337e-4507-ad1a-bdce910e1989"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:14:51.093718 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.093455    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd0977b0-c75d-41ee-b911-4e71bd8c9b3a-istio-data" pod "bd0977b0-c75d-41ee-b911-4e71bd8c9b3a" (UID: "bd0977b0-c75d-41ee-b911-4e71bd8c9b3a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:51.093718 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.093482    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd0977b0-c75d-41ee-b911-4e71bd8c9b3a-istio-envoy" pod "bd0977b0-c75d-41ee-b911-4e71bd8c9b3a" (UID: "bd0977b0-c75d-41ee-b911-4e71bd8c9b3a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:51.093718 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.093554    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bd0977b0-c75d-41ee-b911-4e71bd8c9b3a-istio-podinfo" pod "bd0977b0-c75d-41ee-b911-4e71bd8c9b3a" (UID: "bd0977b0-c75d-41ee-b911-4e71bd8c9b3a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:14:51.093718 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.093571    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd0977b0-c75d-41ee-b911-4e71bd8c9b3a-workload-certs" pod "bd0977b0-c75d-41ee-b911-4e71bd8c9b3a" (UID: "bd0977b0-c75d-41ee-b911-4e71bd8c9b3a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:51.093850 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.093743    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a0856440-337e-4507-ad1a-bdce910e1989-istio-podinfo" pod "a0856440-337e-4507-ad1a-bdce910e1989" (UID: "a0856440-337e-4507-ad1a-bdce910e1989"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:14:51.094025 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.093995    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0856440-337e-4507-ad1a-bdce910e1989-istio-envoy" pod "a0856440-337e-4507-ad1a-bdce910e1989" (UID: "a0856440-337e-4507-ad1a-bdce910e1989"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:51.094231 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.094210    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0856440-337e-4507-ad1a-bdce910e1989-istio-token" pod "a0856440-337e-4507-ad1a-bdce910e1989" (UID: "a0856440-337e-4507-ad1a-bdce910e1989"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:14:51.095228 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.095209    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0856440-337e-4507-ad1a-bdce910e1989-kube-api-access-vjgjr" pod "a0856440-337e-4507-ad1a-bdce910e1989" (UID: "a0856440-337e-4507-ad1a-bdce910e1989"). InnerVolumeSpecName "kube-api-access-vjgjr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:14:51.193314 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.193278    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2099828-c41e-43c1-b174-63e8eb826c05-istio-token\" (UniqueName: \"kubernetes.io/projected/b2099828-c41e-43c1-b174-63e8eb826c05-istio-token\") pod \"b2099828-c41e-43c1-b174-63e8eb826c05\" (UID: \"b2099828-c41e-43c1-b174-63e8eb826c05\") "
Oct 30 19:14:51.193477 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.193328    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2099828-c41e-43c1-b174-63e8eb826c05-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2099828-c41e-43c1-b174-63e8eb826c05-workload-certs\") pod \"b2099828-c41e-43c1-b174-63e8eb826c05\" (UID: \"b2099828-c41e-43c1-b174-63e8eb826c05\") "
Oct 30 19:14:51.193477 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.193450    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2099828-c41e-43c1-b174-63e8eb826c05-kube-api-access-7fr8k\" (UniqueName: \"kubernetes.io/projected/b2099828-c41e-43c1-b174-63e8eb826c05-kube-api-access-7fr8k\") pod \"b2099828-c41e-43c1-b174-63e8eb826c05\" (UID: \"b2099828-c41e-43c1-b174-63e8eb826c05\") "
Oct 30 19:14:51.194187 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.193500    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2099828-c41e-43c1-b174-63e8eb826c05-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2099828-c41e-43c1-b174-63e8eb826c05-workload-socket\") pod \"b2099828-c41e-43c1-b174-63e8eb826c05\" (UID: \"b2099828-c41e-43c1-b174-63e8eb826c05\") "
Oct 30 19:14:51.194187 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.193523    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2099828-c41e-43c1-b174-63e8eb826c05-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2099828-c41e-43c1-b174-63e8eb826c05-istiod-ca-cert\") pod \"b2099828-c41e-43c1-b174-63e8eb826c05\" (UID: \"b2099828-c41e-43c1-b174-63e8eb826c05\") "
Oct 30 19:14:51.194187 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.193670    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2099828-c41e-43c1-b174-63e8eb826c05-workload-certs" pod "b2099828-c41e-43c1-b174-63e8eb826c05" (UID: "b2099828-c41e-43c1-b174-63e8eb826c05"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:51.194187 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.193721    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2099828-c41e-43c1-b174-63e8eb826c05-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2099828-c41e-43c1-b174-63e8eb826c05-credential-socket\") pod \"b2099828-c41e-43c1-b174-63e8eb826c05\" (UID: \"b2099828-c41e-43c1-b174-63e8eb826c05\") "
Oct 30 19:14:51.194187 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.193830    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2099828-c41e-43c1-b174-63e8eb826c05-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2099828-c41e-43c1-b174-63e8eb826c05-istio-data\") pod \"b2099828-c41e-43c1-b174-63e8eb826c05\" (UID: \"b2099828-c41e-43c1-b174-63e8eb826c05\") "
Oct 30 19:14:51.194187 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.193820    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2099828-c41e-43c1-b174-63e8eb826c05-workload-socket" pod "b2099828-c41e-43c1-b174-63e8eb826c05" (UID: "b2099828-c41e-43c1-b174-63e8eb826c05"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:51.194187 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.193892    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b2099828-c41e-43c1-b174-63e8eb826c05-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2099828-c41e-43c1-b174-63e8eb826c05-istio-podinfo\") pod \"b2099828-c41e-43c1-b174-63e8eb826c05\" (UID: \"b2099828-c41e-43c1-b174-63e8eb826c05\") "
Oct 30 19:14:51.194187 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.193937    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2099828-c41e-43c1-b174-63e8eb826c05-credential-socket" pod "b2099828-c41e-43c1-b174-63e8eb826c05" (UID: "b2099828-c41e-43c1-b174-63e8eb826c05"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:51.194187 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.193966    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2099828-c41e-43c1-b174-63e8eb826c05-istiod-ca-cert" pod "b2099828-c41e-43c1-b174-63e8eb826c05" (UID: "b2099828-c41e-43c1-b174-63e8eb826c05"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:14:51.194187 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.193985    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2099828-c41e-43c1-b174-63e8eb826c05-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2099828-c41e-43c1-b174-63e8eb826c05-istio-envoy\") pod \"b2099828-c41e-43c1-b174-63e8eb826c05\" (UID: \"b2099828-c41e-43c1-b174-63e8eb826c05\") "
Oct 30 19:14:51.194613 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.194245    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2099828-c41e-43c1-b174-63e8eb826c05-istio-data" pod "b2099828-c41e-43c1-b174-63e8eb826c05" (UID: "b2099828-c41e-43c1-b174-63e8eb826c05"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:51.195084 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.195058    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8psn\" (UniqueName: \"kubernetes.io/projected/bd0977b0-c75d-41ee-b911-4e71bd8c9b3a-kube-api-access-h8psn\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:51.195153 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.195093    4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd0977b0-c75d-41ee-b911-4e71bd8c9b3a-istio-podinfo\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:51.195153 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.195108    4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2099828-c41e-43c1-b174-63e8eb826c05-workload-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:51.195153 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.195122    4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2099828-c41e-43c1-b174-63e8eb826c05-istiod-ca-cert\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:51.195153 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.195136    4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd0977b0-c75d-41ee-b911-4e71bd8c9b3a-istio-token\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:51.195153 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.195151    4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd0977b0-c75d-41ee-b911-4e71bd8c9b3a-istio-envoy\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:51.195323 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.195165    4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0856440-337e-4507-ad1a-bdce910e1989-istio-token\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:51.195323 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.195178    4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2099828-c41e-43c1-b174-63e8eb826c05-credential-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:51.195323 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.195191    4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2099828-c41e-43c1-b174-63e8eb826c05-istio-data\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:51.195323 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.195206    4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0856440-337e-4507-ad1a-bdce910e1989-istio-podinfo\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:51.195323 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.195222    4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0856440-337e-4507-ad1a-bdce910e1989-credential-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:51.195323 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.195235    4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2099828-c41e-43c1-b174-63e8eb826c05-workload-certs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:51.195323 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.195249    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjgjr\" (UniqueName: \"kubernetes.io/projected/a0856440-337e-4507-ad1a-bdce910e1989-kube-api-access-vjgjr\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:51.195323 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.195263    4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0856440-337e-4507-ad1a-bdce910e1989-istio-envoy\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:51.195323 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.195276    4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd0977b0-c75d-41ee-b911-4e71bd8c9b3a-workload-certs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:51.195323 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.195290    4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd0977b0-c75d-41ee-b911-4e71bd8c9b3a-istio-data\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:51.195323 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.195303    4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0856440-337e-4507-ad1a-bdce910e1989-istiod-ca-cert\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:51.195826 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.195806    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2099828-c41e-43c1-b174-63e8eb826c05-istio-token" pod "b2099828-c41e-43c1-b174-63e8eb826c05" (UID: "b2099828-c41e-43c1-b174-63e8eb826c05"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:14:51.196215 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.196200    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b2099828-c41e-43c1-b174-63e8eb826c05-istio-podinfo" pod "b2099828-c41e-43c1-b174-63e8eb826c05" (UID: "b2099828-c41e-43c1-b174-63e8eb826c05"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:14:51.196486 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.196460    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2099828-c41e-43c1-b174-63e8eb826c05-istio-envoy" pod "b2099828-c41e-43c1-b174-63e8eb826c05" (UID: "b2099828-c41e-43c1-b174-63e8eb826c05"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:51.196581 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.196486    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2099828-c41e-43c1-b174-63e8eb826c05-kube-api-access-7fr8k" pod "b2099828-c41e-43c1-b174-63e8eb826c05" (UID: "b2099828-c41e-43c1-b174-63e8eb826c05"). InnerVolumeSpecName "kube-api-access-7fr8k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:14:51.296747 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.296710    4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2099828-c41e-43c1-b174-63e8eb826c05-istio-podinfo\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:51.296747 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.296742    4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2099828-c41e-43c1-b174-63e8eb826c05-istio-envoy\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:51.296747 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.296752    4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2099828-c41e-43c1-b174-63e8eb826c05-istio-token\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:51.297209 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.296761    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fr8k\" (UniqueName: \"kubernetes.io/projected/b2099828-c41e-43c1-b174-63e8eb826c05-kube-api-access-7fr8k\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:51.546536 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.546503    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a" path="/var/lib/kubelet/pods/33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a/volumes"
Oct 30 19:14:51.547177 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.547158    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adb63756-9989-4246-84d1-5fc1d1c3ebd2" path="/var/lib/kubelet/pods/adb63756-9989-4246-84d1-5fc1d1c3ebd2/volumes"
Oct 30 19:14:51.547658 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.547643    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4071206-077a-4d0f-8f0d-a2b76604d376" path="/var/lib/kubelet/pods/c4071206-077a-4d0f-8f0d-a2b76604d376/volumes"
Oct 30 19:14:51.640601 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.640570    4525 generic.go:356] "Generic (PLEG): container finished" podID="b2099828-c41e-43c1-b174-63e8eb826c05" containerID="d2361ac39ee640bb402fb081951a8bce097a4f38e46382ffd99fc6093953ce05" exitCode=0
Oct 30 19:14:51.640805 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.640624    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx"
Oct 30 19:14:51.640805 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.640664    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx" event={"ID":"b2099828-c41e-43c1-b174-63e8eb826c05","Type":"ContainerDied","Data":"d2361ac39ee640bb402fb081951a8bce097a4f38e46382ffd99fc6093953ce05"}
Oct 30 19:14:51.640805 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.640790    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx" event={"ID":"b2099828-c41e-43c1-b174-63e8eb826c05","Type":"ContainerDied","Data":"0e59cb69624ab07f933963b8efe2d091c529805cad6d5d671d9668648da390f4"}
Oct 30 19:14:51.640969 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.640823    4525 scope.go:122] "RemoveContainer" containerID="d2361ac39ee640bb402fb081951a8bce097a4f38e46382ffd99fc6093953ce05"
Oct 30 19:14:51.642535 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.642462    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-9v7v9"
Oct 30 19:14:51.642535 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.642516    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-9v7v9" event={"ID":"a0856440-337e-4507-ad1a-bdce910e1989","Type":"ContainerDied","Data":"6f9e2954da2f9fb4a249e60fef1c6bc5429fb96ec0b3958fed2ffd57ef6c3d41"}
Oct 30 19:14:51.646328 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.646311    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw"
Oct 30 19:14:51.646425 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.646361    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw" event={"ID":"bd0977b0-c75d-41ee-b911-4e71bd8c9b3a","Type":"ContainerDied","Data":"1b1e851e5e6a96a533427100bf6f1d08619bdca2b328a0240371857132596b24"}
Oct 30 19:14:51.658584 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.658553    4525 scope.go:122] "RemoveContainer" containerID="4e4ca10e76fb3b83340b3f05a3a5934faae758882759cd4acbf5258f4c00f1a7"
Oct 30 19:14:51.674078 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.674056    4525 scope.go:122] "RemoveContainer" containerID="d2361ac39ee640bb402fb081951a8bce097a4f38e46382ffd99fc6093953ce05"
Oct 30 19:14:51.674195 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.674177    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx"]
Oct 30 19:14:51.674439 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:14:51.674418    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2361ac39ee640bb402fb081951a8bce097a4f38e46382ffd99fc6093953ce05\": container with ID starting with d2361ac39ee640bb402fb081951a8bce097a4f38e46382ffd99fc6093953ce05 not found: ID does not exist" containerID="d2361ac39ee640bb402fb081951a8bce097a4f38e46382ffd99fc6093953ce05"
Oct 30 19:14:51.674513 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.674451    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2361ac39ee640bb402fb081951a8bce097a4f38e46382ffd99fc6093953ce05"} err="failed to get container status \"d2361ac39ee640bb402fb081951a8bce097a4f38e46382ffd99fc6093953ce05\": rpc error: code = NotFound desc = could not find container \"d2361ac39ee640bb402fb081951a8bce097a4f38e46382ffd99fc6093953ce05\": container with ID starting with d2361ac39ee640bb402fb081951a8bce097a4f38e46382ffd99fc6093953ce05 not found: ID does not exist"
Oct 30 19:14:51.674513 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.674475    4525 scope.go:122] "RemoveContainer" containerID="4e4ca10e76fb3b83340b3f05a3a5934faae758882759cd4acbf5258f4c00f1a7"
Oct 30 19:14:51.674774 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:14:51.674757    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e4ca10e76fb3b83340b3f05a3a5934faae758882759cd4acbf5258f4c00f1a7\": container with ID starting with 4e4ca10e76fb3b83340b3f05a3a5934faae758882759cd4acbf5258f4c00f1a7 not found: ID does not exist" containerID="4e4ca10e76fb3b83340b3f05a3a5934faae758882759cd4acbf5258f4c00f1a7"
Oct 30 19:14:51.674828 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.674779    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e4ca10e76fb3b83340b3f05a3a5934faae758882759cd4acbf5258f4c00f1a7"} err="failed to get container status \"4e4ca10e76fb3b83340b3f05a3a5934faae758882759cd4acbf5258f4c00f1a7\": rpc error: code = NotFound desc = could not find container \"4e4ca10e76fb3b83340b3f05a3a5934faae758882759cd4acbf5258f4c00f1a7\": container with ID starting with 4e4ca10e76fb3b83340b3f05a3a5934faae758882759cd4acbf5258f4c00f1a7 not found: ID does not exist"
Oct 30 19:14:51.674828 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.674794    4525 scope.go:122] "RemoveContainer" containerID="d5d74098339e199a238c446030a53edb63529d268797ed9df5a630493f0ebcd0"
Oct 30 19:14:51.680270 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.680249    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-s52kx"]
Oct 30 19:14:51.686955 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.686935    4525 scope.go:122] "RemoveContainer" containerID="279e860cf6912f80fd5d65c29c25ee1b35f5b7218d7bafd583abb950d7bcf2f7"
Oct 30 19:14:51.692388 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.692363    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-9v7v9"]
Oct 30 19:14:51.697986 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.697966    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-9v7v9"]
Oct 30 19:14:51.706774 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.706748    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw"]
Oct 30 19:14:51.711882 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:51.711862    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-1dd09d-openshift-default-7477c8cdbf-zp9hw"]
Oct 30 19:14:53.124272 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:53.124234    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90343bb1-k4hvz"]
Oct 30 19:14:53.124969 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:53.124950    4525 cpu_manager.go:414] "containerMap: removing container" podUID="3d8d8931-62df-4486-b331-f5c5013366e8" containerName="revoke-keys"
Oct 30 19:14:53.124969 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:53.124968    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d8d8931-62df-4486-b331-f5c5013366e8" containerName="revoke-keys"
Oct 30 19:14:53.125095 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:53.124985    4525 cpu_manager.go:414] "containerMap: removing container" podUID="a0856440-337e-4507-ad1a-bdce910e1989" containerName="istio-proxy"
Oct 30 19:14:53.125095 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:53.124990    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0856440-337e-4507-ad1a-bdce910e1989" containerName="istio-proxy"
Oct 30 19:14:53.125095 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:53.125001    4525 cpu_manager.go:414] "containerMap: removing container" podUID="33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a" containerName="istio-proxy"
Oct 30 19:14:53.125095 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:53.125006    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a" containerName="istio-proxy"
Oct 30 19:14:53.125095 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:53.125028    4525 cpu_manager.go:414] "containerMap: removing container" podUID="bd0977b0-c75d-41ee-b911-4e71bd8c9b3a" containerName="istio-proxy"
Oct 30 19:14:53.125095 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:53.125033    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd0977b0-c75d-41ee-b911-4e71bd8c9b3a" containerName="istio-proxy"
Oct 30 19:14:53.125095 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:53.125043    4525 cpu_manager.go:414] "containerMap: removing container" podUID="c4071206-077a-4d0f-8f0d-a2b76604d376" containerName="istio-proxy"
Oct 30 19:14:53.125095 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:53.125048    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4071206-077a-4d0f-8f0d-a2b76604d376" containerName="istio-proxy"
Oct 30 19:14:53.125095 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:53.125055    4525 cpu_manager.go:414] "containerMap: removing container" podUID="b2099828-c41e-43c1-b174-63e8eb826c05" containerName="istio-proxy"
Oct 30 19:14:53.125095 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:53.125059    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2099828-c41e-43c1-b174-63e8eb826c05" containerName="istio-proxy"
Oct 30 19:14:53.125095 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:53.125075    4525 cpu_manager.go:414] "containerMap: removing container" podUID="35138e86-4197-404b-83a3-374e5202931f" containerName="revoke-keys"
Oct 30 19:14:53.125095 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:53.125079    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35138e86-4197-404b-83a3-374e5202931f" containerName="revoke-keys"
Oct 30 19:14:53.125095 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:53.125091    4525 cpu_manager.go:414] "containerMap: removing container" podUID="b2099828-c41e-43c1-b174-63e8eb826c05" containerName="istio-proxy"
Oct 30 19:14:53.125095 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:53.125095    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2099828-c41e-43c1-b174-63e8eb826c05" containerName="istio-proxy"
Oct 30 19:14:53.125501 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:53.125104    4525 cpu_manager.go:414] "containerMap: removing container" podUID="adb63756-9989-4246-84d1-5fc1d1c3ebd2" containerName="istio-proxy"
Oct 30 19:14:53.125501 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:53.125108    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adb63756-9989-4246-84d1-5fc1d1c3ebd2" containerName="istio-proxy"
Oct 30 19:14:53.125501 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:53.125200    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="35138e86-4197-404b-83a3-374e5202931f" containerName="revoke-keys"
Oct 30 19:14:53.125501 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:53.125210    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2099828-c41e-43c1-b174-63e8eb826c05" containerName="istio-proxy"
Oct 30 19:14:53.125501 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:53.125216    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="adb63756-9989-4246-84d1-5fc1d1c3ebd2" containerName="istio-proxy"
Oct 30 19:14:53.125501 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:53.125222    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d8d8931-62df-4486-b331-f5c5013366e8" containerName="revoke-keys"
Oct 30 19:14:53.125501 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:53.125230    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0856440-337e-4507-ad1a-bdce910e1989" containerName="istio-proxy"
Oct 30 19:14:53.125501 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:53.125235    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="33b5fccf-6fc7-4a25-9338-9c3c4a5ff14a" containerName="istio-proxy"
Oct 30 19:14:53.125501 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:53.125243    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd0977b0-c75d-41ee-b911-4e71bd8c9b3a" containerName="istio-proxy"
Oct 30 19:14:53.125501 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:53.125248    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4071206-077a-4d0f-8f0d-a2b76604d376" containerName="istio-proxy"
Oct 30 19:14:53.125501 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:53.125255    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2099828-c41e-43c1-b174-63e8eb826c05" containerName="istio-proxy"
Oct 30 19:14:53.131620 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:53.131592    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90343bb1-k4hvz"
Oct 30 19:14:53.135618 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:53.135598    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:14:53.135768 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:53.135698    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94hcn\""
Oct 30 19:14:53.138735 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:53.138699    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90343bb1-k4hvz"]
Oct 30 19:14:53.234496 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:53.234468    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cafe781-d9a6-4a40-b632-9dc7baf6a243-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90343bb1-k4hvz\" (UID: \"9cafe781-d9a6-4a40-b632-9dc7baf6a243\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90343bb1-k4hvz"
Oct 30 19:14:53.234496 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:53.234504    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d64z\" (UniqueName: \"kubernetes.io/projected/9cafe781-d9a6-4a40-b632-9dc7baf6a243-kube-api-access-8d64z\") pod \"maas-api-revoke-sub-90343bb1-k4hvz\" (UID: \"9cafe781-d9a6-4a40-b632-9dc7baf6a243\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90343bb1-k4hvz"
Oct 30 19:14:53.234714 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:53.234697    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cafe781-d9a6-4a40-b632-9dc7baf6a243-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90343bb1-k4hvz\" (UID: \"9cafe781-d9a6-4a40-b632-9dc7baf6a243\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90343bb1-k4hvz"
Oct 30 19:14:53.335860 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:53.335827    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cafe781-d9a6-4a40-b632-9dc7baf6a243-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90343bb1-k4hvz\" (UID: \"9cafe781-d9a6-4a40-b632-9dc7baf6a243\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90343bb1-k4hvz"
Oct 30 19:14:53.336044 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:53.335866    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d64z\" (UniqueName: \"kubernetes.io/projected/9cafe781-d9a6-4a40-b632-9dc7baf6a243-kube-api-access-8d64z\") pod \"maas-api-revoke-sub-90343bb1-k4hvz\" (UID: \"9cafe781-d9a6-4a40-b632-9dc7baf6a243\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90343bb1-k4hvz"
Oct 30 19:14:53.336044 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:53.335995    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cafe781-d9a6-4a40-b632-9dc7baf6a243-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90343bb1-k4hvz\" (UID: \"9cafe781-d9a6-4a40-b632-9dc7baf6a243\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90343bb1-k4hvz"
Oct 30 19:14:53.336602 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:53.336581    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cafe781-d9a6-4a40-b632-9dc7baf6a243-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90343bb1-k4hvz\" (UID: \"9cafe781-d9a6-4a40-b632-9dc7baf6a243\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90343bb1-k4hvz"
Oct 30 19:14:53.338299 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:53.338270    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cafe781-d9a6-4a40-b632-9dc7baf6a243-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90343bb1-k4hvz\" (UID: \"9cafe781-d9a6-4a40-b632-9dc7baf6a243\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90343bb1-k4hvz"
Oct 30 19:14:53.344455 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:53.344429    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d64z\" (UniqueName: \"kubernetes.io/projected/9cafe781-d9a6-4a40-b632-9dc7baf6a243-kube-api-access-8d64z\") pod \"maas-api-revoke-sub-90343bb1-k4hvz\" (UID: \"9cafe781-d9a6-4a40-b632-9dc7baf6a243\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90343bb1-k4hvz"
Oct 30 19:14:53.449485 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:53.449410    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90343bb1-k4hvz"
Oct 30 19:14:53.660387 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:53.660348    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0856440-337e-4507-ad1a-bdce910e1989" path="/var/lib/kubelet/pods/a0856440-337e-4507-ad1a-bdce910e1989/volumes"
Oct 30 19:14:53.661058 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:53.661037    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2099828-c41e-43c1-b174-63e8eb826c05" path="/var/lib/kubelet/pods/b2099828-c41e-43c1-b174-63e8eb826c05/volumes"
Oct 30 19:14:53.661812 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:53.661790    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd0977b0-c75d-41ee-b911-4e71bd8c9b3a" path="/var/lib/kubelet/pods/bd0977b0-c75d-41ee-b911-4e71bd8c9b3a/volumes"
Oct 30 19:14:53.799172 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:14:53.799141    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cafe781_d9a6_4a40_b632_9dc7baf6a243.slice/crio-7e6688bd6e0fbc5c326c2a0705127d3d38274af070b31cb1e8f278b30421db7f WatchSource:0}: Error finding container 7e6688bd6e0fbc5c326c2a0705127d3d38274af070b31cb1e8f278b30421db7f: Status 404 returned error can't find the container with id 7e6688bd6e0fbc5c326c2a0705127d3d38274af070b31cb1e8f278b30421db7f
Oct 30 19:14:53.800043 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:53.800017    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90343bb1-k4hvz"]
Oct 30 19:14:54.702665 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:54.702628    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90343bb1-k4hvz" event={"ID":"9cafe781-d9a6-4a40-b632-9dc7baf6a243","Type":"ContainerStarted","Data":"a98a1446451751afd861d7f454b833a9dc4aa290ce6e0f94038a6c9a177127f2"}
Oct 30 19:14:54.703085 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:54.702705    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90343bb1-k4hvz" event={"ID":"9cafe781-d9a6-4a40-b632-9dc7baf6a243","Type":"ContainerStarted","Data":"7e6688bd6e0fbc5c326c2a0705127d3d38274af070b31cb1e8f278b30421db7f"}
Oct 30 19:14:54.725987 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:54.725943    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90343bb1-k4hvz" podStartSLOduration=1.725929426 podStartE2EDuration="1.725929426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:14:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:14:54.720629449 +0000 UTC m=+2317.346299416" watchObservedRunningTime="2026-10-30 19:14:54.725929426 +0000 UTC m=+2317.351599389"
Oct 30 19:14:55.714637 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:55.714593    4525 generic.go:356] "Generic (PLEG): container finished" podID="9cafe781-d9a6-4a40-b632-9dc7baf6a243" containerID="a98a1446451751afd861d7f454b833a9dc4aa290ce6e0f94038a6c9a177127f2" exitCode=0
Oct 30 19:14:55.715147 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:55.714734    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90343bb1-k4hvz" event={"ID":"9cafe781-d9a6-4a40-b632-9dc7baf6a243","Type":"ContainerDied","Data":"a98a1446451751afd861d7f454b833a9dc4aa290ce6e0f94038a6c9a177127f2"}
Oct 30 19:14:56.899027 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:56.899004    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90343bb1-k4hvz"
Oct 30 19:14:56.934544 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:56.934512    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9cafe781-d9a6-4a40-b632-9dc7baf6a243-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cafe781-d9a6-4a40-b632-9dc7baf6a243-maas-api-xks-ca\") pod \"9cafe781-d9a6-4a40-b632-9dc7baf6a243\" (UID: \"9cafe781-d9a6-4a40-b632-9dc7baf6a243\") "
Oct 30 19:14:56.934803 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:56.934786    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cafe781-d9a6-4a40-b632-9dc7baf6a243-kube-api-access-8d64z\" (UniqueName: \"kubernetes.io/projected/9cafe781-d9a6-4a40-b632-9dc7baf6a243-kube-api-access-8d64z\") pod \"9cafe781-d9a6-4a40-b632-9dc7baf6a243\" (UID: \"9cafe781-d9a6-4a40-b632-9dc7baf6a243\") "
Oct 30 19:14:56.934934 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:56.934914    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cafe781-d9a6-4a40-b632-9dc7baf6a243-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cafe781-d9a6-4a40-b632-9dc7baf6a243-maas-api-openshift-ca\") pod \"9cafe781-d9a6-4a40-b632-9dc7baf6a243\" (UID: \"9cafe781-d9a6-4a40-b632-9dc7baf6a243\") "
Oct 30 19:14:56.935532 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:56.935483    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cafe781-d9a6-4a40-b632-9dc7baf6a243-maas-api-openshift-ca" pod "9cafe781-d9a6-4a40-b632-9dc7baf6a243" (UID: "9cafe781-d9a6-4a40-b632-9dc7baf6a243"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:14:56.935931 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:56.935904    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cafe781-d9a6-4a40-b632-9dc7baf6a243-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:56.937041 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:56.937013    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cafe781-d9a6-4a40-b632-9dc7baf6a243-maas-api-xks-ca" pod "9cafe781-d9a6-4a40-b632-9dc7baf6a243" (UID: "9cafe781-d9a6-4a40-b632-9dc7baf6a243"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:14:56.937336 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:56.937317    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cafe781-d9a6-4a40-b632-9dc7baf6a243-kube-api-access-8d64z" pod "9cafe781-d9a6-4a40-b632-9dc7baf6a243" (UID: "9cafe781-d9a6-4a40-b632-9dc7baf6a243"). InnerVolumeSpecName "kube-api-access-8d64z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:14:57.038108 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:57.038016    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8d64z\" (UniqueName: \"kubernetes.io/projected/9cafe781-d9a6-4a40-b632-9dc7baf6a243-kube-api-access-8d64z\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:57.038108 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:57.038059    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cafe781-d9a6-4a40-b632-9dc7baf6a243-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:57.732617 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:57.732580    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90343bb1-k4hvz"
Oct 30 19:14:57.732617 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:57.732609    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90343bb1-k4hvz" event={"ID":"9cafe781-d9a6-4a40-b632-9dc7baf6a243","Type":"ContainerDied","Data":"7e6688bd6e0fbc5c326c2a0705127d3d38274af070b31cb1e8f278b30421db7f"}
Oct 30 19:14:57.732879 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:57.732654    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e6688bd6e0fbc5c326c2a0705127d3d38274af070b31cb1e8f278b30421db7f"
Oct 30 19:14:59.618706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:59.611808    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e51c10-97m92"]
Oct 30 19:14:59.649710 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:14:59.641700    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e51c10-97m92"]
Oct 30 19:15:00.145045 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.145009    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-ef6b56-29889795-nl7wc"]
Oct 30 19:15:00.147420 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.146115    4525 cpu_manager.go:414] "containerMap: removing container" podUID="9cafe781-d9a6-4a40-b632-9dc7baf6a243" containerName="revoke-keys"
Oct 30 19:15:00.147420 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.146141    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cafe781-d9a6-4a40-b632-9dc7baf6a243" containerName="revoke-keys"
Oct 30 19:15:00.147420 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.146360    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cafe781-d9a6-4a40-b632-9dc7baf6a243" containerName="revoke-keys"
Oct 30 19:15:00.150970 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.150949    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-ef6b56-29889795-nl7wc"
Oct 30 19:15:00.153234 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.153214    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ef6be7-29889795-pf2p2"]
Oct 30 19:15:00.157796 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.157774    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ef6be7-29889795-pf2p2"
Oct 30 19:15:00.160888 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.160853    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a1d55d-29889795-bw7v2"]
Oct 30 19:15:00.181257 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.181222    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-46385c-29889795-hbvv9"]
Oct 30 19:15:00.181554 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.181534    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a1d55d-29889795-bw7v2"
Oct 30 19:15:00.187351 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.187320    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c2e3f9-29889795-5rf8p"]
Oct 30 19:15:00.187600 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.187472    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-46385c-29889795-hbvv9"
Oct 30 19:15:00.194755 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.194389    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-ef6b56-29889795-nl7wc"]
Oct 30 19:15:00.194755 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.194517    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c2e3f9-29889795-5rf8p"
Oct 30 19:15:00.197332 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.197301    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ef6be7-29889795-pf2p2"]
Oct 30 19:15:00.204102 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.203974    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-46385c-29889795-hbvv9"]
Oct 30 19:15:00.217151 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.217050    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c2e3f9-29889795-5rf8p"]
Oct 30 19:15:00.224753 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.222673    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wvhv\" (UniqueName: \"kubernetes.io/projected/90e98322-b727-4a35-9d93-808e0b01d483-kube-api-access-6wvhv\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-a1d55d-29889795-bw7v2\" (UID: \"90e98322-b727-4a35-9d93-808e0b01d483\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a1d55d-29889795-bw7v2"
Oct 30 19:15:00.224753 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.222841    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5149d31-2979-4353-9b41-df84f90cffbd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-46385c-29889795-hbvv9\" (UID: \"e5149d31-2979-4353-9b41-df84f90cffbd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-46385c-29889795-hbvv9"
Oct 30 19:15:00.224753 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.222878    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkjr6\" (UniqueName: \"kubernetes.io/projected/e5149d31-2979-4353-9b41-df84f90cffbd-kube-api-access-mkjr6\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-46385c-29889795-hbvv9\" (UID: \"e5149d31-2979-4353-9b41-df84f90cffbd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-46385c-29889795-hbvv9"
Oct 30 19:15:00.224753 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.222961    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d89pk\" (UniqueName: \"kubernetes.io/projected/4849dbc2-0043-46f1-ab5e-a76c16f85db9-kube-api-access-d89pk\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-c2e3f9-29889795-5rf8p\" (UID: \"4849dbc2-0043-46f1-ab5e-a76c16f85db9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c2e3f9-29889795-5rf8p"
Oct 30 19:15:00.224753 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.223115    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d3ff890-4ac4-4395-b309-3483be728b66-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-ef6b56-29889795-nl7wc\" (UID: \"8d3ff890-4ac4-4395-b309-3483be728b66\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-ef6b56-29889795-nl7wc"
Oct 30 19:15:00.224753 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.223158    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z7jt\" (UniqueName: \"kubernetes.io/projected/8d3ff890-4ac4-4395-b309-3483be728b66-kube-api-access-9z7jt\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-ef6b56-29889795-nl7wc\" (UID: \"8d3ff890-4ac4-4395-b309-3483be728b66\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-ef6b56-29889795-nl7wc"
Oct 30 19:15:00.224753 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.223292    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7shqk\" (UniqueName: \"kubernetes.io/projected/e8fa12c9-9b1d-4033-aebb-eabcd9340222-kube-api-access-7shqk\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-ef6be7-29889795-pf2p2\" (UID: \"e8fa12c9-9b1d-4033-aebb-eabcd9340222\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ef6be7-29889795-pf2p2"
Oct 30 19:15:00.224753 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.223347    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90e98322-b727-4a35-9d93-808e0b01d483-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-a1d55d-29889795-bw7v2\" (UID: \"90e98322-b727-4a35-9d93-808e0b01d483\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a1d55d-29889795-bw7v2"
Oct 30 19:15:00.224753 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.223433    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8fa12c9-9b1d-4033-aebb-eabcd9340222-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-ef6be7-29889795-pf2p2\" (UID: \"e8fa12c9-9b1d-4033-aebb-eabcd9340222\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ef6be7-29889795-pf2p2"
Oct 30 19:15:00.224753 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.223473    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4849dbc2-0043-46f1-ab5e-a76c16f85db9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-c2e3f9-29889795-5rf8p\" (UID: \"4849dbc2-0043-46f1-ab5e-a76c16f85db9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c2e3f9-29889795-5rf8p"
Oct 30 19:15:00.227575 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.227548    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ed012e-29889795-f5xmz"]
Oct 30 19:15:00.233362 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.233335    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ed012e-29889795-f5xmz"
Oct 30 19:15:00.241980 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.239993    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-68b466-29889795-fgfjm"]
Oct 30 19:15:00.252434 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.252405    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-68b466-29889795-fgfjm"
Oct 30 19:15:00.260402 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.254958    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94hcn\""
Oct 30 19:15:00.260402 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.255013    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39bde3-29889795-rzh6s"]
Oct 30 19:15:00.264149 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.263922    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39bde3-29889795-rzh6s"
Oct 30 19:15:00.275995 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.268672    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dcd326-29889795-zj7rm"]
Oct 30 19:15:00.290102 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.289991    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dcd326-29889795-zj7rm"
Oct 30 19:15:00.297326 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.296672    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a1d55d-29889795-bw7v2"]
Oct 30 19:15:00.306719 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.306514    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-baa504-29889795-bwhw4"]
Oct 30 19:15:00.321755 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.321215    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ed012e-29889795-f5xmz"]
Oct 30 19:15:00.321755 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.321333    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-baa504-29889795-bwhw4"
Oct 30 19:15:00.323673 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.322627    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-68b466-29889795-fgfjm"]
Oct 30 19:15:00.324617 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.324595    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5149d31-2979-4353-9b41-df84f90cffbd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-46385c-29889795-hbvv9\" (UID: \"e5149d31-2979-4353-9b41-df84f90cffbd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-46385c-29889795-hbvv9"
Oct 30 19:15:00.324662 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.324631    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfff552f-d7bb-414b-9a43-34922f58d2bd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-dcd326-29889795-zj7rm\" (UID: \"dfff552f-d7bb-414b-9a43-34922f58d2bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dcd326-29889795-zj7rm"
Oct 30 19:15:00.324662 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.324654    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkjr6\" (UniqueName: \"kubernetes.io/projected/e5149d31-2979-4353-9b41-df84f90cffbd-kube-api-access-mkjr6\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-46385c-29889795-hbvv9\" (UID: \"e5149d31-2979-4353-9b41-df84f90cffbd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-46385c-29889795-hbvv9"
Oct 30 19:15:00.324776 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.324673    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqvgj\" (UniqueName: \"kubernetes.io/projected/d3aa2e02-e67a-4418-8fb7-3d805cb7ca7c-kube-api-access-bqvgj\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-ed012e-29889795-f5xmz\" (UID: \"d3aa2e02-e67a-4418-8fb7-3d805cb7ca7c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ed012e-29889795-f5xmz"
Oct 30 19:15:00.324776 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.324758    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfeff9d8-6996-4f87-a933-b41f97627004-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-68b466-29889795-fgfjm\" (UID: \"bfeff9d8-6996-4f87-a933-b41f97627004\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-68b466-29889795-fgfjm"
Oct 30 19:15:00.325447 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.325427    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5149d31-2979-4353-9b41-df84f90cffbd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-46385c-29889795-hbvv9\" (UID: \"e5149d31-2979-4353-9b41-df84f90cffbd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-46385c-29889795-hbvv9"
Oct 30 19:15:00.325654 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.325633    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4xp4\" (UniqueName: \"kubernetes.io/projected/01d8e335-0163-44d7-b811-fb9b99cd285a-kube-api-access-d4xp4\") pod \"maas-api-key-cleanup-e2e-worker-w2-39bde3-29889795-rzh6s\" (UID: \"01d8e335-0163-44d7-b811-fb9b99cd285a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39bde3-29889795-rzh6s"
Oct 30 19:15:00.325771 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.325751    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d89pk\" (UniqueName: \"kubernetes.io/projected/4849dbc2-0043-46f1-ab5e-a76c16f85db9-kube-api-access-d89pk\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-c2e3f9-29889795-5rf8p\" (UID: \"4849dbc2-0043-46f1-ab5e-a76c16f85db9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c2e3f9-29889795-5rf8p"
Oct 30 19:15:00.325846 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.325831    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d3ff890-4ac4-4395-b309-3483be728b66-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-ef6b56-29889795-nl7wc\" (UID: \"8d3ff890-4ac4-4395-b309-3483be728b66\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-ef6b56-29889795-nl7wc"
Oct 30 19:15:00.325881 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.325868    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z7jt\" (UniqueName: \"kubernetes.io/projected/8d3ff890-4ac4-4395-b309-3483be728b66-kube-api-access-9z7jt\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-ef6b56-29889795-nl7wc\" (UID: \"8d3ff890-4ac4-4395-b309-3483be728b66\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-ef6b56-29889795-nl7wc"
Oct 30 19:15:00.325927 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.325911    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgpvw\" (UniqueName: \"kubernetes.io/projected/bfeff9d8-6996-4f87-a933-b41f97627004-kube-api-access-pgpvw\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-68b466-29889795-fgfjm\" (UID: \"bfeff9d8-6996-4f87-a933-b41f97627004\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-68b466-29889795-fgfjm"
Oct 30 19:15:00.326095 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.326049    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7shqk\" (UniqueName: \"kubernetes.io/projected/e8fa12c9-9b1d-4033-aebb-eabcd9340222-kube-api-access-7shqk\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-ef6be7-29889795-pf2p2\" (UID: \"e8fa12c9-9b1d-4033-aebb-eabcd9340222\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ef6be7-29889795-pf2p2"
Oct 30 19:15:00.327428 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.327403    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90e98322-b727-4a35-9d93-808e0b01d483-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-a1d55d-29889795-bw7v2\" (UID: \"90e98322-b727-4a35-9d93-808e0b01d483\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a1d55d-29889795-bw7v2"
Oct 30 19:15:00.327536 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.327511    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8fa12c9-9b1d-4033-aebb-eabcd9340222-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-ef6be7-29889795-pf2p2\" (UID: \"e8fa12c9-9b1d-4033-aebb-eabcd9340222\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ef6be7-29889795-pf2p2"
Oct 30 19:15:00.327594 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.327554    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4849dbc2-0043-46f1-ab5e-a76c16f85db9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-c2e3f9-29889795-5rf8p\" (UID: \"4849dbc2-0043-46f1-ab5e-a76c16f85db9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c2e3f9-29889795-5rf8p"
Oct 30 19:15:00.327669 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.327658    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96f4m\" (UniqueName: \"kubernetes.io/projected/dfff552f-d7bb-414b-9a43-34922f58d2bd-kube-api-access-96f4m\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-dcd326-29889795-zj7rm\" (UID: \"dfff552f-d7bb-414b-9a43-34922f58d2bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dcd326-29889795-zj7rm"
Oct 30 19:15:00.327781 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.327730    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01d8e335-0163-44d7-b811-fb9b99cd285a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-39bde3-29889795-rzh6s\" (UID: \"01d8e335-0163-44d7-b811-fb9b99cd285a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39bde3-29889795-rzh6s"
Oct 30 19:15:00.327781 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.327760    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d3ff890-4ac4-4395-b309-3483be728b66-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-ef6b56-29889795-nl7wc\" (UID: \"8d3ff890-4ac4-4395-b309-3483be728b66\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-ef6b56-29889795-nl7wc"
Oct 30 19:15:00.328150 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.328128    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90e98322-b727-4a35-9d93-808e0b01d483-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-a1d55d-29889795-bw7v2\" (UID: \"90e98322-b727-4a35-9d93-808e0b01d483\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a1d55d-29889795-bw7v2"
Oct 30 19:15:00.328247 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.328229    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4849dbc2-0043-46f1-ab5e-a76c16f85db9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-c2e3f9-29889795-5rf8p\" (UID: \"4849dbc2-0043-46f1-ab5e-a76c16f85db9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c2e3f9-29889795-5rf8p"
Oct 30 19:15:00.328289 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.328254    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wvhv\" (UniqueName: \"kubernetes.io/projected/90e98322-b727-4a35-9d93-808e0b01d483-kube-api-access-6wvhv\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-a1d55d-29889795-bw7v2\" (UID: \"90e98322-b727-4a35-9d93-808e0b01d483\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a1d55d-29889795-bw7v2"
Oct 30 19:15:00.328346 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.328308    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3aa2e02-e67a-4418-8fb7-3d805cb7ca7c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-ed012e-29889795-f5xmz\" (UID: \"d3aa2e02-e67a-4418-8fb7-3d805cb7ca7c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ed012e-29889795-f5xmz"
Oct 30 19:15:00.328658 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.328642    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8fa12c9-9b1d-4033-aebb-eabcd9340222-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-ef6be7-29889795-pf2p2\" (UID: \"e8fa12c9-9b1d-4033-aebb-eabcd9340222\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ef6be7-29889795-pf2p2"
Oct 30 19:15:00.330111 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.330095    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39bde3-29889795-rzh6s"]
Oct 30 19:15:00.336003 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.335983    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9de872-29889795-c7whh"]
Oct 30 19:15:00.343061 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.343018    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9de872-29889795-c7whh"
Oct 30 19:15:00.376646 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.351450    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dcd326-29889795-zj7rm"]
Oct 30 19:15:00.376646 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.352299    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkjr6\" (UniqueName: \"kubernetes.io/projected/e5149d31-2979-4353-9b41-df84f90cffbd-kube-api-access-mkjr6\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-46385c-29889795-hbvv9\" (UID: \"e5149d31-2979-4353-9b41-df84f90cffbd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-46385c-29889795-hbvv9"
Oct 30 19:15:00.376646 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.355366    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7shqk\" (UniqueName: \"kubernetes.io/projected/e8fa12c9-9b1d-4033-aebb-eabcd9340222-kube-api-access-7shqk\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-ef6be7-29889795-pf2p2\" (UID: \"e8fa12c9-9b1d-4033-aebb-eabcd9340222\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ef6be7-29889795-pf2p2"
Oct 30 19:15:00.376646 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.374961    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d89pk\" (UniqueName: \"kubernetes.io/projected/4849dbc2-0043-46f1-ab5e-a76c16f85db9-kube-api-access-d89pk\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-c2e3f9-29889795-5rf8p\" (UID: \"4849dbc2-0043-46f1-ab5e-a76c16f85db9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c2e3f9-29889795-5rf8p"
Oct 30 19:15:00.376646 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.375336    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wvhv\" (UniqueName: \"kubernetes.io/projected/90e98322-b727-4a35-9d93-808e0b01d483-kube-api-access-6wvhv\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-a1d55d-29889795-bw7v2\" (UID: \"90e98322-b727-4a35-9d93-808e0b01d483\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a1d55d-29889795-bw7v2"
Oct 30 19:15:00.377574 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.377548    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z7jt\" (UniqueName: \"kubernetes.io/projected/8d3ff890-4ac4-4395-b309-3483be728b66-kube-api-access-9z7jt\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-ef6b56-29889795-nl7wc\" (UID: \"8d3ff890-4ac4-4395-b309-3483be728b66\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-ef6b56-29889795-nl7wc"
Oct 30 19:15:00.377654 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.377600    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6f956-29889795-vgkfh"]
Oct 30 19:15:00.385123 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.385100    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6f956-29889795-vgkfh"
Oct 30 19:15:00.385291 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.385106    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9de872-29889795-c7whh"]
Oct 30 19:15:00.398825 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.398756    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f3d6a5-29889795-sfmsm"]
Oct 30 19:15:00.404531 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.404510    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f3d6a5-29889795-sfmsm"
Oct 30 19:15:00.406775 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.406752    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-baa504-29889795-bwhw4"]
Oct 30 19:15:00.412359 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.412331    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6f956-29889795-vgkfh"]
Oct 30 19:15:00.417392 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.417369    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f3d6a5-29889795-sfmsm"]
Oct 30 19:15:00.427345 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.427325    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-85cb41-29889795-9wpqw"]
Oct 30 19:15:00.429844 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.429822    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjxcw\" (UniqueName: \"kubernetes.io/projected/e506709f-838a-4255-b3fb-498da9fb0697-kube-api-access-zjxcw\") pod \"maas-api-key-cleanup-e2e-worker-w0-9de872-29889795-c7whh\" (UID: \"e506709f-838a-4255-b3fb-498da9fb0697\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9de872-29889795-c7whh"
Oct 30 19:15:00.429954 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.429854    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2j8n\" (UniqueName: \"kubernetes.io/projected/75f454fd-6c51-42f1-a19d-24d1c739a88c-kube-api-access-k2j8n\") pod \"maas-api-key-cleanup-e2e-extmod-a-d6f956-29889795-vgkfh\" (UID: \"75f454fd-6c51-42f1-a19d-24d1c739a88c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6f956-29889795-vgkfh"
Oct 30 19:15:00.429954 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.429874    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c67ac146-cadf-480c-b1a0-47e60b73d0e5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-f3d6a5-29889795-sfmsm\" (UID: \"c67ac146-cadf-480c-b1a0-47e60b73d0e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f3d6a5-29889795-sfmsm"
Oct 30 19:15:00.430081 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.430057    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96f4m\" (UniqueName: \"kubernetes.io/projected/dfff552f-d7bb-414b-9a43-34922f58d2bd-kube-api-access-96f4m\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-dcd326-29889795-zj7rm\" (UID: \"dfff552f-d7bb-414b-9a43-34922f58d2bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dcd326-29889795-zj7rm"
Oct 30 19:15:00.430123 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.430102    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01d8e335-0163-44d7-b811-fb9b99cd285a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-39bde3-29889795-rzh6s\" (UID: \"01d8e335-0163-44d7-b811-fb9b99cd285a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39bde3-29889795-rzh6s"
Oct 30 19:15:00.430159 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.430134    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e506709f-838a-4255-b3fb-498da9fb0697-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-9de872-29889795-c7whh\" (UID: \"e506709f-838a-4255-b3fb-498da9fb0697\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9de872-29889795-c7whh"
Oct 30 19:15:00.430227 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.430211    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3aa2e02-e67a-4418-8fb7-3d805cb7ca7c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-ed012e-29889795-f5xmz\" (UID: \"d3aa2e02-e67a-4418-8fb7-3d805cb7ca7c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ed012e-29889795-f5xmz"
Oct 30 19:15:00.430290 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.430264    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7590ba0b-baad-4a0a-ad6c-3309f2123c5c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-baa504-29889795-bwhw4\" (UID: \"7590ba0b-baad-4a0a-ad6c-3309f2123c5c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-baa504-29889795-bwhw4"
Oct 30 19:15:00.430370 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.430348    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfff552f-d7bb-414b-9a43-34922f58d2bd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-dcd326-29889795-zj7rm\" (UID: \"dfff552f-d7bb-414b-9a43-34922f58d2bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dcd326-29889795-zj7rm"
Oct 30 19:15:00.430430 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.430391    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqvgj\" (UniqueName: \"kubernetes.io/projected/d3aa2e02-e67a-4418-8fb7-3d805cb7ca7c-kube-api-access-bqvgj\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-ed012e-29889795-f5xmz\" (UID: \"d3aa2e02-e67a-4418-8fb7-3d805cb7ca7c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ed012e-29889795-f5xmz"
Oct 30 19:15:00.430487 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.430466    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfeff9d8-6996-4f87-a933-b41f97627004-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-68b466-29889795-fgfjm\" (UID: \"bfeff9d8-6996-4f87-a933-b41f97627004\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-68b466-29889795-fgfjm"
Oct 30 19:15:00.430552 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.430493    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4xp4\" (UniqueName: \"kubernetes.io/projected/01d8e335-0163-44d7-b811-fb9b99cd285a-kube-api-access-d4xp4\") pod \"maas-api-key-cleanup-e2e-worker-w2-39bde3-29889795-rzh6s\" (UID: \"01d8e335-0163-44d7-b811-fb9b99cd285a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39bde3-29889795-rzh6s"
Oct 30 19:15:00.430552 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.430544    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsv5c\" (UniqueName: \"kubernetes.io/projected/c67ac146-cadf-480c-b1a0-47e60b73d0e5-kube-api-access-gsv5c\") pod \"maas-api-key-cleanup-e2e-extmod-b-f3d6a5-29889795-sfmsm\" (UID: \"c67ac146-cadf-480c-b1a0-47e60b73d0e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f3d6a5-29889795-sfmsm"
Oct 30 19:15:00.430659 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.430629    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjf8x\" (UniqueName: \"kubernetes.io/projected/7590ba0b-baad-4a0a-ad6c-3309f2123c5c-kube-api-access-qjf8x\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-baa504-29889795-bwhw4\" (UID: \"7590ba0b-baad-4a0a-ad6c-3309f2123c5c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-baa504-29889795-bwhw4"
Oct 30 19:15:00.430819 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.430660    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgpvw\" (UniqueName: \"kubernetes.io/projected/bfeff9d8-6996-4f87-a933-b41f97627004-kube-api-access-pgpvw\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-68b466-29889795-fgfjm\" (UID: \"bfeff9d8-6996-4f87-a933-b41f97627004\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-68b466-29889795-fgfjm"
Oct 30 19:15:00.430819 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.430721    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75f454fd-6c51-42f1-a19d-24d1c739a88c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-d6f956-29889795-vgkfh\" (UID: \"75f454fd-6c51-42f1-a19d-24d1c739a88c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6f956-29889795-vgkfh"
Oct 30 19:15:00.430954 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.430928    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01d8e335-0163-44d7-b811-fb9b99cd285a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-39bde3-29889795-rzh6s\" (UID: \"01d8e335-0163-44d7-b811-fb9b99cd285a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39bde3-29889795-rzh6s"
Oct 30 19:15:00.431019 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.430968    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3aa2e02-e67a-4418-8fb7-3d805cb7ca7c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-ed012e-29889795-f5xmz\" (UID: \"d3aa2e02-e67a-4418-8fb7-3d805cb7ca7c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ed012e-29889795-f5xmz"
Oct 30 19:15:00.431163 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.431139    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfff552f-d7bb-414b-9a43-34922f58d2bd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-dcd326-29889795-zj7rm\" (UID: \"dfff552f-d7bb-414b-9a43-34922f58d2bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dcd326-29889795-zj7rm"
Oct 30 19:15:00.431216 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.431160    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfeff9d8-6996-4f87-a933-b41f97627004-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-68b466-29889795-fgfjm\" (UID: \"bfeff9d8-6996-4f87-a933-b41f97627004\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-68b466-29889795-fgfjm"
Oct 30 19:15:00.433294 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.433277    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2d1c71-29889795-p7txw"]
Oct 30 19:15:00.433400 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.433382    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-85cb41-29889795-9wpqw"
Oct 30 19:15:00.437823 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.437806    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-85cb41-29889795-9wpqw"]
Oct 30 19:15:00.437933 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.437905    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2d1c71-29889795-p7txw"
Oct 30 19:15:00.445224 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.445201    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgpvw\" (UniqueName: \"kubernetes.io/projected/bfeff9d8-6996-4f87-a933-b41f97627004-kube-api-access-pgpvw\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-68b466-29889795-fgfjm\" (UID: \"bfeff9d8-6996-4f87-a933-b41f97627004\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-68b466-29889795-fgfjm"
Oct 30 19:15:00.445484 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.445459    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqvgj\" (UniqueName: \"kubernetes.io/projected/d3aa2e02-e67a-4418-8fb7-3d805cb7ca7c-kube-api-access-bqvgj\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-ed012e-29889795-f5xmz\" (UID: \"d3aa2e02-e67a-4418-8fb7-3d805cb7ca7c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ed012e-29889795-f5xmz"
Oct 30 19:15:00.445725 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.445673    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96f4m\" (UniqueName: \"kubernetes.io/projected/dfff552f-d7bb-414b-9a43-34922f58d2bd-kube-api-access-96f4m\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-dcd326-29889795-zj7rm\" (UID: \"dfff552f-d7bb-414b-9a43-34922f58d2bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dcd326-29889795-zj7rm"
Oct 30 19:15:00.445990 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.445969    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4xp4\" (UniqueName: \"kubernetes.io/projected/01d8e335-0163-44d7-b811-fb9b99cd285a-kube-api-access-d4xp4\") pod \"maas-api-key-cleanup-e2e-worker-w2-39bde3-29889795-rzh6s\" (UID: \"01d8e335-0163-44d7-b811-fb9b99cd285a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39bde3-29889795-rzh6s"
Oct 30 19:15:00.449743 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.449717    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2d1c71-29889795-p7txw"]
Oct 30 19:15:00.475668 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.475605    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-ef6b56-29889795-nl7wc"
Oct 30 19:15:00.522394 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.522342    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ef6be7-29889795-pf2p2"
Oct 30 19:15:00.532800 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.532768    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7590ba0b-baad-4a0a-ad6c-3309f2123c5c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-baa504-29889795-bwhw4\" (UID: \"7590ba0b-baad-4a0a-ad6c-3309f2123c5c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-baa504-29889795-bwhw4"
Oct 30 19:15:00.532946 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.532813    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqk7b\" (UniqueName: \"kubernetes.io/projected/403406c5-302f-4563-be3b-63ac8959d517-kube-api-access-rqk7b\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-2d1c71-29889795-p7txw\" (UID: \"403406c5-302f-4563-be3b-63ac8959d517\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2d1c71-29889795-p7txw"
Oct 30 19:15:00.532946 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.532852    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/403406c5-302f-4563-be3b-63ac8959d517-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-2d1c71-29889795-p7txw\" (UID: \"403406c5-302f-4563-be3b-63ac8959d517\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2d1c71-29889795-p7txw"
Oct 30 19:15:00.533022 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.532988    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsv5c\" (UniqueName: \"kubernetes.io/projected/c67ac146-cadf-480c-b1a0-47e60b73d0e5-kube-api-access-gsv5c\") pod \"maas-api-key-cleanup-e2e-extmod-b-f3d6a5-29889795-sfmsm\" (UID: \"c67ac146-cadf-480c-b1a0-47e60b73d0e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f3d6a5-29889795-sfmsm"
Oct 30 19:15:00.533056 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.533045    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjf8x\" (UniqueName: \"kubernetes.io/projected/7590ba0b-baad-4a0a-ad6c-3309f2123c5c-kube-api-access-qjf8x\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-baa504-29889795-bwhw4\" (UID: \"7590ba0b-baad-4a0a-ad6c-3309f2123c5c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-baa504-29889795-bwhw4"
Oct 30 19:15:00.533100 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.533083    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75f454fd-6c51-42f1-a19d-24d1c739a88c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-d6f956-29889795-vgkfh\" (UID: \"75f454fd-6c51-42f1-a19d-24d1c739a88c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6f956-29889795-vgkfh"
Oct 30 19:15:00.533134 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.533118    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4486\" (UniqueName: \"kubernetes.io/projected/3253644f-14ff-452e-851b-694d8724574f-kube-api-access-x4486\") pod \"maas-api-key-cleanup-e2e-worker-w1-85cb41-29889795-9wpqw\" (UID: \"3253644f-14ff-452e-851b-694d8724574f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-85cb41-29889795-9wpqw"
Oct 30 19:15:00.536918 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.533546    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7590ba0b-baad-4a0a-ad6c-3309f2123c5c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-baa504-29889795-bwhw4\" (UID: \"7590ba0b-baad-4a0a-ad6c-3309f2123c5c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-baa504-29889795-bwhw4"
Oct 30 19:15:00.536918 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.534158    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75f454fd-6c51-42f1-a19d-24d1c739a88c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-d6f956-29889795-vgkfh\" (UID: \"75f454fd-6c51-42f1-a19d-24d1c739a88c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6f956-29889795-vgkfh"
Oct 30 19:15:00.536918 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.534875    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjxcw\" (UniqueName: \"kubernetes.io/projected/e506709f-838a-4255-b3fb-498da9fb0697-kube-api-access-zjxcw\") pod \"maas-api-key-cleanup-e2e-worker-w0-9de872-29889795-c7whh\" (UID: \"e506709f-838a-4255-b3fb-498da9fb0697\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9de872-29889795-c7whh"
Oct 30 19:15:00.536918 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.534914    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2j8n\" (UniqueName: \"kubernetes.io/projected/75f454fd-6c51-42f1-a19d-24d1c739a88c-kube-api-access-k2j8n\") pod \"maas-api-key-cleanup-e2e-extmod-a-d6f956-29889795-vgkfh\" (UID: \"75f454fd-6c51-42f1-a19d-24d1c739a88c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6f956-29889795-vgkfh"
Oct 30 19:15:00.536918 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.534940    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c67ac146-cadf-480c-b1a0-47e60b73d0e5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-f3d6a5-29889795-sfmsm\" (UID: \"c67ac146-cadf-480c-b1a0-47e60b73d0e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f3d6a5-29889795-sfmsm"
Oct 30 19:15:00.536918 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.535015    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e506709f-838a-4255-b3fb-498da9fb0697-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-9de872-29889795-c7whh\" (UID: \"e506709f-838a-4255-b3fb-498da9fb0697\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9de872-29889795-c7whh"
Oct 30 19:15:00.536918 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.535089    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3253644f-14ff-452e-851b-694d8724574f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-85cb41-29889795-9wpqw\" (UID: \"3253644f-14ff-452e-851b-694d8724574f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-85cb41-29889795-9wpqw"
Oct 30 19:15:00.536918 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.536186    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c67ac146-cadf-480c-b1a0-47e60b73d0e5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-f3d6a5-29889795-sfmsm\" (UID: \"c67ac146-cadf-480c-b1a0-47e60b73d0e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f3d6a5-29889795-sfmsm"
Oct 30 19:15:00.537584 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.537067    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e506709f-838a-4255-b3fb-498da9fb0697-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-9de872-29889795-c7whh\" (UID: \"e506709f-838a-4255-b3fb-498da9fb0697\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9de872-29889795-c7whh"
Oct 30 19:15:00.547253 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.545151    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsv5c\" (UniqueName: \"kubernetes.io/projected/c67ac146-cadf-480c-b1a0-47e60b73d0e5-kube-api-access-gsv5c\") pod \"maas-api-key-cleanup-e2e-extmod-b-f3d6a5-29889795-sfmsm\" (UID: \"c67ac146-cadf-480c-b1a0-47e60b73d0e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f3d6a5-29889795-sfmsm"
Oct 30 19:15:00.547253 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.546802    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a1d55d-29889795-bw7v2"
Oct 30 19:15:00.552575 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.552514    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2j8n\" (UniqueName: \"kubernetes.io/projected/75f454fd-6c51-42f1-a19d-24d1c739a88c-kube-api-access-k2j8n\") pod \"maas-api-key-cleanup-e2e-extmod-a-d6f956-29889795-vgkfh\" (UID: \"75f454fd-6c51-42f1-a19d-24d1c739a88c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6f956-29889795-vgkfh"
Oct 30 19:15:00.555249 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.554243    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjxcw\" (UniqueName: \"kubernetes.io/projected/e506709f-838a-4255-b3fb-498da9fb0697-kube-api-access-zjxcw\") pod \"maas-api-key-cleanup-e2e-worker-w0-9de872-29889795-c7whh\" (UID: \"e506709f-838a-4255-b3fb-498da9fb0697\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9de872-29889795-c7whh"
Oct 30 19:15:00.579050 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.557573    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjf8x\" (UniqueName: \"kubernetes.io/projected/7590ba0b-baad-4a0a-ad6c-3309f2123c5c-kube-api-access-qjf8x\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-baa504-29889795-bwhw4\" (UID: \"7590ba0b-baad-4a0a-ad6c-3309f2123c5c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-baa504-29889795-bwhw4"
Oct 30 19:15:00.579050 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.562552    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-46385c-29889795-hbvv9"
Oct 30 19:15:00.584241 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.584211    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c2e3f9-29889795-5rf8p"
Oct 30 19:15:00.592934 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.592909    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ed012e-29889795-f5xmz"
Oct 30 19:15:00.608735 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.608704    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-68b466-29889795-fgfjm"
Oct 30 19:15:00.630895 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.630862    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39bde3-29889795-rzh6s"
Oct 30 19:15:00.639461 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.636767    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4486\" (UniqueName: \"kubernetes.io/projected/3253644f-14ff-452e-851b-694d8724574f-kube-api-access-x4486\") pod \"maas-api-key-cleanup-e2e-worker-w1-85cb41-29889795-9wpqw\" (UID: \"3253644f-14ff-452e-851b-694d8724574f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-85cb41-29889795-9wpqw"
Oct 30 19:15:00.639461 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.637045    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3253644f-14ff-452e-851b-694d8724574f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-85cb41-29889795-9wpqw\" (UID: \"3253644f-14ff-452e-851b-694d8724574f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-85cb41-29889795-9wpqw"
Oct 30 19:15:00.639461 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.637098    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqk7b\" (UniqueName: \"kubernetes.io/projected/403406c5-302f-4563-be3b-63ac8959d517-kube-api-access-rqk7b\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-2d1c71-29889795-p7txw\" (UID: \"403406c5-302f-4563-be3b-63ac8959d517\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2d1c71-29889795-p7txw"
Oct 30 19:15:00.639461 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.637130    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/403406c5-302f-4563-be3b-63ac8959d517-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-2d1c71-29889795-p7txw\" (UID: \"403406c5-302f-4563-be3b-63ac8959d517\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2d1c71-29889795-p7txw"
Oct 30 19:15:00.639461 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.637989    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/403406c5-302f-4563-be3b-63ac8959d517-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-2d1c71-29889795-p7txw\" (UID: \"403406c5-302f-4563-be3b-63ac8959d517\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2d1c71-29889795-p7txw"
Oct 30 19:15:00.639461 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.638954    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3253644f-14ff-452e-851b-694d8724574f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-85cb41-29889795-9wpqw\" (UID: \"3253644f-14ff-452e-851b-694d8724574f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-85cb41-29889795-9wpqw"
Oct 30 19:15:00.652283 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.651292    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dcd326-29889795-zj7rm"
Oct 30 19:15:00.680827 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.673702    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4486\" (UniqueName: \"kubernetes.io/projected/3253644f-14ff-452e-851b-694d8724574f-kube-api-access-x4486\") pod \"maas-api-key-cleanup-e2e-worker-w1-85cb41-29889795-9wpqw\" (UID: \"3253644f-14ff-452e-851b-694d8724574f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-85cb41-29889795-9wpqw"
Oct 30 19:15:00.701006 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.700974    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-baa504-29889795-bwhw4"
Oct 30 19:15:00.708039 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.707390    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqk7b\" (UniqueName: \"kubernetes.io/projected/403406c5-302f-4563-be3b-63ac8959d517-kube-api-access-rqk7b\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-2d1c71-29889795-p7txw\" (UID: \"403406c5-302f-4563-be3b-63ac8959d517\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2d1c71-29889795-p7txw"
Oct 30 19:15:00.715205 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.714753    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9de872-29889795-c7whh"
Oct 30 19:15:00.730234 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.729043    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6f956-29889795-vgkfh"
Oct 30 19:15:00.751911 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.750863    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f3d6a5-29889795-sfmsm"
Oct 30 19:15:00.808159 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.793466    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2d1c71-29889795-p7txw"
Oct 30 19:15:00.808159 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.794291    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-85cb41-29889795-9wpqw"
Oct 30 19:15:00.808159 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.801092    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-ef6b56-29889795-nl7wc"]
Oct 30 19:15:00.834517 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:15:00.834469    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d3ff890_4ac4_4395_b309_3483be728b66.slice/crio-66dadb143e15595d94cdb01893bf2578b6caec9a9e67211295cc75d8c7a5d3d4 WatchSource:0}: Error finding container 66dadb143e15595d94cdb01893bf2578b6caec9a9e67211295cc75d8c7a5d3d4: Status 404 returned error can't find the container with id 66dadb143e15595d94cdb01893bf2578b6caec9a9e67211295cc75d8c7a5d3d4
Oct 30 19:15:00.977301 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:00.971249    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-ef6b56-29889795-nl7wc" event={"ID":"8d3ff890-4ac4-4395-b309-3483be728b66","Type":"ContainerStarted","Data":"66dadb143e15595d94cdb01893bf2578b6caec9a9e67211295cc75d8c7a5d3d4"}
Oct 30 19:15:01.115772 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:01.114145    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ef6be7-29889795-pf2p2"]
Oct 30 19:15:01.368597 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:01.368567    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-46385c-29889795-hbvv9"]
Oct 30 19:15:01.380125 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:01.380096    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a1d55d-29889795-bw7v2"]
Oct 30 19:15:01.384593 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:15:01.384556    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5149d31_2979_4353_9b41_df84f90cffbd.slice/crio-c417b4934eb234f909be133b3e9d900d5fb25c41464df24babab393227f44ac3 WatchSource:0}: Error finding container c417b4934eb234f909be133b3e9d900d5fb25c41464df24babab393227f44ac3: Status 404 returned error can't find the container with id c417b4934eb234f909be133b3e9d900d5fb25c41464df24babab393227f44ac3
Oct 30 19:15:01.386054 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:15:01.386026    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90e98322_b727_4a35_9d93_808e0b01d483.slice/crio-635d8514818f3ceaec5c108b24d37b903e0ea0b19677a5757e78e2263ae56448 WatchSource:0}: Error finding container 635d8514818f3ceaec5c108b24d37b903e0ea0b19677a5757e78e2263ae56448: Status 404 returned error can't find the container with id 635d8514818f3ceaec5c108b24d37b903e0ea0b19677a5757e78e2263ae56448
Oct 30 19:15:01.387037 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:01.386955    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c2e3f9-29889795-5rf8p"]
Oct 30 19:15:01.431384 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:01.431352    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-68b466-29889795-fgfjm"]
Oct 30 19:15:01.455193 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:01.455164    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39bde3-29889795-rzh6s"]
Oct 30 19:15:01.466100 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:15:01.466073    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01d8e335_0163_44d7_b811_fb9b99cd285a.slice/crio-5ee5eb6a1b3ef452b0281d4c097c2430af88ebedd827ac4fbed52dc77b70b7b6 WatchSource:0}: Error finding container 5ee5eb6a1b3ef452b0281d4c097c2430af88ebedd827ac4fbed52dc77b70b7b6: Status 404 returned error can't find the container with id 5ee5eb6a1b3ef452b0281d4c097c2430af88ebedd827ac4fbed52dc77b70b7b6
Oct 30 19:15:01.473852 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:01.473829    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9de872-29889795-c7whh"]
Oct 30 19:15:01.480433 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:01.480410    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ed012e-29889795-f5xmz"]
Oct 30 19:15:01.491836 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:01.491801    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dcd326-29889795-zj7rm"]
Oct 30 19:15:01.518110 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:01.518075    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6f956-29889795-vgkfh"]
Oct 30 19:15:01.553533 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:01.553507    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e395ae73-f217-48d0-a277-6798e8221162" path="/var/lib/kubelet/pods/e395ae73-f217-48d0-a277-6798e8221162/volumes"
Oct 30 19:15:01.553963 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:01.553941    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f3d6a5-29889795-sfmsm"]
Oct 30 19:15:01.556784 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:01.556759    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-85cb41-29889795-9wpqw"]
Oct 30 19:15:01.561528 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:01.561508    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2d1c71-29889795-p7txw"]
Oct 30 19:15:01.582073 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:15:01.582049    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75f454fd_6c51_42f1_a19d_24d1c739a88c.slice/crio-170c71440778bac390ea7f4352f77ee39ad60b6c0cff1a980346b5665fa6445f WatchSource:0}: Error finding container 170c71440778bac390ea7f4352f77ee39ad60b6c0cff1a980346b5665fa6445f: Status 404 returned error can't find the container with id 170c71440778bac390ea7f4352f77ee39ad60b6c0cff1a980346b5665fa6445f
Oct 30 19:15:01.628195 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:01.628164    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-baa504-29889795-bwhw4"]
Oct 30 19:15:01.692570 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:15:01.692542    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc67ac146_cadf_480c_b1a0_47e60b73d0e5.slice/crio-a2a4ded5502a00b60df717c0018331aabd836e61e8a49c111c9ca6bc19f1dadc WatchSource:0}: Error finding container a2a4ded5502a00b60df717c0018331aabd836e61e8a49c111c9ca6bc19f1dadc: Status 404 returned error can't find the container with id a2a4ded5502a00b60df717c0018331aabd836e61e8a49c111c9ca6bc19f1dadc
Oct 30 19:15:01.693650 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:15:01.693621    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3253644f_14ff_452e_851b_694d8724574f.slice/crio-8f6676411f7238d87495e3e4532ad49b28f60d29923269a05bde9c9defb19892 WatchSource:0}: Error finding container 8f6676411f7238d87495e3e4532ad49b28f60d29923269a05bde9c9defb19892: Status 404 returned error can't find the container with id 8f6676411f7238d87495e3e4532ad49b28f60d29923269a05bde9c9defb19892
Oct 30 19:15:01.990379 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:01.990343    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6f956-29889795-vgkfh" event={"ID":"75f454fd-6c51-42f1-a19d-24d1c739a88c","Type":"ContainerStarted","Data":"170c71440778bac390ea7f4352f77ee39ad60b6c0cff1a980346b5665fa6445f"}
Oct 30 19:15:01.992094 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:01.992068    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dcd326-29889795-zj7rm" event={"ID":"dfff552f-d7bb-414b-9a43-34922f58d2bd","Type":"ContainerStarted","Data":"0eeacc2258bce752d44fe4cefeeac433c4ca468f0815cadec11a0ac94fd215ad"}
Oct 30 19:15:01.993603 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:01.993578    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c2e3f9-29889795-5rf8p" event={"ID":"4849dbc2-0043-46f1-ab5e-a76c16f85db9","Type":"ContainerStarted","Data":"170f54d843621af69506bce3f068997bd4163bb22dfb2bfee334323dd26b200d"}
Oct 30 19:15:01.995658 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:01.995632    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a1d55d-29889795-bw7v2" event={"ID":"90e98322-b727-4a35-9d93-808e0b01d483","Type":"ContainerStarted","Data":"635d8514818f3ceaec5c108b24d37b903e0ea0b19677a5757e78e2263ae56448"}
Oct 30 19:15:01.997539 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:01.997519    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-baa504-29889795-bwhw4" event={"ID":"7590ba0b-baad-4a0a-ad6c-3309f2123c5c","Type":"ContainerStarted","Data":"b7ff9f2956bf15923b321b9dfc1be26068aea48eaa160eec547aa148ea929def"}
Oct 30 19:15:01.998916 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:01.998885    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f3d6a5-29889795-sfmsm" event={"ID":"c67ac146-cadf-480c-b1a0-47e60b73d0e5","Type":"ContainerStarted","Data":"a2a4ded5502a00b60df717c0018331aabd836e61e8a49c111c9ca6bc19f1dadc"}
Oct 30 19:15:02.000461 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:02.000443    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ef6be7-29889795-pf2p2" event={"ID":"e8fa12c9-9b1d-4033-aebb-eabcd9340222","Type":"ContainerStarted","Data":"a85cad846748772270d87e7d52bb0fdb495ee2dbbfcf4d6690904fc421663855"}
Oct 30 19:15:02.000584 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:02.000571    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ef6be7-29889795-pf2p2" event={"ID":"e8fa12c9-9b1d-4033-aebb-eabcd9340222","Type":"ContainerStarted","Data":"3dcdd4d795a879aacaf8883b61ecf9a49b4bd3204f9ea5726f456e99b9add72d"}
Oct 30 19:15:02.002279 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:02.002258    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9de872-29889795-c7whh" event={"ID":"e506709f-838a-4255-b3fb-498da9fb0697","Type":"ContainerStarted","Data":"745203aa516ddccab003daf04f27ca2d666d8dfd907df91f0db2ecdb9de0989b"}
Oct 30 19:15:02.005293 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:02.005271    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39bde3-29889795-rzh6s" event={"ID":"01d8e335-0163-44d7-b811-fb9b99cd285a","Type":"ContainerStarted","Data":"5ee5eb6a1b3ef452b0281d4c097c2430af88ebedd827ac4fbed52dc77b70b7b6"}
Oct 30 19:15:02.006591 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:02.006570    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-85cb41-29889795-9wpqw" event={"ID":"3253644f-14ff-452e-851b-694d8724574f","Type":"ContainerStarted","Data":"8f6676411f7238d87495e3e4532ad49b28f60d29923269a05bde9c9defb19892"}
Oct 30 19:15:02.007932 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:02.007907    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2d1c71-29889795-p7txw" event={"ID":"403406c5-302f-4563-be3b-63ac8959d517","Type":"ContainerStarted","Data":"6b623b63e028314a893df4dc17eafa6ce4f7d0bc19ab9d44c0552cdf5d19fba9"}
Oct 30 19:15:02.009384 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:02.009361    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-ef6b56-29889795-nl7wc" event={"ID":"8d3ff890-4ac4-4395-b309-3483be728b66","Type":"ContainerStarted","Data":"d9e9b3a6ee0b8a1ab56f32ba0cab339886bfae3870fe819ac9549bd3a1e6db78"}
Oct 30 19:15:02.010601 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:02.010580    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-68b466-29889795-fgfjm" event={"ID":"bfeff9d8-6996-4f87-a933-b41f97627004","Type":"ContainerStarted","Data":"067e08521281614add4d2b29c77d7e746281e0b9c10ef0b0a5505a8a738381b9"}
Oct 30 19:15:02.011846 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:02.011776    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ed012e-29889795-f5xmz" event={"ID":"d3aa2e02-e67a-4418-8fb7-3d805cb7ca7c","Type":"ContainerStarted","Data":"879f81e9f1e85325437817bc417ecc3ad4adb3bd0b7fdc74a2b5b7db7d6e020c"}
Oct 30 19:15:02.013227 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:02.013209    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-46385c-29889795-hbvv9" event={"ID":"e5149d31-2979-4353-9b41-df84f90cffbd","Type":"ContainerStarted","Data":"c417b4934eb234f909be133b3e9d900d5fb25c41464df24babab393227f44ac3"}
Oct 30 19:15:02.016159 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:02.016120    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ef6be7-29889795-pf2p2" podStartSLOduration=2.016109635 podStartE2EDuration="2.016109635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:02.01515597 +0000 UTC m=+2324.640825936" watchObservedRunningTime="2026-10-30 19:15:02.016109635 +0000 UTC m=+2324.641779600"
Oct 30 19:15:02.036134 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:02.036085    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-ef6b56-29889795-nl7wc" podStartSLOduration=2.036073555 podStartE2EDuration="2.036073555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:02.033424197 +0000 UTC m=+2324.659094162" watchObservedRunningTime="2026-10-30 19:15:02.036073555 +0000 UTC m=+2324.661743574"
Oct 30 19:15:03.024746 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:03.024711    4525 generic.go:356] "Generic (PLEG): container finished" podID="8d3ff890-4ac4-4395-b309-3483be728b66" containerID="d9e9b3a6ee0b8a1ab56f32ba0cab339886bfae3870fe819ac9549bd3a1e6db78" exitCode=60
Oct 30 19:15:03.025207 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:03.024801    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-ef6b56-29889795-nl7wc" event={"ID":"8d3ff890-4ac4-4395-b309-3483be728b66","Type":"ContainerDied","Data":"d9e9b3a6ee0b8a1ab56f32ba0cab339886bfae3870fe819ac9549bd3a1e6db78"}
Oct 30 19:15:03.025207 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:03.025134    4525 scope.go:122] "RemoveContainer" containerID="d9e9b3a6ee0b8a1ab56f32ba0cab339886bfae3870fe819ac9549bd3a1e6db78"
Oct 30 19:15:03.026956 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:03.026933    4525 generic.go:356] "Generic (PLEG): container finished" podID="bfeff9d8-6996-4f87-a933-b41f97627004" containerID="a1f0c058fa8445127cc7916103a8a120b3ccd2f52db8877ca91d18b8cd833eb4" exitCode=60
Oct 30 19:15:03.027108 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:03.027016    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-68b466-29889795-fgfjm" event={"ID":"bfeff9d8-6996-4f87-a933-b41f97627004","Type":"ContainerDied","Data":"a1f0c058fa8445127cc7916103a8a120b3ccd2f52db8877ca91d18b8cd833eb4"}
Oct 30 19:15:03.027307 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:03.027293    4525 scope.go:122] "RemoveContainer" containerID="a1f0c058fa8445127cc7916103a8a120b3ccd2f52db8877ca91d18b8cd833eb4"
Oct 30 19:15:03.029252 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:03.029229    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ed012e-29889795-f5xmz" event={"ID":"d3aa2e02-e67a-4418-8fb7-3d805cb7ca7c","Type":"ContainerStarted","Data":"02a994d582066c0c17c2cbb9529a5e6c386dabdca020c91b57c27810dd1ac830"}
Oct 30 19:15:03.034062 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:03.031153    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-46385c-29889795-hbvv9" event={"ID":"e5149d31-2979-4353-9b41-df84f90cffbd","Type":"ContainerStarted","Data":"b970bdc5e20ebdca7b18f452684547878e11c97d622b44738a8e95e4d80b1f38"}
Oct 30 19:15:03.034062 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:03.033265    4525 generic.go:356] "Generic (PLEG): container finished" podID="75f454fd-6c51-42f1-a19d-24d1c739a88c" containerID="bde160edd21d0cf3d038ffab3f359e472df2848569c76cc3fd0034dc68a8ed7c" exitCode=60
Oct 30 19:15:03.034062 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:03.033312    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6f956-29889795-vgkfh" event={"ID":"75f454fd-6c51-42f1-a19d-24d1c739a88c","Type":"ContainerDied","Data":"bde160edd21d0cf3d038ffab3f359e472df2848569c76cc3fd0034dc68a8ed7c"}
Oct 30 19:15:03.034062 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:03.033553    4525 scope.go:122] "RemoveContainer" containerID="bde160edd21d0cf3d038ffab3f359e472df2848569c76cc3fd0034dc68a8ed7c"
Oct 30 19:15:03.035167 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:03.035118    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dcd326-29889795-zj7rm" event={"ID":"dfff552f-d7bb-414b-9a43-34922f58d2bd","Type":"ContainerStarted","Data":"e2b412c6d4a660a15743a37c2102414e3f036268d19e6357ffa11f5fbf7ab9e6"}
Oct 30 19:15:03.037095 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:03.037069    4525 generic.go:356] "Generic (PLEG): container finished" podID="4849dbc2-0043-46f1-ab5e-a76c16f85db9" containerID="e7927bc7bbef5308302236310a5484c9b5c78d02464b206b43cb2ebb24def1c8" exitCode=60
Oct 30 19:15:03.037187 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:03.037172    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c2e3f9-29889795-5rf8p" event={"ID":"4849dbc2-0043-46f1-ab5e-a76c16f85db9","Type":"ContainerDied","Data":"e7927bc7bbef5308302236310a5484c9b5c78d02464b206b43cb2ebb24def1c8"}
Oct 30 19:15:03.037348 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:03.037329    4525 scope.go:122] "RemoveContainer" containerID="e7927bc7bbef5308302236310a5484c9b5c78d02464b206b43cb2ebb24def1c8"
Oct 30 19:15:03.039157 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:03.039133    4525 generic.go:356] "Generic (PLEG): container finished" podID="90e98322-b727-4a35-9d93-808e0b01d483" containerID="c182f62c13c34edba23bbce05d93a58a4395141026e897ea79cae54490baac08" exitCode=60
Oct 30 19:15:03.039278 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:03.039215    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a1d55d-29889795-bw7v2" event={"ID":"90e98322-b727-4a35-9d93-808e0b01d483","Type":"ContainerDied","Data":"c182f62c13c34edba23bbce05d93a58a4395141026e897ea79cae54490baac08"}
Oct 30 19:15:03.039490 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:03.039469    4525 scope.go:122] "RemoveContainer" containerID="c182f62c13c34edba23bbce05d93a58a4395141026e897ea79cae54490baac08"
Oct 30 19:15:03.041065 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:03.041034    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-baa504-29889795-bwhw4" event={"ID":"7590ba0b-baad-4a0a-ad6c-3309f2123c5c","Type":"ContainerStarted","Data":"c0b8c761f56479ea3c417deea8a9e3df0827c40f5e53a1f9eca9716d4f9a6509"}
Oct 30 19:15:03.047017 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:03.044695    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f3d6a5-29889795-sfmsm" event={"ID":"c67ac146-cadf-480c-b1a0-47e60b73d0e5","Type":"ContainerStarted","Data":"f22d3e9a9c67630caa899259823ef232643df8c8b7b18600b40305511171f230"}
Oct 30 19:15:03.047746 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:03.047723    4525 generic.go:356] "Generic (PLEG): container finished" podID="e8fa12c9-9b1d-4033-aebb-eabcd9340222" containerID="a85cad846748772270d87e7d52bb0fdb495ee2dbbfcf4d6690904fc421663855" exitCode=60
Oct 30 19:15:03.047922 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:03.047903    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ef6be7-29889795-pf2p2" event={"ID":"e8fa12c9-9b1d-4033-aebb-eabcd9340222","Type":"ContainerDied","Data":"a85cad846748772270d87e7d52bb0fdb495ee2dbbfcf4d6690904fc421663855"}
Oct 30 19:15:03.048144 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:03.048126    4525 scope.go:122] "RemoveContainer" containerID="a85cad846748772270d87e7d52bb0fdb495ee2dbbfcf4d6690904fc421663855"
Oct 30 19:15:03.050042 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:03.050021    4525 generic.go:356] "Generic (PLEG): container finished" podID="e506709f-838a-4255-b3fb-498da9fb0697" containerID="5bf93b69f0676d4dd71e4ca164b1eb3eae7480c4d5dffeddf18cea1d61300f66" exitCode=60
Oct 30 19:15:03.050138 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:03.050093    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9de872-29889795-c7whh" event={"ID":"e506709f-838a-4255-b3fb-498da9fb0697","Type":"ContainerDied","Data":"5bf93b69f0676d4dd71e4ca164b1eb3eae7480c4d5dffeddf18cea1d61300f66"}
Oct 30 19:15:03.050325 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:03.050307    4525 scope.go:122] "RemoveContainer" containerID="5bf93b69f0676d4dd71e4ca164b1eb3eae7480c4d5dffeddf18cea1d61300f66"
Oct 30 19:15:03.059743 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:03.059706    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39bde3-29889795-rzh6s" event={"ID":"01d8e335-0163-44d7-b811-fb9b99cd285a","Type":"ContainerStarted","Data":"7887a3f1ecce11f0024ef00c7405673dab634853c040ffcfbe17d75231656a6d"}
Oct 30 19:15:03.063828 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:03.063802    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-85cb41-29889795-9wpqw" event={"ID":"3253644f-14ff-452e-851b-694d8724574f","Type":"ContainerStarted","Data":"fa7f2ad6ea407e35a0d08af96429f43d89ab95282ebcbebafb50604c27d205c8"}
Oct 30 19:15:03.066634 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:03.066596    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2d1c71-29889795-p7txw" event={"ID":"403406c5-302f-4563-be3b-63ac8959d517","Type":"ContainerStarted","Data":"38e15cc1581a4c3b8f2f15c91f3fcb46d88b394786f989582ba0f6f7a431066b"}
Oct 30 19:15:03.524317 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:03.381842    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-baa504-29889795-bwhw4" podStartSLOduration=3.381823624 podStartE2EDuration="3.381823624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:03.106456567 +0000 UTC m=+2325.732126532" watchObservedRunningTime="2026-10-30 19:15:03.381823624 +0000 UTC m=+2326.007493593"
Oct 30 19:15:03.577962 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:03.577899    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dcd326-29889795-zj7rm" podStartSLOduration=3.577879717 podStartE2EDuration="3.577879717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:03.21117758 +0000 UTC m=+2325.836847544" watchObservedRunningTime="2026-10-30 19:15:03.577879717 +0000 UTC m=+2326.203549682"
Oct 30 19:15:03.584759 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:03.584633    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ed012e-29889795-f5xmz" podStartSLOduration=3.5846145160000003 podStartE2EDuration="3.584614516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:03.263287544 +0000 UTC m=+2325.888957524" watchObservedRunningTime="2026-10-30 19:15:03.584614516 +0000 UTC m=+2326.210284481"
Oct 30 19:15:03.588342 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:03.588291    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-46385c-29889795-hbvv9" podStartSLOduration=3.588280004 podStartE2EDuration="3.588280004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:03.327847528 +0000 UTC m=+2325.953517493" watchObservedRunningTime="2026-10-30 19:15:03.588280004 +0000 UTC m=+2326.213949967"
Oct 30 19:15:03.803472 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:03.803445    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e67d95-28v72"]
Oct 30 19:15:03.803709 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:03.803668    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e67d95-28v72"]
Oct 30 19:15:03.807813 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:03.805214    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39bde3-29889795-rzh6s" podStartSLOduration=3.805201377 podStartE2EDuration="3.805201377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:03.568996776 +0000 UTC m=+2326.194666744" watchObservedRunningTime="2026-10-30 19:15:03.805201377 +0000 UTC m=+2326.430871341"
Oct 30 19:15:03.812952 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:03.812894    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f3d6a5-29889795-sfmsm" podStartSLOduration=3.812876175 podStartE2EDuration="3.812876175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:03.6033485 +0000 UTC m=+2326.229018466" watchObservedRunningTime="2026-10-30 19:15:03.812876175 +0000 UTC m=+2326.438546152"
Oct 30 19:15:03.820998 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:03.820912    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-85cb41-29889795-9wpqw" podStartSLOduration=3.820892718 podStartE2EDuration="3.820892718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:03.640918741 +0000 UTC m=+2326.266588707" watchObservedRunningTime="2026-10-30 19:15:03.820892718 +0000 UTC m=+2326.446562683"
Oct 30 19:15:03.834672 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:03.834625    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2d1c71-29889795-p7txw" podStartSLOduration=3.834612595 podStartE2EDuration="3.834612595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:03.768420272 +0000 UTC m=+2326.394090240" watchObservedRunningTime="2026-10-30 19:15:03.834612595 +0000 UTC m=+2326.460282559"
Oct 30 19:15:03.872731 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:15:03.872651    4525 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3aa2e02_e67a_4418_8fb7_3d805cb7ca7c.slice/crio-conmon-02a994d582066c0c17c2cbb9529a5e6c386dabdca020c91b57c27810dd1ac830.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfff552f_d7bb_414b_9a43_34922f58d2bd.slice/crio-conmon-e2b412c6d4a660a15743a37c2102414e3f036268d19e6357ffa11f5fbf7ab9e6.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3aa2e02_e67a_4418_8fb7_3d805cb7ca7c.slice/crio-02a994d582066c0c17c2cbb9529a5e6c386dabdca020c91b57c27810dd1ac830.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc67ac146_cadf_480c_b1a0_47e60b73d0e5.slice/crio-conmon-f22d3e9a9c67630caa899259823ef232643df8c8b7b18600b40305511171f230.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 19:15:04.079499 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:04.079466    4525 generic.go:356] "Generic (PLEG): container finished" podID="d3aa2e02-e67a-4418-8fb7-3d805cb7ca7c" containerID="02a994d582066c0c17c2cbb9529a5e6c386dabdca020c91b57c27810dd1ac830" exitCode=60
Oct 30 19:15:04.080061 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:04.079555    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ed012e-29889795-f5xmz" event={"ID":"d3aa2e02-e67a-4418-8fb7-3d805cb7ca7c","Type":"ContainerDied","Data":"02a994d582066c0c17c2cbb9529a5e6c386dabdca020c91b57c27810dd1ac830"}
Oct 30 19:15:04.080061 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:04.079916    4525 scope.go:122] "RemoveContainer" containerID="02a994d582066c0c17c2cbb9529a5e6c386dabdca020c91b57c27810dd1ac830"
Oct 30 19:15:04.081467 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:04.081444    4525 generic.go:356] "Generic (PLEG): container finished" podID="e5149d31-2979-4353-9b41-df84f90cffbd" containerID="b970bdc5e20ebdca7b18f452684547878e11c97d622b44738a8e95e4d80b1f38" exitCode=60
Oct 30 19:15:04.081638 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:04.081533    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-46385c-29889795-hbvv9" event={"ID":"e5149d31-2979-4353-9b41-df84f90cffbd","Type":"ContainerDied","Data":"b970bdc5e20ebdca7b18f452684547878e11c97d622b44738a8e95e4d80b1f38"}
Oct 30 19:15:04.081812 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:04.081795    4525 scope.go:122] "RemoveContainer" containerID="b970bdc5e20ebdca7b18f452684547878e11c97d622b44738a8e95e4d80b1f38"
Oct 30 19:15:04.084151 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:04.084133    4525 generic.go:356] "Generic (PLEG): container finished" podID="dfff552f-d7bb-414b-9a43-34922f58d2bd" containerID="e2b412c6d4a660a15743a37c2102414e3f036268d19e6357ffa11f5fbf7ab9e6" exitCode=60
Oct 30 19:15:04.084288 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:04.084265    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dcd326-29889795-zj7rm" event={"ID":"dfff552f-d7bb-414b-9a43-34922f58d2bd","Type":"ContainerDied","Data":"e2b412c6d4a660a15743a37c2102414e3f036268d19e6357ffa11f5fbf7ab9e6"}
Oct 30 19:15:04.084478 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:04.084463    4525 scope.go:122] "RemoveContainer" containerID="e2b412c6d4a660a15743a37c2102414e3f036268d19e6357ffa11f5fbf7ab9e6"
Oct 30 19:15:04.086475 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:04.086454    4525 generic.go:356] "Generic (PLEG): container finished" podID="7590ba0b-baad-4a0a-ad6c-3309f2123c5c" containerID="c0b8c761f56479ea3c417deea8a9e3df0827c40f5e53a1f9eca9716d4f9a6509" exitCode=60
Oct 30 19:15:04.086565 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:04.086477    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-baa504-29889795-bwhw4" event={"ID":"7590ba0b-baad-4a0a-ad6c-3309f2123c5c","Type":"ContainerDied","Data":"c0b8c761f56479ea3c417deea8a9e3df0827c40f5e53a1f9eca9716d4f9a6509"}
Oct 30 19:15:04.086806 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:04.086784    4525 scope.go:122] "RemoveContainer" containerID="c0b8c761f56479ea3c417deea8a9e3df0827c40f5e53a1f9eca9716d4f9a6509"
Oct 30 19:15:04.088403 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:04.088382    4525 generic.go:356] "Generic (PLEG): container finished" podID="c67ac146-cadf-480c-b1a0-47e60b73d0e5" containerID="f22d3e9a9c67630caa899259823ef232643df8c8b7b18600b40305511171f230" exitCode=60
Oct 30 19:15:04.088530 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:04.088493    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f3d6a5-29889795-sfmsm" event={"ID":"c67ac146-cadf-480c-b1a0-47e60b73d0e5","Type":"ContainerDied","Data":"f22d3e9a9c67630caa899259823ef232643df8c8b7b18600b40305511171f230"}
Oct 30 19:15:04.088747 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:04.088732    4525 scope.go:122] "RemoveContainer" containerID="f22d3e9a9c67630caa899259823ef232643df8c8b7b18600b40305511171f230"
Oct 30 19:15:04.090338 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:04.090315    4525 generic.go:356] "Generic (PLEG): container finished" podID="01d8e335-0163-44d7-b811-fb9b99cd285a" containerID="7887a3f1ecce11f0024ef00c7405673dab634853c040ffcfbe17d75231656a6d" exitCode=60
Oct 30 19:15:04.090434 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:04.090395    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39bde3-29889795-rzh6s" event={"ID":"01d8e335-0163-44d7-b811-fb9b99cd285a","Type":"ContainerDied","Data":"7887a3f1ecce11f0024ef00c7405673dab634853c040ffcfbe17d75231656a6d"}
Oct 30 19:15:04.090644 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:04.090630    4525 scope.go:122] "RemoveContainer" containerID="7887a3f1ecce11f0024ef00c7405673dab634853c040ffcfbe17d75231656a6d"
Oct 30 19:15:04.092279 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:04.092256    4525 generic.go:356] "Generic (PLEG): container finished" podID="3253644f-14ff-452e-851b-694d8724574f" containerID="fa7f2ad6ea407e35a0d08af96429f43d89ab95282ebcbebafb50604c27d205c8" exitCode=60
Oct 30 19:15:04.092486 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:04.092463    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-85cb41-29889795-9wpqw" event={"ID":"3253644f-14ff-452e-851b-694d8724574f","Type":"ContainerDied","Data":"fa7f2ad6ea407e35a0d08af96429f43d89ab95282ebcbebafb50604c27d205c8"}
Oct 30 19:15:04.092598 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:04.092522    4525 scope.go:122] "RemoveContainer" containerID="fa7f2ad6ea407e35a0d08af96429f43d89ab95282ebcbebafb50604c27d205c8"
Oct 30 19:15:04.095361 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:04.095334    4525 generic.go:356] "Generic (PLEG): container finished" podID="403406c5-302f-4563-be3b-63ac8959d517" containerID="38e15cc1581a4c3b8f2f15c91f3fcb46d88b394786f989582ba0f6f7a431066b" exitCode=60
Oct 30 19:15:04.095956 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:04.095363    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2d1c71-29889795-p7txw" event={"ID":"403406c5-302f-4563-be3b-63ac8959d517","Type":"ContainerDied","Data":"38e15cc1581a4c3b8f2f15c91f3fcb46d88b394786f989582ba0f6f7a431066b"}
Oct 30 19:15:04.095956 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:04.095747    4525 scope.go:122] "RemoveContainer" containerID="38e15cc1581a4c3b8f2f15c91f3fcb46d88b394786f989582ba0f6f7a431066b"
Oct 30 19:15:05.456993 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:05.456906    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2d1c71-29889795-p7txw" event={"ID":"403406c5-302f-4563-be3b-63ac8959d517","Type":"ContainerStarted","Data":"c8c24555076ec9b77fba843829b23ccbfde748b78f94fd62b04286465bb42423"}
Oct 30 19:15:05.769319 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:05.769288    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d8d8931-62df-4486-b331-f5c5013366e8" path="/var/lib/kubelet/pods/3d8d8931-62df-4486-b331-f5c5013366e8/volumes"
Oct 30 19:15:05.807049 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:05.806992    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c2e3f9-29889795-5rf8p" podStartSLOduration=5.806977441 podStartE2EDuration="5.806977441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:05.806401581 +0000 UTC m=+2328.432071546" watchObservedRunningTime="2026-10-30 19:15:05.806977441 +0000 UTC m=+2328.432647406"
Oct 30 19:15:06.785604 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:06.785515    4525 generic.go:356] "Generic (PLEG): container finished" podID="c67ac146-cadf-480c-b1a0-47e60b73d0e5" containerID="2fc4e1ec07256c28e0f65ee312a3065750e3717bbaad7d98d9667e316a72d40d" exitCode=60
Oct 30 19:15:06.786123 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:06.785604    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f3d6a5-29889795-sfmsm" event={"ID":"c67ac146-cadf-480c-b1a0-47e60b73d0e5","Type":"ContainerDied","Data":"2fc4e1ec07256c28e0f65ee312a3065750e3717bbaad7d98d9667e316a72d40d"}
Oct 30 19:15:06.786123 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:06.785674    4525 scope.go:122] "RemoveContainer" containerID="f22d3e9a9c67630caa899259823ef232643df8c8b7b18600b40305511171f230"
Oct 30 19:15:06.786123 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:06.786083    4525 scope.go:122] "RemoveContainer" containerID="2fc4e1ec07256c28e0f65ee312a3065750e3717bbaad7d98d9667e316a72d40d"
Oct 30 19:15:06.786531 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:15:06.786508    4525 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-f3d6a5-29889795-sfmsm_odh-ai-gateway-infra(c67ac146-cadf-480c-b1a0-47e60b73d0e5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f3d6a5-29889795-sfmsm" podUID="c67ac146-cadf-480c-b1a0-47e60b73d0e5"
Oct 30 19:15:06.788255 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:06.788224    4525 generic.go:356] "Generic (PLEG): container finished" podID="e8fa12c9-9b1d-4033-aebb-eabcd9340222" containerID="2eb84195c4aa34b40939ff2bfec53476ecda6819602f7c8b20fba705102b3892" exitCode=60
Oct 30 19:15:06.788390 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:06.788302    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ef6be7-29889795-pf2p2" event={"ID":"e8fa12c9-9b1d-4033-aebb-eabcd9340222","Type":"ContainerDied","Data":"2eb84195c4aa34b40939ff2bfec53476ecda6819602f7c8b20fba705102b3892"}
Oct 30 19:15:06.788620 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:06.788602    4525 scope.go:122] "RemoveContainer" containerID="2eb84195c4aa34b40939ff2bfec53476ecda6819602f7c8b20fba705102b3892"
Oct 30 19:15:06.789025 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:15:06.788963    4525 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-ef6be7-29889795-pf2p2_odh-ai-gateway-infra(e8fa12c9-9b1d-4033-aebb-eabcd9340222)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ef6be7-29889795-pf2p2" podUID="e8fa12c9-9b1d-4033-aebb-eabcd9340222"
Oct 30 19:15:06.790587 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:06.790568    4525 generic.go:356] "Generic (PLEG): container finished" podID="e506709f-838a-4255-b3fb-498da9fb0697" containerID="a761973cdcd815f36320a0d6d12e727b7c0c9fb85c8400f753926bb337f7e4e7" exitCode=60
Oct 30 19:15:06.790741 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:06.790643    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9de872-29889795-c7whh" event={"ID":"e506709f-838a-4255-b3fb-498da9fb0697","Type":"ContainerDied","Data":"a761973cdcd815f36320a0d6d12e727b7c0c9fb85c8400f753926bb337f7e4e7"}
Oct 30 19:15:06.790895 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:06.790879    4525 scope.go:122] "RemoveContainer" containerID="a761973cdcd815f36320a0d6d12e727b7c0c9fb85c8400f753926bb337f7e4e7"
Oct 30 19:15:06.791205 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:15:06.791179    4525 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-9de872-29889795-c7whh_odh-ai-gateway-infra(e506709f-838a-4255-b3fb-498da9fb0697)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9de872-29889795-c7whh" podUID="e506709f-838a-4255-b3fb-498da9fb0697"
Oct 30 19:15:06.792731 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:06.792710    4525 generic.go:356] "Generic (PLEG): container finished" podID="01d8e335-0163-44d7-b811-fb9b99cd285a" containerID="b9942eba4f8e91c993f48a29a5a8e2b3f364e68c1c7994b26e54fa1e71319d99" exitCode=60
Oct 30 19:15:06.792825 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:06.792770    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39bde3-29889795-rzh6s" event={"ID":"01d8e335-0163-44d7-b811-fb9b99cd285a","Type":"ContainerDied","Data":"b9942eba4f8e91c993f48a29a5a8e2b3f364e68c1c7994b26e54fa1e71319d99"}
Oct 30 19:15:06.793060 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:06.793041    4525 scope.go:122] "RemoveContainer" containerID="b9942eba4f8e91c993f48a29a5a8e2b3f364e68c1c7994b26e54fa1e71319d99"
Oct 30 19:15:06.793346 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:15:06.793320    4525 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-39bde3-29889795-rzh6s_odh-ai-gateway-infra(01d8e335-0163-44d7-b811-fb9b99cd285a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39bde3-29889795-rzh6s" podUID="01d8e335-0163-44d7-b811-fb9b99cd285a"
Oct 30 19:15:06.794761 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:06.794743    4525 generic.go:356] "Generic (PLEG): container finished" podID="3253644f-14ff-452e-851b-694d8724574f" containerID="b3372beff49735246e3b19c65a4831af6d48c86f5186b928df1fdaa9e598da35" exitCode=60
Oct 30 19:15:06.794859 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:06.794831    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-85cb41-29889795-9wpqw" event={"ID":"3253644f-14ff-452e-851b-694d8724574f","Type":"ContainerDied","Data":"b3372beff49735246e3b19c65a4831af6d48c86f5186b928df1fdaa9e598da35"}
Oct 30 19:15:06.795039 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:06.795025    4525 scope.go:122] "RemoveContainer" containerID="b3372beff49735246e3b19c65a4831af6d48c86f5186b928df1fdaa9e598da35"
Oct 30 19:15:06.795292 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:15:06.795273    4525 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-85cb41-29889795-9wpqw_odh-ai-gateway-infra(3253644f-14ff-452e-851b-694d8724574f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-85cb41-29889795-9wpqw" podUID="3253644f-14ff-452e-851b-694d8724574f"
Oct 30 19:15:06.796863 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:06.796843    4525 generic.go:356] "Generic (PLEG): container finished" podID="403406c5-302f-4563-be3b-63ac8959d517" containerID="c8c24555076ec9b77fba843829b23ccbfde748b78f94fd62b04286465bb42423" exitCode=60
Oct 30 19:15:06.796957 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:06.796924    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2d1c71-29889795-p7txw" event={"ID":"403406c5-302f-4563-be3b-63ac8959d517","Type":"ContainerDied","Data":"c8c24555076ec9b77fba843829b23ccbfde748b78f94fd62b04286465bb42423"}
Oct 30 19:15:06.797122 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:06.797106    4525 scope.go:122] "RemoveContainer" containerID="c8c24555076ec9b77fba843829b23ccbfde748b78f94fd62b04286465bb42423"
Oct 30 19:15:06.797360 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:15:06.797337    4525 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-2d1c71-29889795-p7txw_odh-ai-gateway-infra(403406c5-302f-4563-be3b-63ac8959d517)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2d1c71-29889795-p7txw" podUID="403406c5-302f-4563-be3b-63ac8959d517"
Oct 30 19:15:06.798904 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:06.798885    4525 generic.go:356] "Generic (PLEG): container finished" podID="8d3ff890-4ac4-4395-b309-3483be728b66" containerID="eca90751c5fcfd6ad9dd0ad5ef8f4ea898e4a9f3fbfc5cfa8218dc1a388ca81f" exitCode=60
Oct 30 19:15:06.799011 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:06.798959    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-ef6b56-29889795-nl7wc" event={"ID":"8d3ff890-4ac4-4395-b309-3483be728b66","Type":"ContainerDied","Data":"eca90751c5fcfd6ad9dd0ad5ef8f4ea898e4a9f3fbfc5cfa8218dc1a388ca81f"}
Oct 30 19:15:06.799315 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:06.799295    4525 scope.go:122] "RemoveContainer" containerID="eca90751c5fcfd6ad9dd0ad5ef8f4ea898e4a9f3fbfc5cfa8218dc1a388ca81f"
Oct 30 19:15:06.799573 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:15:06.799539    4525 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-ef6b56-29889795-nl7wc_odh-ai-gateway-infra(8d3ff890-4ac4-4395-b309-3483be728b66)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-ef6b56-29889795-nl7wc" podUID="8d3ff890-4ac4-4395-b309-3483be728b66"
Oct 30 19:15:06.801115 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:06.801094    4525 generic.go:356] "Generic (PLEG): container finished" podID="bfeff9d8-6996-4f87-a933-b41f97627004" containerID="7e5f73f700cc6536dc8dd37121347a657f6b637ae42edbb6d8379380a72b0807" exitCode=60
Oct 30 19:15:06.801889 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:06.801175    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-68b466-29889795-fgfjm" event={"ID":"bfeff9d8-6996-4f87-a933-b41f97627004","Type":"ContainerDied","Data":"7e5f73f700cc6536dc8dd37121347a657f6b637ae42edbb6d8379380a72b0807"}
Oct 30 19:15:06.801889 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:06.801362    4525 scope.go:122] "RemoveContainer" containerID="7e5f73f700cc6536dc8dd37121347a657f6b637ae42edbb6d8379380a72b0807"
Oct 30 19:15:06.801889 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:15:06.801573    4525 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-68b466-29889795-fgfjm_odh-ai-gateway-infra(bfeff9d8-6996-4f87-a933-b41f97627004)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-68b466-29889795-fgfjm" podUID="bfeff9d8-6996-4f87-a933-b41f97627004"
Oct 30 19:15:06.803510 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:06.803493    4525 generic.go:356] "Generic (PLEG): container finished" podID="d3aa2e02-e67a-4418-8fb7-3d805cb7ca7c" containerID="4b6ea5a7fbea7ec6dce81f393e119cc99c56ca9793b41137e4cba649ecf4f9d4" exitCode=60
Oct 30 19:15:06.803590 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:06.803579    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ed012e-29889795-f5xmz" event={"ID":"d3aa2e02-e67a-4418-8fb7-3d805cb7ca7c","Type":"ContainerDied","Data":"4b6ea5a7fbea7ec6dce81f393e119cc99c56ca9793b41137e4cba649ecf4f9d4"}
Oct 30 19:15:06.803898 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:06.803883    4525 scope.go:122] "RemoveContainer" containerID="4b6ea5a7fbea7ec6dce81f393e119cc99c56ca9793b41137e4cba649ecf4f9d4"
Oct 30 19:15:06.804103 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:15:06.804086    4525 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-ed012e-29889795-f5xmz_odh-ai-gateway-infra(d3aa2e02-e67a-4418-8fb7-3d805cb7ca7c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ed012e-29889795-f5xmz" podUID="d3aa2e02-e67a-4418-8fb7-3d805cb7ca7c"
Oct 30 19:15:06.807258 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:06.807228    4525 generic.go:356] "Generic (PLEG): container finished" podID="e5149d31-2979-4353-9b41-df84f90cffbd" containerID="e4f00fabeb42c763cd05b704e39c68c4d896c14f5b0e0dfd0a847ccea5610a3b" exitCode=60
Oct 30 19:15:06.807347 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:06.807300    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-46385c-29889795-hbvv9" event={"ID":"e5149d31-2979-4353-9b41-df84f90cffbd","Type":"ContainerDied","Data":"e4f00fabeb42c763cd05b704e39c68c4d896c14f5b0e0dfd0a847ccea5610a3b"}
Oct 30 19:15:06.807568 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:06.807553    4525 scope.go:122] "RemoveContainer" containerID="e4f00fabeb42c763cd05b704e39c68c4d896c14f5b0e0dfd0a847ccea5610a3b"
Oct 30 19:15:06.808056 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:15:06.807839    4525 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-46385c-29889795-hbvv9_odh-ai-gateway-infra(e5149d31-2979-4353-9b41-df84f90cffbd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-46385c-29889795-hbvv9" podUID="e5149d31-2979-4353-9b41-df84f90cffbd"
Oct 30 19:15:06.809567 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:06.809548    4525 generic.go:356] "Generic (PLEG): container finished" podID="75f454fd-6c51-42f1-a19d-24d1c739a88c" containerID="d5f6d81b7c81c3ac2ba9689576c07628a08dc6d4793847073b2965051fd5c3c7" exitCode=60
Oct 30 19:15:06.809663 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:06.809605    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6f956-29889795-vgkfh" event={"ID":"75f454fd-6c51-42f1-a19d-24d1c739a88c","Type":"ContainerDied","Data":"d5f6d81b7c81c3ac2ba9689576c07628a08dc6d4793847073b2965051fd5c3c7"}
Oct 30 19:15:06.809904 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:06.809885    4525 scope.go:122] "RemoveContainer" containerID="d5f6d81b7c81c3ac2ba9689576c07628a08dc6d4793847073b2965051fd5c3c7"
Oct 30 19:15:06.810174 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:15:06.810154    4525 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-d6f956-29889795-vgkfh_odh-ai-gateway-infra(75f454fd-6c51-42f1-a19d-24d1c739a88c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6f956-29889795-vgkfh" podUID="75f454fd-6c51-42f1-a19d-24d1c739a88c"
Oct 30 19:15:06.811806 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:06.811785    4525 generic.go:356] "Generic (PLEG): container finished" podID="dfff552f-d7bb-414b-9a43-34922f58d2bd" containerID="feaf6beebdb8e6065553808706bb04d1c5767a184d2a939797301ad80eaaccc0" exitCode=60
Oct 30 19:15:06.811911 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:06.811874    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dcd326-29889795-zj7rm" event={"ID":"dfff552f-d7bb-414b-9a43-34922f58d2bd","Type":"ContainerDied","Data":"feaf6beebdb8e6065553808706bb04d1c5767a184d2a939797301ad80eaaccc0"}
Oct 30 19:15:06.812120 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:06.812104    4525 scope.go:122] "RemoveContainer" containerID="feaf6beebdb8e6065553808706bb04d1c5767a184d2a939797301ad80eaaccc0"
Oct 30 19:15:06.812439 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:15:06.812410    4525 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-dcd326-29889795-zj7rm_odh-ai-gateway-infra(dfff552f-d7bb-414b-9a43-34922f58d2bd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dcd326-29889795-zj7rm" podUID="dfff552f-d7bb-414b-9a43-34922f58d2bd"
Oct 30 19:15:06.815424 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:06.815405    4525 scope.go:122] "RemoveContainer" containerID="a85cad846748772270d87e7d52bb0fdb495ee2dbbfcf4d6690904fc421663855"
Oct 30 19:15:06.820518 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:06.820495    4525 generic.go:356] "Generic (PLEG): container finished" podID="4849dbc2-0043-46f1-ab5e-a76c16f85db9" containerID="00cf906bb8c75aa2c30b8558a7196bfbcf98d1a86bedc810da20a0a10de8c885" exitCode=60
Oct 30 19:15:06.820652 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:06.820569    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c2e3f9-29889795-5rf8p" event={"ID":"4849dbc2-0043-46f1-ab5e-a76c16f85db9","Type":"ContainerDied","Data":"00cf906bb8c75aa2c30b8558a7196bfbcf98d1a86bedc810da20a0a10de8c885"}
Oct 30 19:15:06.821037 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:06.821014    4525 scope.go:122] "RemoveContainer" containerID="00cf906bb8c75aa2c30b8558a7196bfbcf98d1a86bedc810da20a0a10de8c885"
Oct 30 19:15:06.821379 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:15:06.821315    4525 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-c2e3f9-29889795-5rf8p_odh-ai-gateway-infra(4849dbc2-0043-46f1-ab5e-a76c16f85db9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c2e3f9-29889795-5rf8p" podUID="4849dbc2-0043-46f1-ab5e-a76c16f85db9"
Oct 30 19:15:06.822766 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:06.822744    4525 generic.go:356] "Generic (PLEG): container finished" podID="90e98322-b727-4a35-9d93-808e0b01d483" containerID="d5f34ddd28f492babd029a602d15f223482a69a95b92f58455cfd81508faad20" exitCode=60
Oct 30 19:15:06.822874 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:06.822820    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a1d55d-29889795-bw7v2" event={"ID":"90e98322-b727-4a35-9d93-808e0b01d483","Type":"ContainerDied","Data":"d5f34ddd28f492babd029a602d15f223482a69a95b92f58455cfd81508faad20"}
Oct 30 19:15:06.823059 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:06.823046    4525 scope.go:122] "RemoveContainer" containerID="d5f34ddd28f492babd029a602d15f223482a69a95b92f58455cfd81508faad20"
Oct 30 19:15:06.823320 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:15:06.823304    4525 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-a1d55d-29889795-bw7v2_odh-ai-gateway-infra(90e98322-b727-4a35-9d93-808e0b01d483)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a1d55d-29889795-bw7v2" podUID="90e98322-b727-4a35-9d93-808e0b01d483"
Oct 30 19:15:06.825370 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:06.825350    4525 generic.go:356] "Generic (PLEG): container finished" podID="7590ba0b-baad-4a0a-ad6c-3309f2123c5c" containerID="93f3d16ab8437bb3f0e6e01f8d0e559d1ba8c715397f0588f5f88b7f45902ea6" exitCode=60
Oct 30 19:15:06.825493 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:06.825418    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-baa504-29889795-bwhw4" event={"ID":"7590ba0b-baad-4a0a-ad6c-3309f2123c5c","Type":"ContainerDied","Data":"93f3d16ab8437bb3f0e6e01f8d0e559d1ba8c715397f0588f5f88b7f45902ea6"}
Oct 30 19:15:06.825657 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:06.825640    4525 scope.go:122] "RemoveContainer" containerID="93f3d16ab8437bb3f0e6e01f8d0e559d1ba8c715397f0588f5f88b7f45902ea6"
Oct 30 19:15:06.825954 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:15:06.825935    4525 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-baa504-29889795-bwhw4_odh-ai-gateway-infra(7590ba0b-baad-4a0a-ad6c-3309f2123c5c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-baa504-29889795-bwhw4" podUID="7590ba0b-baad-4a0a-ad6c-3309f2123c5c"
Oct 30 19:15:06.852537 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:06.852517    4525 scope.go:122] "RemoveContainer" containerID="5bf93b69f0676d4dd71e4ca164b1eb3eae7480c4d5dffeddf18cea1d61300f66"
Oct 30 19:15:06.869611 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:06.869581    4525 scope.go:122] "RemoveContainer" containerID="7887a3f1ecce11f0024ef00c7405673dab634853c040ffcfbe17d75231656a6d"
Oct 30 19:15:06.885160 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:06.885138    4525 scope.go:122] "RemoveContainer" containerID="fa7f2ad6ea407e35a0d08af96429f43d89ab95282ebcbebafb50604c27d205c8"
Oct 30 19:15:06.902328 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:06.902300    4525 scope.go:122] "RemoveContainer" containerID="38e15cc1581a4c3b8f2f15c91f3fcb46d88b394786f989582ba0f6f7a431066b"
Oct 30 19:15:06.918338 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:06.918314    4525 scope.go:122] "RemoveContainer" containerID="d9e9b3a6ee0b8a1ab56f32ba0cab339886bfae3870fe819ac9549bd3a1e6db78"
Oct 30 19:15:06.934128 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:06.934110    4525 scope.go:122] "RemoveContainer" containerID="a1f0c058fa8445127cc7916103a8a120b3ccd2f52db8877ca91d18b8cd833eb4"
Oct 30 19:15:06.950919 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:06.950891    4525 scope.go:122] "RemoveContainer" containerID="02a994d582066c0c17c2cbb9529a5e6c386dabdca020c91b57c27810dd1ac830"
Oct 30 19:15:06.967165 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:06.967134    4525 scope.go:122] "RemoveContainer" containerID="b970bdc5e20ebdca7b18f452684547878e11c97d622b44738a8e95e4d80b1f38"
Oct 30 19:15:06.983202 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:06.983175    4525 scope.go:122] "RemoveContainer" containerID="bde160edd21d0cf3d038ffab3f359e472df2848569c76cc3fd0034dc68a8ed7c"
Oct 30 19:15:06.998788 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:06.998769    4525 scope.go:122] "RemoveContainer" containerID="e2b412c6d4a660a15743a37c2102414e3f036268d19e6357ffa11f5fbf7ab9e6"
Oct 30 19:15:07.017304 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:07.017283    4525 scope.go:122] "RemoveContainer" containerID="e7927bc7bbef5308302236310a5484c9b5c78d02464b206b43cb2ebb24def1c8"
Oct 30 19:15:07.033951 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:07.033928    4525 scope.go:122] "RemoveContainer" containerID="c182f62c13c34edba23bbce05d93a58a4395141026e897ea79cae54490baac08"
Oct 30 19:15:07.130908 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:07.130793    4525 scope.go:122] "RemoveContainer" containerID="c0b8c761f56479ea3c417deea8a9e3df0827c40f5e53a1f9eca9716d4f9a6509"
Oct 30 19:15:07.836084 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:07.836050    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-lkl5j"]
Oct 30 19:15:07.836595 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:07.836297    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-lkl5j" podUID="865afd2d-c5e2-4ee5-bfdb-303102de6616" containerName="istio-proxy" containerID="cri-o://94fdd2bda395afacbf6ccac28c4a3256280c56c1e509c8f03220ecbe8e53a885" gracePeriod=30
Oct 30 19:15:07.853999 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:07.853968    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b"]
Oct 30 19:15:07.854238 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:07.854206    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b" podUID="a8e73ad5-17f6-4205-bc5f-8965afa49f57" containerName="istio-proxy" containerID="cri-o://23114b3fc0f0f25c8de13319f8e7cb8a2484deaf68608b286435ca9cb76b69ef" gracePeriod=30
Oct 30 19:15:07.856807 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:07.856784    4525 scope.go:122] "RemoveContainer" containerID="2eb84195c4aa34b40939ff2bfec53476ecda6819602f7c8b20fba705102b3892"
Oct 30 19:15:07.857247 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:15:07.857224    4525 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-ef6be7-29889795-pf2p2_odh-ai-gateway-infra(e8fa12c9-9b1d-4033-aebb-eabcd9340222)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ef6be7-29889795-pf2p2" podUID="e8fa12c9-9b1d-4033-aebb-eabcd9340222"
Oct 30 19:15:07.863151 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:07.863078    4525 scope.go:122] "RemoveContainer" containerID="a761973cdcd815f36320a0d6d12e727b7c0c9fb85c8400f753926bb337f7e4e7"
Oct 30 19:15:07.864071 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:07.864038    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-baa504-29889795-bwhw4"]
Oct 30 19:15:07.864416 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:15:07.864383    4525 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-9de872-29889795-c7whh_odh-ai-gateway-infra(e506709f-838a-4255-b3fb-498da9fb0697)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9de872-29889795-c7whh" podUID="e506709f-838a-4255-b3fb-498da9fb0697"
Oct 30 19:15:07.869238 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:07.869214    4525 scope.go:122] "RemoveContainer" containerID="b9942eba4f8e91c993f48a29a5a8e2b3f364e68c1c7994b26e54fa1e71319d99"
Oct 30 19:15:07.870394 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:15:07.869845    4525 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-39bde3-29889795-rzh6s_odh-ai-gateway-infra(01d8e335-0163-44d7-b811-fb9b99cd285a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39bde3-29889795-rzh6s" podUID="01d8e335-0163-44d7-b811-fb9b99cd285a"
Oct 30 19:15:07.874361 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:07.874339    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-tvbgw"]
Oct 30 19:15:07.874768 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:07.874737    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-tvbgw" podUID="e9973f1f-61b0-4d38-a7c2-b9c6fde9da94" containerName="istio-proxy" containerID="cri-o://b0d994c2d6d9338b52f452d8fc13f4c468c9af8dccd52cca1fda8baab62cb723" gracePeriod=30
Oct 30 19:15:07.876747 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:07.876724    4525 scope.go:122] "RemoveContainer" containerID="b3372beff49735246e3b19c65a4831af6d48c86f5186b928df1fdaa9e598da35"
Oct 30 19:15:07.886341 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:15:07.886312    4525 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-85cb41-29889795-9wpqw_odh-ai-gateway-infra(3253644f-14ff-452e-851b-694d8724574f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-85cb41-29889795-9wpqw" podUID="3253644f-14ff-452e-851b-694d8724574f"
Oct 30 19:15:07.889133 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:07.889113    4525 scope.go:122] "RemoveContainer" containerID="c8c24555076ec9b77fba843829b23ccbfde748b78f94fd62b04286465bb42423"
Oct 30 19:15:07.889575 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:15:07.889556    4525 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-2d1c71-29889795-p7txw_odh-ai-gateway-infra(403406c5-302f-4563-be3b-63ac8959d517)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2d1c71-29889795-p7txw" podUID="403406c5-302f-4563-be3b-63ac8959d517"
Oct 30 19:15:07.894052 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:07.894032    4525 scope.go:122] "RemoveContainer" containerID="7e5f73f700cc6536dc8dd37121347a657f6b637ae42edbb6d8379380a72b0807"
Oct 30 19:15:07.894348 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:15:07.894327    4525 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-68b466-29889795-fgfjm_odh-ai-gateway-infra(bfeff9d8-6996-4f87-a933-b41f97627004)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-68b466-29889795-fgfjm" podUID="bfeff9d8-6996-4f87-a933-b41f97627004"
Oct 30 19:15:07.894446 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:07.894421    4525 scope.go:122] "RemoveContainer" containerID="4b6ea5a7fbea7ec6dce81f393e119cc99c56ca9793b41137e4cba649ecf4f9d4"
Oct 30 19:15:07.894787 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:15:07.894766    4525 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-ed012e-29889795-f5xmz_odh-ai-gateway-infra(d3aa2e02-e67a-4418-8fb7-3d805cb7ca7c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ed012e-29889795-f5xmz" podUID="d3aa2e02-e67a-4418-8fb7-3d805cb7ca7c"
Oct 30 19:15:07.896115 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:07.896099    4525 scope.go:122] "RemoveContainer" containerID="e4f00fabeb42c763cd05b704e39c68c4d896c14f5b0e0dfd0a847ccea5610a3b"
Oct 30 19:15:07.896398 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:15:07.896368    4525 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-46385c-29889795-hbvv9_odh-ai-gateway-infra(e5149d31-2979-4353-9b41-df84f90cffbd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-46385c-29889795-hbvv9" podUID="e5149d31-2979-4353-9b41-df84f90cffbd"
Oct 30 19:15:07.898326 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:07.898287    4525 scope.go:122] "RemoveContainer" containerID="d5f6d81b7c81c3ac2ba9689576c07628a08dc6d4793847073b2965051fd5c3c7"
Oct 30 19:15:07.898667 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:15:07.898642    4525 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-d6f956-29889795-vgkfh_odh-ai-gateway-infra(75f454fd-6c51-42f1-a19d-24d1c739a88c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6f956-29889795-vgkfh" podUID="75f454fd-6c51-42f1-a19d-24d1c739a88c"
Oct 30 19:15:07.900297 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:07.900275    4525 scope.go:122] "RemoveContainer" containerID="feaf6beebdb8e6065553808706bb04d1c5767a184d2a939797301ad80eaaccc0"
Oct 30 19:15:07.900693 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:15:07.900567    4525 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-dcd326-29889795-zj7rm_odh-ai-gateway-infra(dfff552f-d7bb-414b-9a43-34922f58d2bd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dcd326-29889795-zj7rm" podUID="dfff552f-d7bb-414b-9a43-34922f58d2bd"
Oct 30 19:15:07.904507 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:07.904486    4525 scope.go:122] "RemoveContainer" containerID="d5f34ddd28f492babd029a602d15f223482a69a95b92f58455cfd81508faad20"
Oct 30 19:15:07.904824 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:15:07.904801    4525 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-a1d55d-29889795-bw7v2_odh-ai-gateway-infra(90e98322-b727-4a35-9d93-808e0b01d483)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a1d55d-29889795-bw7v2" podUID="90e98322-b727-4a35-9d93-808e0b01d483"
Oct 30 19:15:07.906066 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:07.906044    4525 scope.go:122] "RemoveContainer" containerID="93f3d16ab8437bb3f0e6e01f8d0e559d1ba8c715397f0588f5f88b7f45902ea6"
Oct 30 19:15:07.906575 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:15:07.906515    4525 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-baa504-29889795-bwhw4_odh-ai-gateway-infra(7590ba0b-baad-4a0a-ad6c-3309f2123c5c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-baa504-29889795-bwhw4" podUID="7590ba0b-baad-4a0a-ad6c-3309f2123c5c"
Oct 30 19:15:07.911654 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:07.911638    4525 scope.go:122] "RemoveContainer" containerID="2fc4e1ec07256c28e0f65ee312a3065750e3717bbaad7d98d9667e316a72d40d"
Oct 30 19:15:07.911906 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:15:07.911889    4525 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-f3d6a5-29889795-sfmsm_odh-ai-gateway-infra(c67ac146-cadf-480c-b1a0-47e60b73d0e5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f3d6a5-29889795-sfmsm" podUID="c67ac146-cadf-480c-b1a0-47e60b73d0e5"
Oct 30 19:15:08.482003 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:08.481962    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-baa504-hgbht"]
Oct 30 19:15:08.494528 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:08.494491    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-baa504-hgbht"]
Oct 30 19:15:09.114243 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:09.114218    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-baa504-29889795-bwhw4"
Oct 30 19:15:09.226777 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:09.226668    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7590ba0b-baad-4a0a-ad6c-3309f2123c5c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7590ba0b-baad-4a0a-ad6c-3309f2123c5c-maas-api-service-ca\") pod \"7590ba0b-baad-4a0a-ad6c-3309f2123c5c\" (UID: \"7590ba0b-baad-4a0a-ad6c-3309f2123c5c\") "
Oct 30 19:15:09.227012 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:09.226987    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7590ba0b-baad-4a0a-ad6c-3309f2123c5c-maas-api-service-ca" pod "7590ba0b-baad-4a0a-ad6c-3309f2123c5c" (UID: "7590ba0b-baad-4a0a-ad6c-3309f2123c5c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:09.227087 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:09.227037    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7590ba0b-baad-4a0a-ad6c-3309f2123c5c-kube-api-access-qjf8x\" (UniqueName: \"kubernetes.io/projected/7590ba0b-baad-4a0a-ad6c-3309f2123c5c-kube-api-access-qjf8x\") pod \"7590ba0b-baad-4a0a-ad6c-3309f2123c5c\" (UID: \"7590ba0b-baad-4a0a-ad6c-3309f2123c5c\") "
Oct 30 19:15:09.227774 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:09.227752    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7590ba0b-baad-4a0a-ad6c-3309f2123c5c-maas-api-service-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:09.229246 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:09.229225    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7590ba0b-baad-4a0a-ad6c-3309f2123c5c-kube-api-access-qjf8x" pod "7590ba0b-baad-4a0a-ad6c-3309f2123c5c" (UID: "7590ba0b-baad-4a0a-ad6c-3309f2123c5c"). InnerVolumeSpecName "kube-api-access-qjf8x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:09.329308 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:09.329267    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjf8x\" (UniqueName: \"kubernetes.io/projected/7590ba0b-baad-4a0a-ad6c-3309f2123c5c-kube-api-access-qjf8x\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:09.550816 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:09.550735    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35138e86-4197-404b-83a3-374e5202931f" path="/var/lib/kubelet/pods/35138e86-4197-404b-83a3-374e5202931f/volumes"
Oct 30 19:15:09.932732 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:09.932667    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-baa504-29889795-bwhw4"
Oct 30 19:15:09.932910 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:09.932709    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-baa504-29889795-bwhw4" event={"ID":"7590ba0b-baad-4a0a-ad6c-3309f2123c5c","Type":"ContainerDied","Data":"b7ff9f2956bf15923b321b9dfc1be26068aea48eaa160eec547aa148ea929def"}
Oct 30 19:15:09.932910 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:09.932841    4525 scope.go:122] "RemoveContainer" containerID="93f3d16ab8437bb3f0e6e01f8d0e559d1ba8c715397f0588f5f88b7f45902ea6"
Oct 30 19:15:09.963457 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:09.963415    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-baa504-29889795-bwhw4"]
Oct 30 19:15:09.970976 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:09.970943    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-baa504-29889795-bwhw4"]
Oct 30 19:15:11.096925 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:11.096814    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7"]
Oct 30 19:15:11.097448 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:11.097156    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7" podUID="77adecb1-af53-4dd9-bb52-d8e22ebb03e4" containerName="istio-proxy" containerID="cri-o://4249fc47d1098fc9d135027761840864157ed12000e934280f9bcd9b3ed2d5be" gracePeriod=30
Oct 30 19:15:11.103549 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:11.103512    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-99fmr"]
Oct 30 19:15:11.104617 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:11.103792    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-99fmr" podUID="be563a8d-e16a-4b6a-b82d-4d6c4847a527" containerName="istio-proxy" containerID="cri-o://a50a4388e06b290cfb03ef75805f28936ed1f88439497a667234524251e1ac2e" gracePeriod=30
Oct 30 19:15:11.109353 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:11.109198    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-mfdt7"]
Oct 30 19:15:11.109465 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:11.109367    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-mfdt7" podUID="be16530b-c26d-4da7-b25e-f5246eff6c3d" containerName="istio-proxy" containerID="cri-o://09f5a51a5134a8acef4c236f5cd80462bf04576891b2f0856f510489f51e1a9e" gracePeriod=30
Oct 30 19:15:11.548236 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:11.548199    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7590ba0b-baad-4a0a-ad6c-3309f2123c5c" path="/var/lib/kubelet/pods/7590ba0b-baad-4a0a-ad6c-3309f2123c5c/volumes"
Oct 30 19:15:12.964454 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:12.964425    4525 generic.go:356] "Generic (PLEG): container finished" podID="865afd2d-c5e2-4ee5-bfdb-303102de6616" containerID="94fdd2bda395afacbf6ccac28c4a3256280c56c1e509c8f03220ecbe8e53a885" exitCode=0
Oct 30 19:15:12.964851 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:12.964499    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-lkl5j" event={"ID":"865afd2d-c5e2-4ee5-bfdb-303102de6616","Type":"ContainerDied","Data":"94fdd2bda395afacbf6ccac28c4a3256280c56c1e509c8f03220ecbe8e53a885"}
Oct 30 19:15:13.199824 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.199792    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b"
Oct 30 19:15:13.240747 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.236729    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-lkl5j"
Oct 30 19:15:13.284794 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.284324    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-tvbgw"
Oct 30 19:15:13.326264 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.326234    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8e73ad5-17f6-4205-bc5f-8965afa49f57-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8e73ad5-17f6-4205-bc5f-8965afa49f57-workload-socket\") pod \"a8e73ad5-17f6-4205-bc5f-8965afa49f57\" (UID: \"a8e73ad5-17f6-4205-bc5f-8965afa49f57\") "
Oct 30 19:15:13.326446 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.326270    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8e73ad5-17f6-4205-bc5f-8965afa49f57-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8e73ad5-17f6-4205-bc5f-8965afa49f57-credential-socket\") pod \"a8e73ad5-17f6-4205-bc5f-8965afa49f57\" (UID: \"a8e73ad5-17f6-4205-bc5f-8965afa49f57\") "
Oct 30 19:15:13.326446 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.326307    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8e73ad5-17f6-4205-bc5f-8965afa49f57-istio-token\" (UniqueName: \"kubernetes.io/projected/a8e73ad5-17f6-4205-bc5f-8965afa49f57-istio-token\") pod \"a8e73ad5-17f6-4205-bc5f-8965afa49f57\" (UID: \"a8e73ad5-17f6-4205-bc5f-8965afa49f57\") "
Oct 30 19:15:13.326446 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.326354    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/865afd2d-c5e2-4ee5-bfdb-303102de6616-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/865afd2d-c5e2-4ee5-bfdb-303102de6616-workload-certs\") pod \"865afd2d-c5e2-4ee5-bfdb-303102de6616\" (UID: \"865afd2d-c5e2-4ee5-bfdb-303102de6616\") "
Oct 30 19:15:13.326446 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.326412    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8e73ad5-17f6-4205-bc5f-8965afa49f57-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8e73ad5-17f6-4205-bc5f-8965afa49f57-istio-envoy\") pod \"a8e73ad5-17f6-4205-bc5f-8965afa49f57\" (UID: \"a8e73ad5-17f6-4205-bc5f-8965afa49f57\") "
Oct 30 19:15:13.326666 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.326458    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8e73ad5-17f6-4205-bc5f-8965afa49f57-kube-api-access-hdlfj\" (UniqueName: \"kubernetes.io/projected/a8e73ad5-17f6-4205-bc5f-8965afa49f57-kube-api-access-hdlfj\") pod \"a8e73ad5-17f6-4205-bc5f-8965afa49f57\" (UID: \"a8e73ad5-17f6-4205-bc5f-8965afa49f57\") "
Oct 30 19:15:13.326666 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.326484    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/865afd2d-c5e2-4ee5-bfdb-303102de6616-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/865afd2d-c5e2-4ee5-bfdb-303102de6616-istiod-ca-cert\") pod \"865afd2d-c5e2-4ee5-bfdb-303102de6616\" (UID: \"865afd2d-c5e2-4ee5-bfdb-303102de6616\") "
Oct 30 19:15:13.326666 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.326558    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8e73ad5-17f6-4205-bc5f-8965afa49f57-workload-socket" pod "a8e73ad5-17f6-4205-bc5f-8965afa49f57" (UID: "a8e73ad5-17f6-4205-bc5f-8965afa49f57"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:13.326666 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.326598    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8e73ad5-17f6-4205-bc5f-8965afa49f57-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8e73ad5-17f6-4205-bc5f-8965afa49f57-workload-certs\") pod \"a8e73ad5-17f6-4205-bc5f-8965afa49f57\" (UID: \"a8e73ad5-17f6-4205-bc5f-8965afa49f57\") "
Oct 30 19:15:13.326666 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.326618    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8e73ad5-17f6-4205-bc5f-8965afa49f57-credential-socket" pod "a8e73ad5-17f6-4205-bc5f-8965afa49f57" (UID: "a8e73ad5-17f6-4205-bc5f-8965afa49f57"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:13.326969 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.326740    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/865afd2d-c5e2-4ee5-bfdb-303102de6616-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/865afd2d-c5e2-4ee5-bfdb-303102de6616-istio-data\") pod \"865afd2d-c5e2-4ee5-bfdb-303102de6616\" (UID: \"865afd2d-c5e2-4ee5-bfdb-303102de6616\") "
Oct 30 19:15:13.326969 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.326900    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8e73ad5-17f6-4205-bc5f-8965afa49f57-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8e73ad5-17f6-4205-bc5f-8965afa49f57-istiod-ca-cert\") pod \"a8e73ad5-17f6-4205-bc5f-8965afa49f57\" (UID: \"a8e73ad5-17f6-4205-bc5f-8965afa49f57\") "
Oct 30 19:15:13.326969 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.326954    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/865afd2d-c5e2-4ee5-bfdb-303102de6616-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/865afd2d-c5e2-4ee5-bfdb-303102de6616-istio-envoy\") pod \"865afd2d-c5e2-4ee5-bfdb-303102de6616\" (UID: \"865afd2d-c5e2-4ee5-bfdb-303102de6616\") "
Oct 30 19:15:13.327136 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.326990    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8e73ad5-17f6-4205-bc5f-8965afa49f57-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8e73ad5-17f6-4205-bc5f-8965afa49f57-istio-data\") pod \"a8e73ad5-17f6-4205-bc5f-8965afa49f57\" (UID: \"a8e73ad5-17f6-4205-bc5f-8965afa49f57\") "
Oct 30 19:15:13.327136 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.326993    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/865afd2d-c5e2-4ee5-bfdb-303102de6616-istiod-ca-cert" pod "865afd2d-c5e2-4ee5-bfdb-303102de6616" (UID: "865afd2d-c5e2-4ee5-bfdb-303102de6616"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:13.327136 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.327037    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/865afd2d-c5e2-4ee5-bfdb-303102de6616-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/865afd2d-c5e2-4ee5-bfdb-303102de6616-istio-podinfo\") pod \"865afd2d-c5e2-4ee5-bfdb-303102de6616\" (UID: \"865afd2d-c5e2-4ee5-bfdb-303102de6616\") "
Oct 30 19:15:13.327136 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.327041    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8e73ad5-17f6-4205-bc5f-8965afa49f57-workload-certs" pod "a8e73ad5-17f6-4205-bc5f-8965afa49f57" (UID: "a8e73ad5-17f6-4205-bc5f-8965afa49f57"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:13.327136 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.327101    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/865afd2d-c5e2-4ee5-bfdb-303102de6616-istio-token\" (UniqueName: \"kubernetes.io/projected/865afd2d-c5e2-4ee5-bfdb-303102de6616-istio-token\") pod \"865afd2d-c5e2-4ee5-bfdb-303102de6616\" (UID: \"865afd2d-c5e2-4ee5-bfdb-303102de6616\") "
Oct 30 19:15:13.327136 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.327114    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/865afd2d-c5e2-4ee5-bfdb-303102de6616-workload-certs" pod "865afd2d-c5e2-4ee5-bfdb-303102de6616" (UID: "865afd2d-c5e2-4ee5-bfdb-303102de6616"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:13.327526 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.327133    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/865afd2d-c5e2-4ee5-bfdb-303102de6616-istio-data" pod "865afd2d-c5e2-4ee5-bfdb-303102de6616" (UID: "865afd2d-c5e2-4ee5-bfdb-303102de6616"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:13.327526 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.327168    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a8e73ad5-17f6-4205-bc5f-8965afa49f57-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8e73ad5-17f6-4205-bc5f-8965afa49f57-istio-podinfo\") pod \"a8e73ad5-17f6-4205-bc5f-8965afa49f57\" (UID: \"a8e73ad5-17f6-4205-bc5f-8965afa49f57\") "
Oct 30 19:15:13.327526 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.327276    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/865afd2d-c5e2-4ee5-bfdb-303102de6616-kube-api-access-dlkhf\" (UniqueName: \"kubernetes.io/projected/865afd2d-c5e2-4ee5-bfdb-303102de6616-kube-api-access-dlkhf\") pod \"865afd2d-c5e2-4ee5-bfdb-303102de6616\" (UID: \"865afd2d-c5e2-4ee5-bfdb-303102de6616\") "
Oct 30 19:15:13.327526 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.327360    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8e73ad5-17f6-4205-bc5f-8965afa49f57-istiod-ca-cert" pod "a8e73ad5-17f6-4205-bc5f-8965afa49f57" (UID: "a8e73ad5-17f6-4205-bc5f-8965afa49f57"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:13.327526 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.327411    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/865afd2d-c5e2-4ee5-bfdb-303102de6616-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/865afd2d-c5e2-4ee5-bfdb-303102de6616-credential-socket\") pod \"865afd2d-c5e2-4ee5-bfdb-303102de6616\" (UID: \"865afd2d-c5e2-4ee5-bfdb-303102de6616\") "
Oct 30 19:15:13.327526 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.327481    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/865afd2d-c5e2-4ee5-bfdb-303102de6616-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/865afd2d-c5e2-4ee5-bfdb-303102de6616-workload-socket\") pod \"865afd2d-c5e2-4ee5-bfdb-303102de6616\" (UID: \"865afd2d-c5e2-4ee5-bfdb-303102de6616\") "
Oct 30 19:15:13.328052 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.327660    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8e73ad5-17f6-4205-bc5f-8965afa49f57-istio-data" pod "a8e73ad5-17f6-4205-bc5f-8965afa49f57" (UID: "a8e73ad5-17f6-4205-bc5f-8965afa49f57"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:13.332944 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.332753    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/865afd2d-c5e2-4ee5-bfdb-303102de6616-istio-podinfo" pod "865afd2d-c5e2-4ee5-bfdb-303102de6616" (UID: "865afd2d-c5e2-4ee5-bfdb-303102de6616"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:15:13.333091 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.332944    4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/865afd2d-c5e2-4ee5-bfdb-303102de6616-istio-data\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:13.333091 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.332967    4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8e73ad5-17f6-4205-bc5f-8965afa49f57-istiod-ca-cert\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:13.333091 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.332982    4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8e73ad5-17f6-4205-bc5f-8965afa49f57-istio-data\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:13.333091 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.333006    4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8e73ad5-17f6-4205-bc5f-8965afa49f57-workload-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:13.333091 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.333021    4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8e73ad5-17f6-4205-bc5f-8965afa49f57-credential-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:13.333091 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.333035    4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/865afd2d-c5e2-4ee5-bfdb-303102de6616-workload-certs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:13.333091 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.333048    4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/865afd2d-c5e2-4ee5-bfdb-303102de6616-istiod-ca-cert\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:13.333091 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.333067    4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8e73ad5-17f6-4205-bc5f-8965afa49f57-workload-certs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:13.333621 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.333298    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/865afd2d-c5e2-4ee5-bfdb-303102de6616-credential-socket" pod "865afd2d-c5e2-4ee5-bfdb-303102de6616" (UID: "865afd2d-c5e2-4ee5-bfdb-303102de6616"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:13.333621 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.333475    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/865afd2d-c5e2-4ee5-bfdb-303102de6616-workload-socket" pod "865afd2d-c5e2-4ee5-bfdb-303102de6616" (UID: "865afd2d-c5e2-4ee5-bfdb-303102de6616"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:13.333621 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.333555    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/865afd2d-c5e2-4ee5-bfdb-303102de6616-istio-envoy" pod "865afd2d-c5e2-4ee5-bfdb-303102de6616" (UID: "865afd2d-c5e2-4ee5-bfdb-303102de6616"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:13.333813 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.333711    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8e73ad5-17f6-4205-bc5f-8965afa49f57-istio-envoy" pod "a8e73ad5-17f6-4205-bc5f-8965afa49f57" (UID: "a8e73ad5-17f6-4205-bc5f-8965afa49f57"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:13.334634 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.333907    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8e73ad5-17f6-4205-bc5f-8965afa49f57-istio-token" pod "a8e73ad5-17f6-4205-bc5f-8965afa49f57" (UID: "a8e73ad5-17f6-4205-bc5f-8965afa49f57"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:13.334634 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.334019    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8e73ad5-17f6-4205-bc5f-8965afa49f57-kube-api-access-hdlfj" pod "a8e73ad5-17f6-4205-bc5f-8965afa49f57" (UID: "a8e73ad5-17f6-4205-bc5f-8965afa49f57"). InnerVolumeSpecName "kube-api-access-hdlfj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:13.334634 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.334059    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/865afd2d-c5e2-4ee5-bfdb-303102de6616-istio-token" pod "865afd2d-c5e2-4ee5-bfdb-303102de6616" (UID: "865afd2d-c5e2-4ee5-bfdb-303102de6616"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:13.334634 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.334094    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/865afd2d-c5e2-4ee5-bfdb-303102de6616-kube-api-access-dlkhf" pod "865afd2d-c5e2-4ee5-bfdb-303102de6616" (UID: "865afd2d-c5e2-4ee5-bfdb-303102de6616"). InnerVolumeSpecName "kube-api-access-dlkhf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:13.334634 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.334239    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a8e73ad5-17f6-4205-bc5f-8965afa49f57-istio-podinfo" pod "a8e73ad5-17f6-4205-bc5f-8965afa49f57" (UID: "a8e73ad5-17f6-4205-bc5f-8965afa49f57"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:15:13.433831 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.433748    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9973f1f-61b0-4d38-a7c2-b9c6fde9da94-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9973f1f-61b0-4d38-a7c2-b9c6fde9da94-istiod-ca-cert\") pod \"e9973f1f-61b0-4d38-a7c2-b9c6fde9da94\" (UID: \"e9973f1f-61b0-4d38-a7c2-b9c6fde9da94\") "
Oct 30 19:15:13.433831 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.433790    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9973f1f-61b0-4d38-a7c2-b9c6fde9da94-istio-token\" (UniqueName: \"kubernetes.io/projected/e9973f1f-61b0-4d38-a7c2-b9c6fde9da94-istio-token\") pod \"e9973f1f-61b0-4d38-a7c2-b9c6fde9da94\" (UID: \"e9973f1f-61b0-4d38-a7c2-b9c6fde9da94\") "
Oct 30 19:15:13.434030 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.433907    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9973f1f-61b0-4d38-a7c2-b9c6fde9da94-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9973f1f-61b0-4d38-a7c2-b9c6fde9da94-workload-socket\") pod \"e9973f1f-61b0-4d38-a7c2-b9c6fde9da94\" (UID: \"e9973f1f-61b0-4d38-a7c2-b9c6fde9da94\") "
Oct 30 19:15:13.434075 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.434022    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9973f1f-61b0-4d38-a7c2-b9c6fde9da94-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9973f1f-61b0-4d38-a7c2-b9c6fde9da94-credential-socket\") pod \"e9973f1f-61b0-4d38-a7c2-b9c6fde9da94\" (UID: \"e9973f1f-61b0-4d38-a7c2-b9c6fde9da94\") "
Oct 30 19:15:13.434134 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.434105    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9973f1f-61b0-4d38-a7c2-b9c6fde9da94-kube-api-access-6jnx7\" (UniqueName: \"kubernetes.io/projected/e9973f1f-61b0-4d38-a7c2-b9c6fde9da94-kube-api-access-6jnx7\") pod \"e9973f1f-61b0-4d38-a7c2-b9c6fde9da94\" (UID: \"e9973f1f-61b0-4d38-a7c2-b9c6fde9da94\") "
Oct 30 19:15:13.434196 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.434143    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e9973f1f-61b0-4d38-a7c2-b9c6fde9da94-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9973f1f-61b0-4d38-a7c2-b9c6fde9da94-istio-podinfo\") pod \"e9973f1f-61b0-4d38-a7c2-b9c6fde9da94\" (UID: \"e9973f1f-61b0-4d38-a7c2-b9c6fde9da94\") "
Oct 30 19:15:13.434196 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.434173    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9973f1f-61b0-4d38-a7c2-b9c6fde9da94-workload-socket" pod "e9973f1f-61b0-4d38-a7c2-b9c6fde9da94" (UID: "e9973f1f-61b0-4d38-a7c2-b9c6fde9da94"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:13.434311 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.434196    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9973f1f-61b0-4d38-a7c2-b9c6fde9da94-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9973f1f-61b0-4d38-a7c2-b9c6fde9da94-workload-certs\") pod \"e9973f1f-61b0-4d38-a7c2-b9c6fde9da94\" (UID: \"e9973f1f-61b0-4d38-a7c2-b9c6fde9da94\") "
Oct 30 19:15:13.434311 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.434216    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9973f1f-61b0-4d38-a7c2-b9c6fde9da94-istiod-ca-cert" pod "e9973f1f-61b0-4d38-a7c2-b9c6fde9da94" (UID: "e9973f1f-61b0-4d38-a7c2-b9c6fde9da94"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:13.434311 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.434296    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9973f1f-61b0-4d38-a7c2-b9c6fde9da94-credential-socket" pod "e9973f1f-61b0-4d38-a7c2-b9c6fde9da94" (UID: "e9973f1f-61b0-4d38-a7c2-b9c6fde9da94"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:13.434480 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.434310    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9973f1f-61b0-4d38-a7c2-b9c6fde9da94-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9973f1f-61b0-4d38-a7c2-b9c6fde9da94-istio-data\") pod \"e9973f1f-61b0-4d38-a7c2-b9c6fde9da94\" (UID: \"e9973f1f-61b0-4d38-a7c2-b9c6fde9da94\") "
Oct 30 19:15:13.434480 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.434425    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9973f1f-61b0-4d38-a7c2-b9c6fde9da94-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9973f1f-61b0-4d38-a7c2-b9c6fde9da94-istio-envoy\") pod \"e9973f1f-61b0-4d38-a7c2-b9c6fde9da94\" (UID: \"e9973f1f-61b0-4d38-a7c2-b9c6fde9da94\") "
Oct 30 19:15:13.434480 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.434462    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9973f1f-61b0-4d38-a7c2-b9c6fde9da94-workload-certs" pod "e9973f1f-61b0-4d38-a7c2-b9c6fde9da94" (UID: "e9973f1f-61b0-4d38-a7c2-b9c6fde9da94"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:13.434649 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.434554    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9973f1f-61b0-4d38-a7c2-b9c6fde9da94-istio-data" pod "e9973f1f-61b0-4d38-a7c2-b9c6fde9da94" (UID: "e9973f1f-61b0-4d38-a7c2-b9c6fde9da94"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:13.435233 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.435209    4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9973f1f-61b0-4d38-a7c2-b9c6fde9da94-workload-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:13.435233 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.435232    4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9973f1f-61b0-4d38-a7c2-b9c6fde9da94-credential-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:13.435439 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.435242    4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/865afd2d-c5e2-4ee5-bfdb-303102de6616-istio-podinfo\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:13.435439 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.435251    4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/865afd2d-c5e2-4ee5-bfdb-303102de6616-istio-token\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:13.435439 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.435259    4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9973f1f-61b0-4d38-a7c2-b9c6fde9da94-workload-certs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:13.435439 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.435268    4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8e73ad5-17f6-4205-bc5f-8965afa49f57-istio-podinfo\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:13.435439 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.435276    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlkhf\" (UniqueName: \"kubernetes.io/projected/865afd2d-c5e2-4ee5-bfdb-303102de6616-kube-api-access-dlkhf\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:13.435439 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.435285    4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/865afd2d-c5e2-4ee5-bfdb-303102de6616-credential-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:13.435439 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.435293    4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/865afd2d-c5e2-4ee5-bfdb-303102de6616-workload-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:13.435439 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.435302    4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9973f1f-61b0-4d38-a7c2-b9c6fde9da94-istio-data\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:13.435439 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.435309    4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8e73ad5-17f6-4205-bc5f-8965afa49f57-istio-token\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:13.435439 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.435318    4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8e73ad5-17f6-4205-bc5f-8965afa49f57-istio-envoy\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:13.435439 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.435327    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdlfj\" (UniqueName: \"kubernetes.io/projected/a8e73ad5-17f6-4205-bc5f-8965afa49f57-kube-api-access-hdlfj\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:13.435439 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.435335    4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9973f1f-61b0-4d38-a7c2-b9c6fde9da94-istiod-ca-cert\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:13.435439 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.435344    4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/865afd2d-c5e2-4ee5-bfdb-303102de6616-istio-envoy\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:13.436403 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.436376    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e9973f1f-61b0-4d38-a7c2-b9c6fde9da94-istio-podinfo" pod "e9973f1f-61b0-4d38-a7c2-b9c6fde9da94" (UID: "e9973f1f-61b0-4d38-a7c2-b9c6fde9da94"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:15:13.436542 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.436523    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9973f1f-61b0-4d38-a7c2-b9c6fde9da94-istio-token" pod "e9973f1f-61b0-4d38-a7c2-b9c6fde9da94" (UID: "e9973f1f-61b0-4d38-a7c2-b9c6fde9da94"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:13.437021 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.436999    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9973f1f-61b0-4d38-a7c2-b9c6fde9da94-istio-envoy" pod "e9973f1f-61b0-4d38-a7c2-b9c6fde9da94" (UID: "e9973f1f-61b0-4d38-a7c2-b9c6fde9da94"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:13.437107 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.437017    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9973f1f-61b0-4d38-a7c2-b9c6fde9da94-kube-api-access-6jnx7" pod "e9973f1f-61b0-4d38-a7c2-b9c6fde9da94" (UID: "e9973f1f-61b0-4d38-a7c2-b9c6fde9da94"). InnerVolumeSpecName "kube-api-access-6jnx7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:13.536615 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.536575    4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9973f1f-61b0-4d38-a7c2-b9c6fde9da94-istio-envoy\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:13.536615 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.536609    4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9973f1f-61b0-4d38-a7c2-b9c6fde9da94-istio-token\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:13.536834 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.536623    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jnx7\" (UniqueName: \"kubernetes.io/projected/e9973f1f-61b0-4d38-a7c2-b9c6fde9da94-kube-api-access-6jnx7\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:13.536834 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.536638    4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9973f1f-61b0-4d38-a7c2-b9c6fde9da94-istio-podinfo\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:13.975954 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.975922    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-lkl5j"
Oct 30 19:15:13.975954 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.975938    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-lkl5j" event={"ID":"865afd2d-c5e2-4ee5-bfdb-303102de6616","Type":"ContainerDied","Data":"b3343dfa8dd428fec24596c5600c4b9f15907f3f540f62d1bf897cc5514734a0"}
Oct 30 19:15:13.976497 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.976029    4525 scope.go:122] "RemoveContainer" containerID="94fdd2bda395afacbf6ccac28c4a3256280c56c1e509c8f03220ecbe8e53a885"
Oct 30 19:15:13.978340 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.978312    4525 generic.go:356] "Generic (PLEG): container finished" podID="a8e73ad5-17f6-4205-bc5f-8965afa49f57" containerID="23114b3fc0f0f25c8de13319f8e7cb8a2484deaf68608b286435ca9cb76b69ef" exitCode=0
Oct 30 19:15:13.978340 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.978338    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b" event={"ID":"a8e73ad5-17f6-4205-bc5f-8965afa49f57","Type":"ContainerDied","Data":"23114b3fc0f0f25c8de13319f8e7cb8a2484deaf68608b286435ca9cb76b69ef"}
Oct 30 19:15:13.978571 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.978394    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b"
Oct 30 19:15:13.978632 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.978399    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b" event={"ID":"a8e73ad5-17f6-4205-bc5f-8965afa49f57","Type":"ContainerDied","Data":"974c48842cdd2e0e53e65c05d6dd060e7865303ea74d8976ce5a8ab5ca56e8c0"}
Oct 30 19:15:13.980468 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.980441    4525 generic.go:356] "Generic (PLEG): container finished" podID="e9973f1f-61b0-4d38-a7c2-b9c6fde9da94" containerID="b0d994c2d6d9338b52f452d8fc13f4c468c9af8dccd52cca1fda8baab62cb723" exitCode=0
Oct 30 19:15:13.980745 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.980521    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-tvbgw"
Oct 30 19:15:13.980745 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.980527    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-tvbgw" event={"ID":"e9973f1f-61b0-4d38-a7c2-b9c6fde9da94","Type":"ContainerDied","Data":"b0d994c2d6d9338b52f452d8fc13f4c468c9af8dccd52cca1fda8baab62cb723"}
Oct 30 19:15:13.980745 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.980568    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-tvbgw" event={"ID":"e9973f1f-61b0-4d38-a7c2-b9c6fde9da94","Type":"ContainerDied","Data":"6fc7abc9c57392b378b758f895740cd9cdc0d5acbad62fdc2bf22eb0a7e62fb5"}
Oct 30 19:15:13.991853 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:13.991645    4525 scope.go:122] "RemoveContainer" containerID="23114b3fc0f0f25c8de13319f8e7cb8a2484deaf68608b286435ca9cb76b69ef"
Oct 30 19:15:14.005377 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:14.005352    4525 scope.go:122] "RemoveContainer" containerID="1cfe61de8850135e71e9c6cd76641bf7efac9ef158b8006ebc4be79165fb08e3"
Oct 30 19:15:14.006050 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:14.005966    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-lkl5j"]
Oct 30 19:15:14.012597 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:14.012576    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-lkl5j"]
Oct 30 19:15:14.024094 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:14.022445    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-tvbgw"]
Oct 30 19:15:14.025791 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:14.025768    4525 scope.go:122] "RemoveContainer" containerID="23114b3fc0f0f25c8de13319f8e7cb8a2484deaf68608b286435ca9cb76b69ef"
Oct 30 19:15:14.026266 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:15:14.026240    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23114b3fc0f0f25c8de13319f8e7cb8a2484deaf68608b286435ca9cb76b69ef\": container with ID starting with 23114b3fc0f0f25c8de13319f8e7cb8a2484deaf68608b286435ca9cb76b69ef not found: ID does not exist" containerID="23114b3fc0f0f25c8de13319f8e7cb8a2484deaf68608b286435ca9cb76b69ef"
Oct 30 19:15:14.026361 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:14.026274    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23114b3fc0f0f25c8de13319f8e7cb8a2484deaf68608b286435ca9cb76b69ef"} err="failed to get container status \"23114b3fc0f0f25c8de13319f8e7cb8a2484deaf68608b286435ca9cb76b69ef\": rpc error: code = NotFound desc = could not find container \"23114b3fc0f0f25c8de13319f8e7cb8a2484deaf68608b286435ca9cb76b69ef\": container with ID starting with 23114b3fc0f0f25c8de13319f8e7cb8a2484deaf68608b286435ca9cb76b69ef not found: ID does not exist"
Oct 30 19:15:14.026361 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:14.026301    4525 scope.go:122] "RemoveContainer" containerID="1cfe61de8850135e71e9c6cd76641bf7efac9ef158b8006ebc4be79165fb08e3"
Oct 30 19:15:14.026646 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:15:14.026629    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cfe61de8850135e71e9c6cd76641bf7efac9ef158b8006ebc4be79165fb08e3\": container with ID starting with 1cfe61de8850135e71e9c6cd76641bf7efac9ef158b8006ebc4be79165fb08e3 not found: ID does not exist" containerID="1cfe61de8850135e71e9c6cd76641bf7efac9ef158b8006ebc4be79165fb08e3"
Oct 30 19:15:14.026763 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:14.026657    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cfe61de8850135e71e9c6cd76641bf7efac9ef158b8006ebc4be79165fb08e3"} err="failed to get container status \"1cfe61de8850135e71e9c6cd76641bf7efac9ef158b8006ebc4be79165fb08e3\": rpc error: code = NotFound desc = could not find container \"1cfe61de8850135e71e9c6cd76641bf7efac9ef158b8006ebc4be79165fb08e3\": container with ID starting with 1cfe61de8850135e71e9c6cd76641bf7efac9ef158b8006ebc4be79165fb08e3 not found: ID does not exist"
Oct 30 19:15:14.026763 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:14.026722    4525 scope.go:122] "RemoveContainer" containerID="b0d994c2d6d9338b52f452d8fc13f4c468c9af8dccd52cca1fda8baab62cb723"
Oct 30 19:15:14.034014 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:14.032126    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-tvbgw"]
Oct 30 19:15:14.044591 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:14.044522    4525 scope.go:122] "RemoveContainer" containerID="b0d994c2d6d9338b52f452d8fc13f4c468c9af8dccd52cca1fda8baab62cb723"
Oct 30 19:15:14.045028 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:15:14.045005    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0d994c2d6d9338b52f452d8fc13f4c468c9af8dccd52cca1fda8baab62cb723\": container with ID starting with b0d994c2d6d9338b52f452d8fc13f4c468c9af8dccd52cca1fda8baab62cb723 not found: ID does not exist" containerID="b0d994c2d6d9338b52f452d8fc13f4c468c9af8dccd52cca1fda8baab62cb723"
Oct 30 19:15:14.045134 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:14.045038    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0d994c2d6d9338b52f452d8fc13f4c468c9af8dccd52cca1fda8baab62cb723"} err="failed to get container status \"b0d994c2d6d9338b52f452d8fc13f4c468c9af8dccd52cca1fda8baab62cb723\": rpc error: code = NotFound desc = could not find container \"b0d994c2d6d9338b52f452d8fc13f4c468c9af8dccd52cca1fda8baab62cb723\": container with ID starting with b0d994c2d6d9338b52f452d8fc13f4c468c9af8dccd52cca1fda8baab62cb723 not found: ID does not exist"
Oct 30 19:15:14.053938 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:14.053912    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b"]
Oct 30 19:15:14.060127 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:14.060104    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-e51c10-openshift-default-54b5584fb5-sm85b"]
Oct 30 19:15:15.489402 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:15.489368    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm"]
Oct 30 19:15:15.489919 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:15.489659    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm" podUID="6ccf27c7-f6d3-457b-9ff8-69b54f9b9287" containerName="istio-proxy" containerID="cri-o://ebc7f76d96aafbd104dfa29a040cffffaf1790bdc6806aa0827284f5fe573d09" gracePeriod=30
Oct 30 19:15:15.548151 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:15.548110    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="865afd2d-c5e2-4ee5-bfdb-303102de6616" path="/var/lib/kubelet/pods/865afd2d-c5e2-4ee5-bfdb-303102de6616/volumes"
Oct 30 19:15:15.548713 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:15.548697    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8e73ad5-17f6-4205-bc5f-8965afa49f57" path="/var/lib/kubelet/pods/a8e73ad5-17f6-4205-bc5f-8965afa49f57/volumes"
Oct 30 19:15:15.549197 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:15.549183    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9973f1f-61b0-4d38-a7c2-b9c6fde9da94" path="/var/lib/kubelet/pods/e9973f1f-61b0-4d38-a7c2-b9c6fde9da94/volumes"
Oct 30 19:15:15.674264 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:15.673829    4525 patch_prober.go:29] interesting pod/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:15:15.674264 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:15.673887    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7" podUID="77adecb1-af53-4dd9-bb52-d8e22ebb03e4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:15:15.906708 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:15.906026    4525 patch_prober.go:29] interesting pod/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-99fmr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:15:15.906708 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:15.906121    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-99fmr" podUID="be563a8d-e16a-4b6a-b82d-4d6c4847a527" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:15:16.785338 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:16.785308    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-99fmr"
Oct 30 19:15:16.867723 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:16.867620    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-mfdt7"
Oct 30 19:15:16.880525 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:16.880499    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7"
Oct 30 19:15:16.946182 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:16.946148    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be16530b-c26d-4da7-b25e-f5246eff6c3d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be16530b-c26d-4da7-b25e-f5246eff6c3d-credential-socket\") pod \"be16530b-c26d-4da7-b25e-f5246eff6c3d\" (UID: \"be16530b-c26d-4da7-b25e-f5246eff6c3d\") "
Oct 30 19:15:16.946342 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:16.946231    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be16530b-c26d-4da7-b25e-f5246eff6c3d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be16530b-c26d-4da7-b25e-f5246eff6c3d-istio-data\") pod \"be16530b-c26d-4da7-b25e-f5246eff6c3d\" (UID: \"be16530b-c26d-4da7-b25e-f5246eff6c3d\") "
Oct 30 19:15:16.946342 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:16.946262    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be16530b-c26d-4da7-b25e-f5246eff6c3d-istio-token\" (UniqueName: \"kubernetes.io/projected/be16530b-c26d-4da7-b25e-f5246eff6c3d-istio-token\") pod \"be16530b-c26d-4da7-b25e-f5246eff6c3d\" (UID: \"be16530b-c26d-4da7-b25e-f5246eff6c3d\") "
Oct 30 19:15:16.946342 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:16.946283    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be16530b-c26d-4da7-b25e-f5246eff6c3d-kube-api-access-m5dqn\" (UniqueName: \"kubernetes.io/projected/be16530b-c26d-4da7-b25e-f5246eff6c3d-kube-api-access-m5dqn\") pod \"be16530b-c26d-4da7-b25e-f5246eff6c3d\" (UID: \"be16530b-c26d-4da7-b25e-f5246eff6c3d\") "
Oct 30 19:15:16.946342 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:16.946338    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be563a8d-e16a-4b6a-b82d-4d6c4847a527-istio-token\" (UniqueName: \"kubernetes.io/projected/be563a8d-e16a-4b6a-b82d-4d6c4847a527-istio-token\") pod \"be563a8d-e16a-4b6a-b82d-4d6c4847a527\" (UID: \"be563a8d-e16a-4b6a-b82d-4d6c4847a527\") "
Oct 30 19:15:16.946580 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:16.946397    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be563a8d-e16a-4b6a-b82d-4d6c4847a527-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be563a8d-e16a-4b6a-b82d-4d6c4847a527-workload-certs\") pod \"be563a8d-e16a-4b6a-b82d-4d6c4847a527\" (UID: \"be563a8d-e16a-4b6a-b82d-4d6c4847a527\") "
Oct 30 19:15:16.946580 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:16.946464    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be16530b-c26d-4da7-b25e-f5246eff6c3d-credential-socket" pod "be16530b-c26d-4da7-b25e-f5246eff6c3d" (UID: "be16530b-c26d-4da7-b25e-f5246eff6c3d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:16.946580 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:16.946493    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be16530b-c26d-4da7-b25e-f5246eff6c3d-istio-data" pod "be16530b-c26d-4da7-b25e-f5246eff6c3d" (UID: "be16530b-c26d-4da7-b25e-f5246eff6c3d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:16.946580 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:16.946535    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/be563a8d-e16a-4b6a-b82d-4d6c4847a527-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be563a8d-e16a-4b6a-b82d-4d6c4847a527-istio-podinfo\") pod \"be563a8d-e16a-4b6a-b82d-4d6c4847a527\" (UID: \"be563a8d-e16a-4b6a-b82d-4d6c4847a527\") "
Oct 30 19:15:16.946580 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:16.946575    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be563a8d-e16a-4b6a-b82d-4d6c4847a527-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be563a8d-e16a-4b6a-b82d-4d6c4847a527-workload-socket\") pod \"be563a8d-e16a-4b6a-b82d-4d6c4847a527\" (UID: \"be563a8d-e16a-4b6a-b82d-4d6c4847a527\") "
Oct 30 19:15:16.946885 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:16.946606    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be563a8d-e16a-4b6a-b82d-4d6c4847a527-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be563a8d-e16a-4b6a-b82d-4d6c4847a527-credential-socket\") pod \"be563a8d-e16a-4b6a-b82d-4d6c4847a527\" (UID: \"be563a8d-e16a-4b6a-b82d-4d6c4847a527\") "
Oct 30 19:15:16.946885 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:16.946664    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/be16530b-c26d-4da7-b25e-f5246eff6c3d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be16530b-c26d-4da7-b25e-f5246eff6c3d-istio-podinfo\") pod \"be16530b-c26d-4da7-b25e-f5246eff6c3d\" (UID: \"be16530b-c26d-4da7-b25e-f5246eff6c3d\") "
Oct 30 19:15:16.946885 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:16.946815    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be563a8d-e16a-4b6a-b82d-4d6c4847a527-kube-api-access-lwlwk\" (UniqueName: \"kubernetes.io/projected/be563a8d-e16a-4b6a-b82d-4d6c4847a527-kube-api-access-lwlwk\") pod \"be563a8d-e16a-4b6a-b82d-4d6c4847a527\" (UID: \"be563a8d-e16a-4b6a-b82d-4d6c4847a527\") "
Oct 30 19:15:16.946885 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:16.946843    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be563a8d-e16a-4b6a-b82d-4d6c4847a527-workload-certs" pod "be563a8d-e16a-4b6a-b82d-4d6c4847a527" (UID: "be563a8d-e16a-4b6a-b82d-4d6c4847a527"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:16.946885 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:16.946866    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be16530b-c26d-4da7-b25e-f5246eff6c3d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be16530b-c26d-4da7-b25e-f5246eff6c3d-istiod-ca-cert\") pod \"be16530b-c26d-4da7-b25e-f5246eff6c3d\" (UID: \"be16530b-c26d-4da7-b25e-f5246eff6c3d\") "
Oct 30 19:15:16.947315 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:16.946908    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be563a8d-e16a-4b6a-b82d-4d6c4847a527-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be563a8d-e16a-4b6a-b82d-4d6c4847a527-istiod-ca-cert\") pod \"be563a8d-e16a-4b6a-b82d-4d6c4847a527\" (UID: \"be563a8d-e16a-4b6a-b82d-4d6c4847a527\") "
Oct 30 19:15:16.947315 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:16.946967    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be563a8d-e16a-4b6a-b82d-4d6c4847a527-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be563a8d-e16a-4b6a-b82d-4d6c4847a527-istio-envoy\") pod \"be563a8d-e16a-4b6a-b82d-4d6c4847a527\" (UID: \"be563a8d-e16a-4b6a-b82d-4d6c4847a527\") "
Oct 30 19:15:16.947315 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:16.946999    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be16530b-c26d-4da7-b25e-f5246eff6c3d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be16530b-c26d-4da7-b25e-f5246eff6c3d-workload-certs\") pod \"be16530b-c26d-4da7-b25e-f5246eff6c3d\" (UID: \"be16530b-c26d-4da7-b25e-f5246eff6c3d\") "
Oct 30 19:15:16.947315 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:16.947139    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be563a8d-e16a-4b6a-b82d-4d6c4847a527-credential-socket" pod "be563a8d-e16a-4b6a-b82d-4d6c4847a527" (UID: "be563a8d-e16a-4b6a-b82d-4d6c4847a527"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:16.947315 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:16.947230    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be16530b-c26d-4da7-b25e-f5246eff6c3d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be16530b-c26d-4da7-b25e-f5246eff6c3d-istio-envoy\") pod \"be16530b-c26d-4da7-b25e-f5246eff6c3d\" (UID: \"be16530b-c26d-4da7-b25e-f5246eff6c3d\") "
Oct 30 19:15:16.947315 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:16.947267    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be16530b-c26d-4da7-b25e-f5246eff6c3d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be16530b-c26d-4da7-b25e-f5246eff6c3d-workload-socket\") pod \"be16530b-c26d-4da7-b25e-f5246eff6c3d\" (UID: \"be16530b-c26d-4da7-b25e-f5246eff6c3d\") "
Oct 30 19:15:16.947639 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:16.947354    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be563a8d-e16a-4b6a-b82d-4d6c4847a527-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be563a8d-e16a-4b6a-b82d-4d6c4847a527-istio-data\") pod \"be563a8d-e16a-4b6a-b82d-4d6c4847a527\" (UID: \"be563a8d-e16a-4b6a-b82d-4d6c4847a527\") "
Oct 30 19:15:16.947932 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:16.947904    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be563a8d-e16a-4b6a-b82d-4d6c4847a527-workload-socket" pod "be563a8d-e16a-4b6a-b82d-4d6c4847a527" (UID: "be563a8d-e16a-4b6a-b82d-4d6c4847a527"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:16.948317 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:16.948294    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be563a8d-e16a-4b6a-b82d-4d6c4847a527-istio-data" pod "be563a8d-e16a-4b6a-b82d-4d6c4847a527" (UID: "be563a8d-e16a-4b6a-b82d-4d6c4847a527"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:16.948501 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:16.948478    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be16530b-c26d-4da7-b25e-f5246eff6c3d-workload-certs" pod "be16530b-c26d-4da7-b25e-f5246eff6c3d" (UID: "be16530b-c26d-4da7-b25e-f5246eff6c3d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:16.948696 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:16.948654    4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be563a8d-e16a-4b6a-b82d-4d6c4847a527-istio-data\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:16.948836 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:16.948809    4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be16530b-c26d-4da7-b25e-f5246eff6c3d-credential-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:16.948836 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:16.948838    4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be16530b-c26d-4da7-b25e-f5246eff6c3d-istio-data\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:16.949066 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:16.948852    4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be563a8d-e16a-4b6a-b82d-4d6c4847a527-workload-certs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:16.949066 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:16.948866    4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be563a8d-e16a-4b6a-b82d-4d6c4847a527-workload-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:16.949066 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:16.948880    4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be563a8d-e16a-4b6a-b82d-4d6c4847a527-credential-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:16.949349 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:16.949153    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be16530b-c26d-4da7-b25e-f5246eff6c3d-istiod-ca-cert" pod "be16530b-c26d-4da7-b25e-f5246eff6c3d" (UID: "be16530b-c26d-4da7-b25e-f5246eff6c3d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:16.950485 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:16.950300    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be16530b-c26d-4da7-b25e-f5246eff6c3d-istio-token" pod "be16530b-c26d-4da7-b25e-f5246eff6c3d" (UID: "be16530b-c26d-4da7-b25e-f5246eff6c3d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:16.950485 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:16.950345    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be16530b-c26d-4da7-b25e-f5246eff6c3d-kube-api-access-m5dqn" pod "be16530b-c26d-4da7-b25e-f5246eff6c3d" (UID: "be16530b-c26d-4da7-b25e-f5246eff6c3d"). InnerVolumeSpecName "kube-api-access-m5dqn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:16.950485 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:16.950355    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/be16530b-c26d-4da7-b25e-f5246eff6c3d-istio-podinfo" pod "be16530b-c26d-4da7-b25e-f5246eff6c3d" (UID: "be16530b-c26d-4da7-b25e-f5246eff6c3d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:15:16.950709 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:16.950594    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be16530b-c26d-4da7-b25e-f5246eff6c3d-workload-socket" pod "be16530b-c26d-4da7-b25e-f5246eff6c3d" (UID: "be16530b-c26d-4da7-b25e-f5246eff6c3d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:16.950709 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:16.950664    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be563a8d-e16a-4b6a-b82d-4d6c4847a527-istiod-ca-cert" pod "be563a8d-e16a-4b6a-b82d-4d6c4847a527" (UID: "be563a8d-e16a-4b6a-b82d-4d6c4847a527"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:16.950969 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:16.950920    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be16530b-c26d-4da7-b25e-f5246eff6c3d-istio-envoy" pod "be16530b-c26d-4da7-b25e-f5246eff6c3d" (UID: "be16530b-c26d-4da7-b25e-f5246eff6c3d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:16.951421 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:16.951397    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/be563a8d-e16a-4b6a-b82d-4d6c4847a527-istio-podinfo" pod "be563a8d-e16a-4b6a-b82d-4d6c4847a527" (UID: "be563a8d-e16a-4b6a-b82d-4d6c4847a527"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:15:16.951856 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:16.951736    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be563a8d-e16a-4b6a-b82d-4d6c4847a527-istio-token" pod "be563a8d-e16a-4b6a-b82d-4d6c4847a527" (UID: "be563a8d-e16a-4b6a-b82d-4d6c4847a527"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:16.951856 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:16.951768    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be563a8d-e16a-4b6a-b82d-4d6c4847a527-istio-envoy" pod "be563a8d-e16a-4b6a-b82d-4d6c4847a527" (UID: "be563a8d-e16a-4b6a-b82d-4d6c4847a527"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:16.952189 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:16.952170    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be563a8d-e16a-4b6a-b82d-4d6c4847a527-kube-api-access-lwlwk" pod "be563a8d-e16a-4b6a-b82d-4d6c4847a527" (UID: "be563a8d-e16a-4b6a-b82d-4d6c4847a527"). InnerVolumeSpecName "kube-api-access-lwlwk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:17.049993 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:17.049902    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/77adecb1-af53-4dd9-bb52-d8e22ebb03e4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77adecb1-af53-4dd9-bb52-d8e22ebb03e4-istio-podinfo\") pod \"77adecb1-af53-4dd9-bb52-d8e22ebb03e4\" (UID: \"77adecb1-af53-4dd9-bb52-d8e22ebb03e4\") "
Oct 30 19:15:17.049993 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:17.049954    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77adecb1-af53-4dd9-bb52-d8e22ebb03e4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77adecb1-af53-4dd9-bb52-d8e22ebb03e4-credential-socket\") pod \"77adecb1-af53-4dd9-bb52-d8e22ebb03e4\" (UID: \"77adecb1-af53-4dd9-bb52-d8e22ebb03e4\") "
Oct 30 19:15:17.050234 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:17.050030    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77adecb1-af53-4dd9-bb52-d8e22ebb03e4-kube-api-access-nzx5n\" (UniqueName: \"kubernetes.io/projected/77adecb1-af53-4dd9-bb52-d8e22ebb03e4-kube-api-access-nzx5n\") pod \"77adecb1-af53-4dd9-bb52-d8e22ebb03e4\" (UID: \"77adecb1-af53-4dd9-bb52-d8e22ebb03e4\") "
Oct 30 19:15:17.050234 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:17.050175    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77adecb1-af53-4dd9-bb52-d8e22ebb03e4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77adecb1-af53-4dd9-bb52-d8e22ebb03e4-workload-socket\") pod \"77adecb1-af53-4dd9-bb52-d8e22ebb03e4\" (UID: \"77adecb1-af53-4dd9-bb52-d8e22ebb03e4\") "
Oct 30 19:15:17.050345 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:17.050227    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77adecb1-af53-4dd9-bb52-d8e22ebb03e4-credential-socket" pod "77adecb1-af53-4dd9-bb52-d8e22ebb03e4" (UID: "77adecb1-af53-4dd9-bb52-d8e22ebb03e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:17.050345 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:17.050243    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77adecb1-af53-4dd9-bb52-d8e22ebb03e4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77adecb1-af53-4dd9-bb52-d8e22ebb03e4-istio-data\") pod \"77adecb1-af53-4dd9-bb52-d8e22ebb03e4\" (UID: \"77adecb1-af53-4dd9-bb52-d8e22ebb03e4\") "
Oct 30 19:15:17.050345 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:17.050340    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77adecb1-af53-4dd9-bb52-d8e22ebb03e4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77adecb1-af53-4dd9-bb52-d8e22ebb03e4-istiod-ca-cert\") pod \"77adecb1-af53-4dd9-bb52-d8e22ebb03e4\" (UID: \"77adecb1-af53-4dd9-bb52-d8e22ebb03e4\") "
Oct 30 19:15:17.050499 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:17.050385    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77adecb1-af53-4dd9-bb52-d8e22ebb03e4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77adecb1-af53-4dd9-bb52-d8e22ebb03e4-workload-certs\") pod \"77adecb1-af53-4dd9-bb52-d8e22ebb03e4\" (UID: \"77adecb1-af53-4dd9-bb52-d8e22ebb03e4\") "
Oct 30 19:15:17.050499 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:17.050451    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77adecb1-af53-4dd9-bb52-d8e22ebb03e4-workload-socket" pod "77adecb1-af53-4dd9-bb52-d8e22ebb03e4" (UID: "77adecb1-af53-4dd9-bb52-d8e22ebb03e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:17.050606 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:17.050493    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77adecb1-af53-4dd9-bb52-d8e22ebb03e4-istio-data" pod "77adecb1-af53-4dd9-bb52-d8e22ebb03e4" (UID: "77adecb1-af53-4dd9-bb52-d8e22ebb03e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:17.050606 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:17.050552    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77adecb1-af53-4dd9-bb52-d8e22ebb03e4-istio-token\" (UniqueName: \"kubernetes.io/projected/77adecb1-af53-4dd9-bb52-d8e22ebb03e4-istio-token\") pod \"77adecb1-af53-4dd9-bb52-d8e22ebb03e4\" (UID: \"77adecb1-af53-4dd9-bb52-d8e22ebb03e4\") "
Oct 30 19:15:17.050750 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:17.050602    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77adecb1-af53-4dd9-bb52-d8e22ebb03e4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77adecb1-af53-4dd9-bb52-d8e22ebb03e4-istio-envoy\") pod \"77adecb1-af53-4dd9-bb52-d8e22ebb03e4\" (UID: \"77adecb1-af53-4dd9-bb52-d8e22ebb03e4\") "
Oct 30 19:15:17.050978 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:17.050955    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77adecb1-af53-4dd9-bb52-d8e22ebb03e4-workload-certs" pod "77adecb1-af53-4dd9-bb52-d8e22ebb03e4" (UID: "77adecb1-af53-4dd9-bb52-d8e22ebb03e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:17.050978 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:17.050963    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77adecb1-af53-4dd9-bb52-d8e22ebb03e4-istiod-ca-cert" pod "77adecb1-af53-4dd9-bb52-d8e22ebb03e4" (UID: "77adecb1-af53-4dd9-bb52-d8e22ebb03e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:17.051408 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:17.051379    4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be563a8d-e16a-4b6a-b82d-4d6c4847a527-istio-podinfo\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:17.051561 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:17.051411    4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be16530b-c26d-4da7-b25e-f5246eff6c3d-istio-podinfo\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:17.051561 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:17.051426    4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77adecb1-af53-4dd9-bb52-d8e22ebb03e4-workload-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:17.051561 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:17.051460    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwlwk\" (UniqueName: \"kubernetes.io/projected/be563a8d-e16a-4b6a-b82d-4d6c4847a527-kube-api-access-lwlwk\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:17.051784 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:17.051590    4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be16530b-c26d-4da7-b25e-f5246eff6c3d-istiod-ca-cert\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:17.051784 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:17.051632    4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77adecb1-af53-4dd9-bb52-d8e22ebb03e4-istio-data\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:17.051784 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:17.051646    4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be563a8d-e16a-4b6a-b82d-4d6c4847a527-istiod-ca-cert\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:17.051784 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:17.051661    4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be563a8d-e16a-4b6a-b82d-4d6c4847a527-istio-envoy\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:17.051784 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:17.051675    4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be16530b-c26d-4da7-b25e-f5246eff6c3d-workload-certs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:17.051784 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:17.051706    4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77adecb1-af53-4dd9-bb52-d8e22ebb03e4-istiod-ca-cert\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:17.051784 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:17.051718    4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be16530b-c26d-4da7-b25e-f5246eff6c3d-istio-envoy\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:17.051784 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:17.051732    4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be16530b-c26d-4da7-b25e-f5246eff6c3d-workload-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:17.051784 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:17.051747    4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77adecb1-af53-4dd9-bb52-d8e22ebb03e4-workload-certs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:17.051784 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:17.051762    4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be16530b-c26d-4da7-b25e-f5246eff6c3d-istio-token\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:17.051784 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:17.051777    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5dqn\" (UniqueName: \"kubernetes.io/projected/be16530b-c26d-4da7-b25e-f5246eff6c3d-kube-api-access-m5dqn\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:17.052378 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:17.051793    4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be563a8d-e16a-4b6a-b82d-4d6c4847a527-istio-token\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:17.052378 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:17.051806    4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77adecb1-af53-4dd9-bb52-d8e22ebb03e4-credential-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:17.052575 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:17.052550    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77adecb1-af53-4dd9-bb52-d8e22ebb03e4-kube-api-access-nzx5n" pod "77adecb1-af53-4dd9-bb52-d8e22ebb03e4" (UID: "77adecb1-af53-4dd9-bb52-d8e22ebb03e4"). InnerVolumeSpecName "kube-api-access-nzx5n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:17.052667 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:17.052649    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/77adecb1-af53-4dd9-bb52-d8e22ebb03e4-istio-podinfo" pod "77adecb1-af53-4dd9-bb52-d8e22ebb03e4" (UID: "77adecb1-af53-4dd9-bb52-d8e22ebb03e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:15:17.053193 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:17.053172    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77adecb1-af53-4dd9-bb52-d8e22ebb03e4-istio-envoy" pod "77adecb1-af53-4dd9-bb52-d8e22ebb03e4" (UID: "77adecb1-af53-4dd9-bb52-d8e22ebb03e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:17.053304 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:17.053211    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77adecb1-af53-4dd9-bb52-d8e22ebb03e4-istio-token" pod "77adecb1-af53-4dd9-bb52-d8e22ebb03e4" (UID: "77adecb1-af53-4dd9-bb52-d8e22ebb03e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:17.061593 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:17.061566    4525 generic.go:356] "Generic (PLEG): container finished" podID="be16530b-c26d-4da7-b25e-f5246eff6c3d" containerID="09f5a51a5134a8acef4c236f5cd80462bf04576891b2f0856f510489f51e1a9e" exitCode=0
Oct 30 19:15:17.061707 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:17.061621    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-mfdt7"
Oct 30 19:15:17.061707 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:17.061650    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-mfdt7" event={"ID":"be16530b-c26d-4da7-b25e-f5246eff6c3d","Type":"ContainerDied","Data":"09f5a51a5134a8acef4c236f5cd80462bf04576891b2f0856f510489f51e1a9e"}
Oct 30 19:15:17.061817 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:17.061775    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-mfdt7" event={"ID":"be16530b-c26d-4da7-b25e-f5246eff6c3d","Type":"ContainerDied","Data":"fb6a77bb5fce8a7da75eda45f47dcf56bd0ec1ca4ad69532fc887d503568a238"}
Oct 30 19:15:17.061817 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:17.061803    4525 scope.go:122] "RemoveContainer" containerID="09f5a51a5134a8acef4c236f5cd80462bf04576891b2f0856f510489f51e1a9e"
Oct 30 19:15:17.063784 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:17.063763    4525 generic.go:356] "Generic (PLEG): container finished" podID="77adecb1-af53-4dd9-bb52-d8e22ebb03e4" containerID="4249fc47d1098fc9d135027761840864157ed12000e934280f9bcd9b3ed2d5be" exitCode=0
Oct 30 19:15:17.063891 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:17.063818    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7"
Oct 30 19:15:17.063891 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:17.063854    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7" event={"ID":"77adecb1-af53-4dd9-bb52-d8e22ebb03e4","Type":"ContainerDied","Data":"4249fc47d1098fc9d135027761840864157ed12000e934280f9bcd9b3ed2d5be"}
Oct 30 19:15:17.063987 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:17.063903    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7" event={"ID":"77adecb1-af53-4dd9-bb52-d8e22ebb03e4","Type":"ContainerDied","Data":"8463bda4408213f0eb97c7e2bb4af7faca48abf10a01e5b15707f925908e05b0"}
Oct 30 19:15:17.065329 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:17.065303    4525 generic.go:356] "Generic (PLEG): container finished" podID="be563a8d-e16a-4b6a-b82d-4d6c4847a527" containerID="a50a4388e06b290cfb03ef75805f28936ed1f88439497a667234524251e1ac2e" exitCode=0
Oct 30 19:15:17.065409 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:17.065342    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-99fmr" event={"ID":"be563a8d-e16a-4b6a-b82d-4d6c4847a527","Type":"ContainerDied","Data":"a50a4388e06b290cfb03ef75805f28936ed1f88439497a667234524251e1ac2e"}
Oct 30 19:15:17.065409 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:17.065358    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-99fmr"
Oct 30 19:15:17.065409 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:17.065394    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-99fmr" event={"ID":"be563a8d-e16a-4b6a-b82d-4d6c4847a527","Type":"ContainerDied","Data":"01c8aa1cc13d1ab9ad1df6ad4bd4371d7b645d515d04439c43665b57d9bb2eff"}
Oct 30 19:15:17.076418 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:17.076391    4525 scope.go:122] "RemoveContainer" containerID="09f5a51a5134a8acef4c236f5cd80462bf04576891b2f0856f510489f51e1a9e"
Oct 30 19:15:17.076721 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:15:17.076703    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09f5a51a5134a8acef4c236f5cd80462bf04576891b2f0856f510489f51e1a9e\": container with ID starting with 09f5a51a5134a8acef4c236f5cd80462bf04576891b2f0856f510489f51e1a9e not found: ID does not exist" containerID="09f5a51a5134a8acef4c236f5cd80462bf04576891b2f0856f510489f51e1a9e"
Oct 30 19:15:17.076771 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:17.076730    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09f5a51a5134a8acef4c236f5cd80462bf04576891b2f0856f510489f51e1a9e"} err="failed to get container status \"09f5a51a5134a8acef4c236f5cd80462bf04576891b2f0856f510489f51e1a9e\": rpc error: code = NotFound desc = could not find container \"09f5a51a5134a8acef4c236f5cd80462bf04576891b2f0856f510489f51e1a9e\": container with ID starting with 09f5a51a5134a8acef4c236f5cd80462bf04576891b2f0856f510489f51e1a9e not found: ID does not exist"
Oct 30 19:15:17.076771 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:17.076747    4525 scope.go:122] "RemoveContainer" containerID="4249fc47d1098fc9d135027761840864157ed12000e934280f9bcd9b3ed2d5be"
Oct 30 19:15:17.089819 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:17.089794    4525 scope.go:122] "RemoveContainer" containerID="052da6685f672c65b36233b5ccaf20546eb09e6184d1ec4d7da5fc1a377ff3ec"
Oct 30 19:15:17.099794 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:17.099770    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-99fmr"]
Oct 30 19:15:17.107483 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:17.107460    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-99fmr"]
Oct 30 19:15:17.109200 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:17.109179    4525 scope.go:122] "RemoveContainer" containerID="4249fc47d1098fc9d135027761840864157ed12000e934280f9bcd9b3ed2d5be"
Oct 30 19:15:17.109495 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:15:17.109472    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4249fc47d1098fc9d135027761840864157ed12000e934280f9bcd9b3ed2d5be\": container with ID starting with 4249fc47d1098fc9d135027761840864157ed12000e934280f9bcd9b3ed2d5be not found: ID does not exist" containerID="4249fc47d1098fc9d135027761840864157ed12000e934280f9bcd9b3ed2d5be"
Oct 30 19:15:17.109546 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:17.109506    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4249fc47d1098fc9d135027761840864157ed12000e934280f9bcd9b3ed2d5be"} err="failed to get container status \"4249fc47d1098fc9d135027761840864157ed12000e934280f9bcd9b3ed2d5be\": rpc error: code = NotFound desc = could not find container \"4249fc47d1098fc9d135027761840864157ed12000e934280f9bcd9b3ed2d5be\": container with ID starting with 4249fc47d1098fc9d135027761840864157ed12000e934280f9bcd9b3ed2d5be not found: ID does not exist"
Oct 30 19:15:17.109546 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:17.109525    4525 scope.go:122] "RemoveContainer" containerID="052da6685f672c65b36233b5ccaf20546eb09e6184d1ec4d7da5fc1a377ff3ec"
Oct 30 19:15:17.109863 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:15:17.109842    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"052da6685f672c65b36233b5ccaf20546eb09e6184d1ec4d7da5fc1a377ff3ec\": container with ID starting with 052da6685f672c65b36233b5ccaf20546eb09e6184d1ec4d7da5fc1a377ff3ec not found: ID does not exist" containerID="052da6685f672c65b36233b5ccaf20546eb09e6184d1ec4d7da5fc1a377ff3ec"
Oct 30 19:15:17.109925 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:17.109868    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"052da6685f672c65b36233b5ccaf20546eb09e6184d1ec4d7da5fc1a377ff3ec"} err="failed to get container status \"052da6685f672c65b36233b5ccaf20546eb09e6184d1ec4d7da5fc1a377ff3ec\": rpc error: code = NotFound desc = could not find container \"052da6685f672c65b36233b5ccaf20546eb09e6184d1ec4d7da5fc1a377ff3ec\": container with ID starting with 052da6685f672c65b36233b5ccaf20546eb09e6184d1ec4d7da5fc1a377ff3ec not found: ID does not exist"
Oct 30 19:15:17.109925 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:17.109884    4525 scope.go:122] "RemoveContainer" containerID="a50a4388e06b290cfb03ef75805f28936ed1f88439497a667234524251e1ac2e"
Oct 30 19:15:17.115298 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:17.115278    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-mfdt7"]
Oct 30 19:15:17.120450 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:17.120430    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-mfdt7"]
Oct 30 19:15:17.123006 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:17.122990    4525 scope.go:122] "RemoveContainer" containerID="a50a4388e06b290cfb03ef75805f28936ed1f88439497a667234524251e1ac2e"
Oct 30 19:15:17.123316 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:15:17.123298    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a50a4388e06b290cfb03ef75805f28936ed1f88439497a667234524251e1ac2e\": container with ID starting with a50a4388e06b290cfb03ef75805f28936ed1f88439497a667234524251e1ac2e not found: ID does not exist" containerID="a50a4388e06b290cfb03ef75805f28936ed1f88439497a667234524251e1ac2e"
Oct 30 19:15:17.123356 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:17.123324    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a50a4388e06b290cfb03ef75805f28936ed1f88439497a667234524251e1ac2e"} err="failed to get container status \"a50a4388e06b290cfb03ef75805f28936ed1f88439497a667234524251e1ac2e\": rpc error: code = NotFound desc = could not find container \"a50a4388e06b290cfb03ef75805f28936ed1f88439497a667234524251e1ac2e\": container with ID starting with a50a4388e06b290cfb03ef75805f28936ed1f88439497a667234524251e1ac2e not found: ID does not exist"
Oct 30 19:15:17.134976 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:17.134943    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7"]
Oct 30 19:15:17.139723 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:17.139702    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-e67d95-openshift-default-7db5c6f57c-g55d7"]
Oct 30 19:15:17.153236 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:17.153214    4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77adecb1-af53-4dd9-bb52-d8e22ebb03e4-istio-token\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:17.153631 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:17.153238    4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77adecb1-af53-4dd9-bb52-d8e22ebb03e4-istio-envoy\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:17.153631 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:17.153249    4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77adecb1-af53-4dd9-bb52-d8e22ebb03e4-istio-podinfo\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:17.153631 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:17.153259    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzx5n\" (UniqueName: \"kubernetes.io/projected/77adecb1-af53-4dd9-bb52-d8e22ebb03e4-kube-api-access-nzx5n\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:17.400882 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:15:17.400845    4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d49l" containerName="extractor"
Oct 30 19:15:17.548035 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:17.548006    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77adecb1-af53-4dd9-bb52-d8e22ebb03e4" path="/var/lib/kubelet/pods/77adecb1-af53-4dd9-bb52-d8e22ebb03e4/volumes"
Oct 30 19:15:17.548589 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:17.548575    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be16530b-c26d-4da7-b25e-f5246eff6c3d" path="/var/lib/kubelet/pods/be16530b-c26d-4da7-b25e-f5246eff6c3d/volumes"
Oct 30 19:15:17.549118 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:17.549102    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be563a8d-e16a-4b6a-b82d-4d6c4847a527" path="/var/lib/kubelet/pods/be563a8d-e16a-4b6a-b82d-4d6c4847a527/volumes"
Oct 30 19:15:17.550419 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:17.550394    4525 scope.go:122] "RemoveContainer" containerID="b3372beff49735246e3b19c65a4831af6d48c86f5186b928df1fdaa9e598da35"
Oct 30 19:15:17.550663 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:17.550645    4525 scope.go:122] "RemoveContainer" containerID="2fc4e1ec07256c28e0f65ee312a3065750e3717bbaad7d98d9667e316a72d40d"
Oct 30 19:15:17.550820 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:17.550802    4525 scope.go:122] "RemoveContainer" containerID="d5f6d81b7c81c3ac2ba9689576c07628a08dc6d4793847073b2965051fd5c3c7"
Oct 30 19:15:17.551080 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:17.551063    4525 scope.go:122] "RemoveContainer" containerID="b9942eba4f8e91c993f48a29a5a8e2b3f364e68c1c7994b26e54fa1e71319d99"
Oct 30 19:15:17.551185 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:17.551167    4525 scope.go:122] "RemoveContainer" containerID="00cf906bb8c75aa2c30b8558a7196bfbcf98d1a86bedc810da20a0a10de8c885"
Oct 30 19:15:18.075968 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:18.075933    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c2e3f9-29889795-5rf8p" event={"ID":"4849dbc2-0043-46f1-ab5e-a76c16f85db9","Type":"ContainerStarted","Data":"993b8e6c04c57589b491dc069d7331dda58fe0fee6eec073625ae184c9f40d8b"}
Oct 30 19:15:18.537731 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:18.537618    4525 scope.go:122] "RemoveContainer" containerID="7e5f73f700cc6536dc8dd37121347a657f6b637ae42edbb6d8379380a72b0807"
Oct 30 19:15:18.537884 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:18.537848    4525 scope.go:122] "RemoveContainer" containerID="2eb84195c4aa34b40939ff2bfec53476ecda6819602f7c8b20fba705102b3892"
Oct 30 19:15:18.538032 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:18.538014    4525 scope.go:122] "RemoveContainer" containerID="4b6ea5a7fbea7ec6dce81f393e119cc99c56ca9793b41137e4cba649ecf4f9d4"
Oct 30 19:15:18.538192 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:18.538172    4525 scope.go:122] "RemoveContainer" containerID="feaf6beebdb8e6065553808706bb04d1c5767a184d2a939797301ad80eaaccc0"
Oct 30 19:15:18.605504 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:18.605476    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c2e3f9-29889795-5rf8p"]
Oct 30 19:15:19.138425 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:19.138395    4525 generic.go:356] "Generic (PLEG): container finished" podID="75f454fd-6c51-42f1-a19d-24d1c739a88c" containerID="a3d65c9942188c9c4bea2d69bd8488b954f47e5bb21b2310f2f509bd629aed77" exitCode=60
Oct 30 19:15:19.138930 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:19.138796    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6f956-29889795-vgkfh" event={"ID":"75f454fd-6c51-42f1-a19d-24d1c739a88c","Type":"ContainerDied","Data":"a3d65c9942188c9c4bea2d69bd8488b954f47e5bb21b2310f2f509bd629aed77"}
Oct 30 19:15:19.139051 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:19.138977    4525 scope.go:122] "RemoveContainer" containerID="d5f6d81b7c81c3ac2ba9689576c07628a08dc6d4793847073b2965051fd5c3c7"
Oct 30 19:15:19.140114 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:19.140093    4525 scope.go:122] "RemoveContainer" containerID="a3d65c9942188c9c4bea2d69bd8488b954f47e5bb21b2310f2f509bd629aed77"
Oct 30 19:15:19.140892 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:15:19.140833    4525 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-d6f956-29889795-vgkfh_odh-ai-gateway-infra(75f454fd-6c51-42f1-a19d-24d1c739a88c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6f956-29889795-vgkfh" podUID="75f454fd-6c51-42f1-a19d-24d1c739a88c"
Oct 30 19:15:19.145738 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:19.143752    4525 generic.go:356] "Generic (PLEG): container finished" podID="4849dbc2-0043-46f1-ab5e-a76c16f85db9" containerID="993b8e6c04c57589b491dc069d7331dda58fe0fee6eec073625ae184c9f40d8b" exitCode=60
Oct 30 19:15:19.145738 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:19.143916    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c2e3f9-29889795-5rf8p" event={"ID":"4849dbc2-0043-46f1-ab5e-a76c16f85db9","Type":"ContainerDied","Data":"993b8e6c04c57589b491dc069d7331dda58fe0fee6eec073625ae184c9f40d8b"}
Oct 30 19:15:19.147143 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:19.147120    4525 generic.go:356] "Generic (PLEG): container finished" podID="c67ac146-cadf-480c-b1a0-47e60b73d0e5" containerID="c42d919b14615b93f20a7bb046bc087510d6aeb0edfe2ac016b32934b7999aa4" exitCode=60
Oct 30 19:15:19.147244 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:19.147225    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f3d6a5-29889795-sfmsm" event={"ID":"c67ac146-cadf-480c-b1a0-47e60b73d0e5","Type":"ContainerDied","Data":"c42d919b14615b93f20a7bb046bc087510d6aeb0edfe2ac016b32934b7999aa4"}
Oct 30 19:15:19.147656 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:19.147637    4525 scope.go:122] "RemoveContainer" containerID="c42d919b14615b93f20a7bb046bc087510d6aeb0edfe2ac016b32934b7999aa4"
Oct 30 19:15:19.148348 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:15:19.148241    4525 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-f3d6a5-29889795-sfmsm_odh-ai-gateway-infra(c67ac146-cadf-480c-b1a0-47e60b73d0e5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f3d6a5-29889795-sfmsm" podUID="c67ac146-cadf-480c-b1a0-47e60b73d0e5"
Oct 30 19:15:19.150864 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:19.150843    4525 generic.go:356] "Generic (PLEG): container finished" podID="01d8e335-0163-44d7-b811-fb9b99cd285a" containerID="e5892e16af907e018d3182d0070ab093d9393d6bb952e3fbd502c922f26d0c0a" exitCode=60
Oct 30 19:15:19.150979 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:19.150958    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39bde3-29889795-rzh6s" event={"ID":"01d8e335-0163-44d7-b811-fb9b99cd285a","Type":"ContainerDied","Data":"e5892e16af907e018d3182d0070ab093d9393d6bb952e3fbd502c922f26d0c0a"}
Oct 30 19:15:19.152898 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:19.152728    4525 scope.go:122] "RemoveContainer" containerID="e5892e16af907e018d3182d0070ab093d9393d6bb952e3fbd502c922f26d0c0a"
Oct 30 19:15:19.153295 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:15:19.153269    4525 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-39bde3-29889795-rzh6s_odh-ai-gateway-infra(01d8e335-0163-44d7-b811-fb9b99cd285a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39bde3-29889795-rzh6s" podUID="01d8e335-0163-44d7-b811-fb9b99cd285a"
Oct 30 19:15:19.153630 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:19.153614    4525 generic.go:356] "Generic (PLEG): container finished" podID="3253644f-14ff-452e-851b-694d8724574f" containerID="db2ce61338c1a1eded6961a0a0bba46a899de76f57513e2e54869927664e716f" exitCode=60
Oct 30 19:15:19.153713 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:19.153646    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-85cb41-29889795-9wpqw" event={"ID":"3253644f-14ff-452e-851b-694d8724574f","Type":"ContainerDied","Data":"db2ce61338c1a1eded6961a0a0bba46a899de76f57513e2e54869927664e716f"}
Oct 30 19:15:19.154067 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:19.154050    4525 scope.go:122] "RemoveContainer" containerID="db2ce61338c1a1eded6961a0a0bba46a899de76f57513e2e54869927664e716f"
Oct 30 19:15:19.155008 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:15:19.154505    4525 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-85cb41-29889795-9wpqw_odh-ai-gateway-infra(3253644f-14ff-452e-851b-694d8724574f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-85cb41-29889795-9wpqw" podUID="3253644f-14ff-452e-851b-694d8724574f"
Oct 30 19:15:19.159897 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:19.159877    4525 scope.go:122] "RemoveContainer" containerID="00cf906bb8c75aa2c30b8558a7196bfbcf98d1a86bedc810da20a0a10de8c885"
Oct 30 19:15:19.179006 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:19.178914    4525 scope.go:122] "RemoveContainer" containerID="2fc4e1ec07256c28e0f65ee312a3065750e3717bbaad7d98d9667e316a72d40d"
Oct 30 19:15:19.221217 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:19.221104    4525 scope.go:122] "RemoveContainer" containerID="b9942eba4f8e91c993f48a29a5a8e2b3f364e68c1c7994b26e54fa1e71319d99"
Oct 30 19:15:19.247623 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:19.247600    4525 scope.go:122] "RemoveContainer" containerID="b3372beff49735246e3b19c65a4831af6d48c86f5186b928df1fdaa9e598da35"
Oct 30 19:15:19.387190 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:19.387168    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c2e3f9-29889795-5rf8p"
Oct 30 19:15:19.428946 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:19.428918    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4849dbc2-0043-46f1-ab5e-a76c16f85db9-kube-api-access-d89pk\" (UniqueName: \"kubernetes.io/projected/4849dbc2-0043-46f1-ab5e-a76c16f85db9-kube-api-access-d89pk\") pod \"4849dbc2-0043-46f1-ab5e-a76c16f85db9\" (UID: \"4849dbc2-0043-46f1-ab5e-a76c16f85db9\") "
Oct 30 19:15:19.429139 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:19.429006    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4849dbc2-0043-46f1-ab5e-a76c16f85db9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4849dbc2-0043-46f1-ab5e-a76c16f85db9-maas-api-service-ca\") pod \"4849dbc2-0043-46f1-ab5e-a76c16f85db9\" (UID: \"4849dbc2-0043-46f1-ab5e-a76c16f85db9\") "
Oct 30 19:15:19.429426 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:19.429402    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4849dbc2-0043-46f1-ab5e-a76c16f85db9-maas-api-service-ca" pod "4849dbc2-0043-46f1-ab5e-a76c16f85db9" (UID: "4849dbc2-0043-46f1-ab5e-a76c16f85db9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:19.429949 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:19.429928    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4849dbc2-0043-46f1-ab5e-a76c16f85db9-maas-api-service-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:19.476651 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:19.476619    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4849dbc2-0043-46f1-ab5e-a76c16f85db9-kube-api-access-d89pk" pod "4849dbc2-0043-46f1-ab5e-a76c16f85db9" (UID: "4849dbc2-0043-46f1-ab5e-a76c16f85db9"). InnerVolumeSpecName "kube-api-access-d89pk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:19.531832 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:19.531802    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d89pk\" (UniqueName: \"kubernetes.io/projected/4849dbc2-0043-46f1-ab5e-a76c16f85db9-kube-api-access-d89pk\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:19.549097 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:19.549071    4525 scope.go:122] "RemoveContainer" containerID="a761973cdcd815f36320a0d6d12e727b7c0c9fb85c8400f753926bb337f7e4e7"
Oct 30 19:15:19.549276 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:19.549254    4525 scope.go:122] "RemoveContainer" containerID="e4f00fabeb42c763cd05b704e39c68c4d896c14f5b0e0dfd0a847ccea5610a3b"
Oct 30 19:15:19.549999 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:19.549575    4525 scope.go:122] "RemoveContainer" containerID="c8c24555076ec9b77fba843829b23ccbfde748b78f94fd62b04286465bb42423"
Oct 30 19:15:19.549999 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:19.549693    4525 scope.go:122] "RemoveContainer" containerID="d5f34ddd28f492babd029a602d15f223482a69a95b92f58455cfd81508faad20"
Oct 30 19:15:20.173015 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:20.172986    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6f956-29889795-vgkfh"]
Oct 30 19:15:20.173479 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:20.173455    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dcd326-29889795-zj7rm" event={"ID":"dfff552f-d7bb-414b-9a43-34922f58d2bd","Type":"ContainerDied","Data":"1dfe7079ffb0e89ed989e66b35646feae6b4f2d34408ee276b16de581de82239"}
Oct 30 19:15:20.173594 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:20.173584    4525 scope.go:122] "RemoveContainer" containerID="feaf6beebdb8e6065553808706bb04d1c5767a184d2a939797301ad80eaaccc0"
Oct 30 19:15:20.173726 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:20.173038    4525 generic.go:356] "Generic (PLEG): container finished" podID="dfff552f-d7bb-414b-9a43-34922f58d2bd" containerID="1dfe7079ffb0e89ed989e66b35646feae6b4f2d34408ee276b16de581de82239" exitCode=60
Oct 30 19:15:20.174228 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:20.174213    4525 scope.go:122] "RemoveContainer" containerID="1dfe7079ffb0e89ed989e66b35646feae6b4f2d34408ee276b16de581de82239"
Oct 30 19:15:20.174640 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:15:20.174623    4525 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-dcd326-29889795-zj7rm_odh-ai-gateway-infra(dfff552f-d7bb-414b-9a43-34922f58d2bd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dcd326-29889795-zj7rm" podUID="dfff552f-d7bb-414b-9a43-34922f58d2bd"
Oct 30 19:15:20.182446 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:20.181206    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c2e3f9-29889795-5rf8p" event={"ID":"4849dbc2-0043-46f1-ab5e-a76c16f85db9","Type":"ContainerDied","Data":"170f54d843621af69506bce3f068997bd4163bb22dfb2bfee334323dd26b200d"}
Oct 30 19:15:20.182446 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:20.181325    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c2e3f9-29889795-5rf8p"
Oct 30 19:15:20.194517 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:20.194478    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39bde3-29889795-rzh6s"]
Oct 30 19:15:20.195758 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:20.194942    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ef6be7-29889795-pf2p2" event={"ID":"e8fa12c9-9b1d-4033-aebb-eabcd9340222","Type":"ContainerStarted","Data":"6564aa36056e3bf97906ae42625916bc3c85107cd696a2e6bf1c5c3533bb47cb"}
Oct 30 19:15:20.219085 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:20.219053    4525 generic.go:356] "Generic (PLEG): container finished" podID="bfeff9d8-6996-4f87-a933-b41f97627004" containerID="9b1d63191a3a6857923331f5c32d8827e37d0cadc77b3dfd281d62e8119a672d" exitCode=60
Oct 30 19:15:20.219224 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:20.219176    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-68b466-29889795-fgfjm" event={"ID":"bfeff9d8-6996-4f87-a933-b41f97627004","Type":"ContainerDied","Data":"9b1d63191a3a6857923331f5c32d8827e37d0cadc77b3dfd281d62e8119a672d"}
Oct 30 19:15:20.220513 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:20.220489    4525 scope.go:122] "RemoveContainer" containerID="9b1d63191a3a6857923331f5c32d8827e37d0cadc77b3dfd281d62e8119a672d"
Oct 30 19:15:20.221115 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:15:20.221083    4525 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-68b466-29889795-fgfjm_odh-ai-gateway-infra(bfeff9d8-6996-4f87-a933-b41f97627004)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-68b466-29889795-fgfjm" podUID="bfeff9d8-6996-4f87-a933-b41f97627004"
Oct 30 19:15:20.221945 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:20.221916    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ed012e-29889795-f5xmz" event={"ID":"d3aa2e02-e67a-4418-8fb7-3d805cb7ca7c","Type":"ContainerStarted","Data":"6a2f621edf85a6c5a49b5d844078fc035bdf29dd61fea58b46d37425fec3b34a"}
Oct 30 19:15:20.285763 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:20.264136    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f3d6a5-29889795-sfmsm"]
Oct 30 19:15:20.324740 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:20.324597    4525 scope.go:122] "RemoveContainer" containerID="993b8e6c04c57589b491dc069d7331dda58fe0fee6eec073625ae184c9f40d8b"
Oct 30 19:15:20.350061 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:20.346975    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-85cb41-29889795-9wpqw"]
Oct 30 19:15:20.374292 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:20.374270    4525 scope.go:122] "RemoveContainer" containerID="7e5f73f700cc6536dc8dd37121347a657f6b637ae42edbb6d8379380a72b0807"
Oct 30 19:15:20.425337 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:20.422436    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c2e3f9-29889795-5rf8p"]
Oct 30 19:15:20.428288 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:20.428235    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c2e3f9-29889795-5rf8p"]
Oct 30 19:15:20.539413 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:20.537049    4525 scope.go:122] "RemoveContainer" containerID="eca90751c5fcfd6ad9dd0ad5ef8f4ea898e4a9f3fbfc5cfa8218dc1a388ca81f"
Oct 30 19:15:20.669471 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:20.669442    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39bde3-29889795-rzh6s"
Oct 30 19:15:20.778333 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:20.775225    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01d8e335-0163-44d7-b811-fb9b99cd285a-kube-api-access-d4xp4\" (UniqueName: \"kubernetes.io/projected/01d8e335-0163-44d7-b811-fb9b99cd285a-kube-api-access-d4xp4\") pod \"01d8e335-0163-44d7-b811-fb9b99cd285a\" (UID: \"01d8e335-0163-44d7-b811-fb9b99cd285a\") "
Oct 30 19:15:20.778333 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:20.775539    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01d8e335-0163-44d7-b811-fb9b99cd285a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01d8e335-0163-44d7-b811-fb9b99cd285a-maas-api-service-ca\") pod \"01d8e335-0163-44d7-b811-fb9b99cd285a\" (UID: \"01d8e335-0163-44d7-b811-fb9b99cd285a\") "
Oct 30 19:15:20.779706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:20.779174    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01d8e335-0163-44d7-b811-fb9b99cd285a-maas-api-service-ca" pod "01d8e335-0163-44d7-b811-fb9b99cd285a" (UID: "01d8e335-0163-44d7-b811-fb9b99cd285a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:20.780136 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:20.780106    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01d8e335-0163-44d7-b811-fb9b99cd285a-maas-api-service-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:20.802087 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:20.802059    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01d8e335-0163-44d7-b811-fb9b99cd285a-kube-api-access-d4xp4" pod "01d8e335-0163-44d7-b811-fb9b99cd285a" (UID: "01d8e335-0163-44d7-b811-fb9b99cd285a"). InnerVolumeSpecName "kube-api-access-d4xp4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:20.855810 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:20.855786    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f3d6a5-29889795-sfmsm"
Oct 30 19:15:20.857859 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:20.857835    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6f956-29889795-vgkfh"
Oct 30 19:15:20.861748 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:20.861730    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-85cb41-29889795-9wpqw"
Oct 30 19:15:20.866066 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:20.866049    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm"
Oct 30 19:15:20.883067 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:20.883033    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4xp4\" (UniqueName: \"kubernetes.io/projected/01d8e335-0163-44d7-b811-fb9b99cd285a-kube-api-access-d4xp4\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:20.985906 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:20.985871    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ccf27c7-f6d3-457b-9ff8-69b54f9b9287-kube-api-access-l7xvt\" (UniqueName: \"kubernetes.io/projected/6ccf27c7-f6d3-457b-9ff8-69b54f9b9287-kube-api-access-l7xvt\") pod \"6ccf27c7-f6d3-457b-9ff8-69b54f9b9287\" (UID: \"6ccf27c7-f6d3-457b-9ff8-69b54f9b9287\") "
Oct 30 19:15:20.986077 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:20.985942    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75f454fd-6c51-42f1-a19d-24d1c739a88c-kube-api-access-k2j8n\" (UniqueName: \"kubernetes.io/projected/75f454fd-6c51-42f1-a19d-24d1c739a88c-kube-api-access-k2j8n\") pod \"75f454fd-6c51-42f1-a19d-24d1c739a88c\" (UID: \"75f454fd-6c51-42f1-a19d-24d1c739a88c\") "
Oct 30 19:15:20.986077 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:20.986007    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75f454fd-6c51-42f1-a19d-24d1c739a88c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75f454fd-6c51-42f1-a19d-24d1c739a88c-maas-api-service-ca\") pod \"75f454fd-6c51-42f1-a19d-24d1c739a88c\" (UID: \"75f454fd-6c51-42f1-a19d-24d1c739a88c\") "
Oct 30 19:15:20.986523 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:20.986496    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ccf27c7-f6d3-457b-9ff8-69b54f9b9287-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ccf27c7-f6d3-457b-9ff8-69b54f9b9287-workload-socket\") pod \"6ccf27c7-f6d3-457b-9ff8-69b54f9b9287\" (UID: \"6ccf27c7-f6d3-457b-9ff8-69b54f9b9287\") "
Oct 30 19:15:20.986735 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:20.986553    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ccf27c7-f6d3-457b-9ff8-69b54f9b9287-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ccf27c7-f6d3-457b-9ff8-69b54f9b9287-istiod-ca-cert\") pod \"6ccf27c7-f6d3-457b-9ff8-69b54f9b9287\" (UID: \"6ccf27c7-f6d3-457b-9ff8-69b54f9b9287\") "
Oct 30 19:15:20.986735 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:20.986596    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3253644f-14ff-452e-851b-694d8724574f-kube-api-access-x4486\" (UniqueName: \"kubernetes.io/projected/3253644f-14ff-452e-851b-694d8724574f-kube-api-access-x4486\") pod \"3253644f-14ff-452e-851b-694d8724574f\" (UID: \"3253644f-14ff-452e-851b-694d8724574f\") "
Oct 30 19:15:20.986735 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:20.986639    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3253644f-14ff-452e-851b-694d8724574f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3253644f-14ff-452e-851b-694d8724574f-maas-api-service-ca\") pod \"3253644f-14ff-452e-851b-694d8724574f\" (UID: \"3253644f-14ff-452e-851b-694d8724574f\") "
Oct 30 19:15:20.986945 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:20.986770    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ccf27c7-f6d3-457b-9ff8-69b54f9b9287-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ccf27c7-f6d3-457b-9ff8-69b54f9b9287-credential-socket\") pod \"6ccf27c7-f6d3-457b-9ff8-69b54f9b9287\" (UID: \"6ccf27c7-f6d3-457b-9ff8-69b54f9b9287\") "
Oct 30 19:15:20.986945 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:20.986808    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ccf27c7-f6d3-457b-9ff8-69b54f9b9287-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ccf27c7-f6d3-457b-9ff8-69b54f9b9287-workload-certs\") pod \"6ccf27c7-f6d3-457b-9ff8-69b54f9b9287\" (UID: \"6ccf27c7-f6d3-457b-9ff8-69b54f9b9287\") "
Oct 30 19:15:20.986945 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:20.986905    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6ccf27c7-f6d3-457b-9ff8-69b54f9b9287-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ccf27c7-f6d3-457b-9ff8-69b54f9b9287-istio-podinfo\") pod \"6ccf27c7-f6d3-457b-9ff8-69b54f9b9287\" (UID: \"6ccf27c7-f6d3-457b-9ff8-69b54f9b9287\") "
Oct 30 19:15:20.987098 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:20.986954    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c67ac146-cadf-480c-b1a0-47e60b73d0e5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c67ac146-cadf-480c-b1a0-47e60b73d0e5-maas-api-service-ca\") pod \"c67ac146-cadf-480c-b1a0-47e60b73d0e5\" (UID: \"c67ac146-cadf-480c-b1a0-47e60b73d0e5\") "
Oct 30 19:15:20.987098 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:20.987042    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ccf27c7-f6d3-457b-9ff8-69b54f9b9287-istio-token\" (UniqueName: \"kubernetes.io/projected/6ccf27c7-f6d3-457b-9ff8-69b54f9b9287-istio-token\") pod \"6ccf27c7-f6d3-457b-9ff8-69b54f9b9287\" (UID: \"6ccf27c7-f6d3-457b-9ff8-69b54f9b9287\") "
Oct 30 19:15:20.987098 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:20.987046    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75f454fd-6c51-42f1-a19d-24d1c739a88c-maas-api-service-ca" pod "75f454fd-6c51-42f1-a19d-24d1c739a88c" (UID: "75f454fd-6c51-42f1-a19d-24d1c739a88c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:20.987248 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:20.987148    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3253644f-14ff-452e-851b-694d8724574f-maas-api-service-ca" pod "3253644f-14ff-452e-851b-694d8724574f" (UID: "3253644f-14ff-452e-851b-694d8724574f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:20.987248 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:20.987176    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ccf27c7-f6d3-457b-9ff8-69b54f9b9287-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ccf27c7-f6d3-457b-9ff8-69b54f9b9287-istio-data\") pod \"6ccf27c7-f6d3-457b-9ff8-69b54f9b9287\" (UID: \"6ccf27c7-f6d3-457b-9ff8-69b54f9b9287\") "
Oct 30 19:15:20.987359 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:20.987246    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c67ac146-cadf-480c-b1a0-47e60b73d0e5-kube-api-access-gsv5c\" (UniqueName: \"kubernetes.io/projected/c67ac146-cadf-480c-b1a0-47e60b73d0e5-kube-api-access-gsv5c\") pod \"c67ac146-cadf-480c-b1a0-47e60b73d0e5\" (UID: \"c67ac146-cadf-480c-b1a0-47e60b73d0e5\") "
Oct 30 19:15:20.987359 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:20.987283    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ccf27c7-f6d3-457b-9ff8-69b54f9b9287-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ccf27c7-f6d3-457b-9ff8-69b54f9b9287-istio-envoy\") pod \"6ccf27c7-f6d3-457b-9ff8-69b54f9b9287\" (UID: \"6ccf27c7-f6d3-457b-9ff8-69b54f9b9287\") "
Oct 30 19:15:20.987455 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:20.987344    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ccf27c7-f6d3-457b-9ff8-69b54f9b9287-workload-certs" pod "6ccf27c7-f6d3-457b-9ff8-69b54f9b9287" (UID: "6ccf27c7-f6d3-457b-9ff8-69b54f9b9287"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:20.987455 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:20.987412    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ccf27c7-f6d3-457b-9ff8-69b54f9b9287-istiod-ca-cert" pod "6ccf27c7-f6d3-457b-9ff8-69b54f9b9287" (UID: "6ccf27c7-f6d3-457b-9ff8-69b54f9b9287"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:20.988218 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:20.987798    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ccf27c7-f6d3-457b-9ff8-69b54f9b9287-credential-socket" pod "6ccf27c7-f6d3-457b-9ff8-69b54f9b9287" (UID: "6ccf27c7-f6d3-457b-9ff8-69b54f9b9287"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:20.988218 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:20.987753    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ccf27c7-f6d3-457b-9ff8-69b54f9b9287-workload-socket" pod "6ccf27c7-f6d3-457b-9ff8-69b54f9b9287" (UID: "6ccf27c7-f6d3-457b-9ff8-69b54f9b9287"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:20.988218 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:20.987833    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ccf27c7-f6d3-457b-9ff8-69b54f9b9287-istio-data" pod "6ccf27c7-f6d3-457b-9ff8-69b54f9b9287" (UID: "6ccf27c7-f6d3-457b-9ff8-69b54f9b9287"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:20.988218 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:20.988161    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c67ac146-cadf-480c-b1a0-47e60b73d0e5-maas-api-service-ca" pod "c67ac146-cadf-480c-b1a0-47e60b73d0e5" (UID: "c67ac146-cadf-480c-b1a0-47e60b73d0e5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:20.988983 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:20.988954    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c67ac146-cadf-480c-b1a0-47e60b73d0e5-maas-api-service-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:20.989086 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:20.988989    4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ccf27c7-f6d3-457b-9ff8-69b54f9b9287-istio-data\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:20.989086 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:20.989005    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75f454fd-6c51-42f1-a19d-24d1c739a88c-maas-api-service-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:20.989086 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:20.989019    4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ccf27c7-f6d3-457b-9ff8-69b54f9b9287-workload-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:20.989086 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:20.989034    4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ccf27c7-f6d3-457b-9ff8-69b54f9b9287-istiod-ca-cert\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:20.989086 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:20.989047    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3253644f-14ff-452e-851b-694d8724574f-maas-api-service-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:20.989086 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:20.989059    4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ccf27c7-f6d3-457b-9ff8-69b54f9b9287-credential-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:20.989086 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:20.989072    4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ccf27c7-f6d3-457b-9ff8-69b54f9b9287-workload-certs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:20.989672 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:20.989642    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3253644f-14ff-452e-851b-694d8724574f-kube-api-access-x4486" pod "3253644f-14ff-452e-851b-694d8724574f" (UID: "3253644f-14ff-452e-851b-694d8724574f"). InnerVolumeSpecName "kube-api-access-x4486". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:20.990799 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:20.990749    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75f454fd-6c51-42f1-a19d-24d1c739a88c-kube-api-access-k2j8n" pod "75f454fd-6c51-42f1-a19d-24d1c739a88c" (UID: "75f454fd-6c51-42f1-a19d-24d1c739a88c"). InnerVolumeSpecName "kube-api-access-k2j8n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:20.991045 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:20.990999    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ccf27c7-f6d3-457b-9ff8-69b54f9b9287-kube-api-access-l7xvt" pod "6ccf27c7-f6d3-457b-9ff8-69b54f9b9287" (UID: "6ccf27c7-f6d3-457b-9ff8-69b54f9b9287"). InnerVolumeSpecName "kube-api-access-l7xvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:20.991912 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:20.991885    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ccf27c7-f6d3-457b-9ff8-69b54f9b9287-istio-envoy" pod "6ccf27c7-f6d3-457b-9ff8-69b54f9b9287" (UID: "6ccf27c7-f6d3-457b-9ff8-69b54f9b9287"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:20.992116 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:20.992086    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c67ac146-cadf-480c-b1a0-47e60b73d0e5-kube-api-access-gsv5c" pod "c67ac146-cadf-480c-b1a0-47e60b73d0e5" (UID: "c67ac146-cadf-480c-b1a0-47e60b73d0e5"). InnerVolumeSpecName "kube-api-access-gsv5c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:20.993399 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:20.993366    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ccf27c7-f6d3-457b-9ff8-69b54f9b9287-istio-token" pod "6ccf27c7-f6d3-457b-9ff8-69b54f9b9287" (UID: "6ccf27c7-f6d3-457b-9ff8-69b54f9b9287"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:20.993492 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:20.993420    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6ccf27c7-f6d3-457b-9ff8-69b54f9b9287-istio-podinfo" pod "6ccf27c7-f6d3-457b-9ff8-69b54f9b9287" (UID: "6ccf27c7-f6d3-457b-9ff8-69b54f9b9287"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:15:21.090696 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.090645    4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ccf27c7-f6d3-457b-9ff8-69b54f9b9287-istio-podinfo\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:21.090803 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.090710    4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ccf27c7-f6d3-457b-9ff8-69b54f9b9287-istio-token\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:21.090803 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.090727    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsv5c\" (UniqueName: \"kubernetes.io/projected/c67ac146-cadf-480c-b1a0-47e60b73d0e5-kube-api-access-gsv5c\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:21.090803 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.090741    4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ccf27c7-f6d3-457b-9ff8-69b54f9b9287-istio-envoy\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:21.090803 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.090754    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7xvt\" (UniqueName: \"kubernetes.io/projected/6ccf27c7-f6d3-457b-9ff8-69b54f9b9287-kube-api-access-l7xvt\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:21.090803 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.090769    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2j8n\" (UniqueName: \"kubernetes.io/projected/75f454fd-6c51-42f1-a19d-24d1c739a88c-kube-api-access-k2j8n\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:21.090803 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.090781    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4486\" (UniqueName: \"kubernetes.io/projected/3253644f-14ff-452e-851b-694d8724574f-kube-api-access-x4486\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:21.212898 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.212868    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dcd326-29889795-zj7rm"]
Oct 30 19:15:21.235797 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.235750    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6f956-29889795-vgkfh" event={"ID":"75f454fd-6c51-42f1-a19d-24d1c739a88c","Type":"ContainerDied","Data":"170c71440778bac390ea7f4352f77ee39ad60b6c0cff1a980346b5665fa6445f"}
Oct 30 19:15:21.235960 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.235823    4525 scope.go:122] "RemoveContainer" containerID="a3d65c9942188c9c4bea2d69bd8488b954f47e5bb21b2310f2f509bd629aed77"
Oct 30 19:15:21.235960 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.235772    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6f956-29889795-vgkfh"
Oct 30 19:15:21.244368 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.244344    4525 generic.go:356] "Generic (PLEG): container finished" podID="90e98322-b727-4a35-9d93-808e0b01d483" containerID="56f03a114cb5b7d194dd0e964a1c4aa77ad4f048a182ba64f291aee9b9705bc2" exitCode=60
Oct 30 19:15:21.244500 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.244412    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a1d55d-29889795-bw7v2" event={"ID":"90e98322-b727-4a35-9d93-808e0b01d483","Type":"ContainerDied","Data":"56f03a114cb5b7d194dd0e964a1c4aa77ad4f048a182ba64f291aee9b9705bc2"}
Oct 30 19:15:21.244926 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.244901    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ef6be7-29889795-pf2p2"]
Oct 30 19:15:21.245017 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.244984    4525 scope.go:122] "RemoveContainer" containerID="56f03a114cb5b7d194dd0e964a1c4aa77ad4f048a182ba64f291aee9b9705bc2"
Oct 30 19:15:21.245388 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:15:21.245364    4525 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-a1d55d-29889795-bw7v2_odh-ai-gateway-infra(90e98322-b727-4a35-9d93-808e0b01d483)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a1d55d-29889795-bw7v2" podUID="90e98322-b727-4a35-9d93-808e0b01d483"
Oct 30 19:15:21.246124 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.246096    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f3d6a5-29889795-sfmsm" event={"ID":"c67ac146-cadf-480c-b1a0-47e60b73d0e5","Type":"ContainerDied","Data":"a2a4ded5502a00b60df717c0018331aabd836e61e8a49c111c9ca6bc19f1dadc"}
Oct 30 19:15:21.246231 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.246110    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f3d6a5-29889795-sfmsm"
Oct 30 19:15:21.248530 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.248507    4525 generic.go:356] "Generic (PLEG): container finished" podID="e8fa12c9-9b1d-4033-aebb-eabcd9340222" containerID="6564aa36056e3bf97906ae42625916bc3c85107cd696a2e6bf1c5c3533bb47cb" exitCode=60
Oct 30 19:15:21.248792 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.248769    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ef6be7-29889795-pf2p2" event={"ID":"e8fa12c9-9b1d-4033-aebb-eabcd9340222","Type":"ContainerDied","Data":"6564aa36056e3bf97906ae42625916bc3c85107cd696a2e6bf1c5c3533bb47cb"}
Oct 30 19:15:21.252492 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.252466    4525 generic.go:356] "Generic (PLEG): container finished" podID="e506709f-838a-4255-b3fb-498da9fb0697" containerID="1aeaae8bc8126bbf16736be43accdfb188a626d8ece3315273da050766cd86e4" exitCode=60
Oct 30 19:15:21.252592 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.252560    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9de872-29889795-c7whh" event={"ID":"e506709f-838a-4255-b3fb-498da9fb0697","Type":"ContainerDied","Data":"1aeaae8bc8126bbf16736be43accdfb188a626d8ece3315273da050766cd86e4"}
Oct 30 19:15:21.253093 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.253026    4525 scope.go:122] "RemoveContainer" containerID="1aeaae8bc8126bbf16736be43accdfb188a626d8ece3315273da050766cd86e4"
Oct 30 19:15:21.253408 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:15:21.253382    4525 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-9de872-29889795-c7whh_odh-ai-gateway-infra(e506709f-838a-4255-b3fb-498da9fb0697)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9de872-29889795-c7whh" podUID="e506709f-838a-4255-b3fb-498da9fb0697"
Oct 30 19:15:21.254732 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.254470    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39bde3-29889795-rzh6s"
Oct 30 19:15:21.254732 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.254476    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39bde3-29889795-rzh6s" event={"ID":"01d8e335-0163-44d7-b811-fb9b99cd285a","Type":"ContainerDied","Data":"5ee5eb6a1b3ef452b0281d4c097c2430af88ebedd827ac4fbed52dc77b70b7b6"}
Oct 30 19:15:21.257810 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.257294    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-85cb41-29889795-9wpqw" event={"ID":"3253644f-14ff-452e-851b-694d8724574f","Type":"ContainerDied","Data":"8f6676411f7238d87495e3e4532ad49b28f60d29923269a05bde9c9defb19892"}
Oct 30 19:15:21.257810 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.257397    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-85cb41-29889795-9wpqw"
Oct 30 19:15:21.269460 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.269251    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ed012e-29889795-f5xmz"]
Oct 30 19:15:21.271080 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.270787    4525 generic.go:356] "Generic (PLEG): container finished" podID="403406c5-302f-4563-be3b-63ac8959d517" containerID="f9e32a2808a4f22c8102aaec61db4a750f30c2d4ab5b20a279534913ad60f4b8" exitCode=60
Oct 30 19:15:21.271080 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.270908    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2d1c71-29889795-p7txw" event={"ID":"403406c5-302f-4563-be3b-63ac8959d517","Type":"ContainerDied","Data":"f9e32a2808a4f22c8102aaec61db4a750f30c2d4ab5b20a279534913ad60f4b8"}
Oct 30 19:15:21.272620 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.272193    4525 scope.go:122] "RemoveContainer" containerID="f9e32a2808a4f22c8102aaec61db4a750f30c2d4ab5b20a279534913ad60f4b8"
Oct 30 19:15:21.272620 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:15:21.272572    4525 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-2d1c71-29889795-p7txw_odh-ai-gateway-infra(403406c5-302f-4563-be3b-63ac8959d517)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2d1c71-29889795-p7txw" podUID="403406c5-302f-4563-be3b-63ac8959d517"
Oct 30 19:15:21.276723 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.275941    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-ef6b56-29889795-nl7wc" event={"ID":"8d3ff890-4ac4-4395-b309-3483be728b66","Type":"ContainerStarted","Data":"28e04894d6d40bc263aa74f435a69fd6886115c0ebeac6425bc9f4cce1659983"}
Oct 30 19:15:21.285815 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.284319    4525 generic.go:356] "Generic (PLEG): container finished" podID="d3aa2e02-e67a-4418-8fb7-3d805cb7ca7c" containerID="6a2f621edf85a6c5a49b5d844078fc035bdf29dd61fea58b46d37425fec3b34a" exitCode=60
Oct 30 19:15:21.285815 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.284439    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ed012e-29889795-f5xmz" event={"ID":"d3aa2e02-e67a-4418-8fb7-3d805cb7ca7c","Type":"ContainerDied","Data":"6a2f621edf85a6c5a49b5d844078fc035bdf29dd61fea58b46d37425fec3b34a"}
Oct 30 19:15:21.292376 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.292350    4525 generic.go:356] "Generic (PLEG): container finished" podID="e5149d31-2979-4353-9b41-df84f90cffbd" containerID="50dae2a0619d5ea06ef3d80104186a43595bb131ec10c864da3e7d3394a2691a" exitCode=60
Oct 30 19:15:21.292576 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.292429    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-46385c-29889795-hbvv9" event={"ID":"e5149d31-2979-4353-9b41-df84f90cffbd","Type":"ContainerDied","Data":"50dae2a0619d5ea06ef3d80104186a43595bb131ec10c864da3e7d3394a2691a"}
Oct 30 19:15:21.294450 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.294431    4525 scope.go:122] "RemoveContainer" containerID="50dae2a0619d5ea06ef3d80104186a43595bb131ec10c864da3e7d3394a2691a"
Oct 30 19:15:21.295145 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.294973    4525 generic.go:356] "Generic (PLEG): container finished" podID="6ccf27c7-f6d3-457b-9ff8-69b54f9b9287" containerID="ebc7f76d96aafbd104dfa29a040cffffaf1790bdc6806aa0827284f5fe573d09" exitCode=0
Oct 30 19:15:21.295248 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.294993    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm" event={"ID":"6ccf27c7-f6d3-457b-9ff8-69b54f9b9287","Type":"ContainerDied","Data":"ebc7f76d96aafbd104dfa29a040cffffaf1790bdc6806aa0827284f5fe573d09"}
Oct 30 19:15:21.295248 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.295195    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm" event={"ID":"6ccf27c7-f6d3-457b-9ff8-69b54f9b9287","Type":"ContainerDied","Data":"1d1aa37a8a7180af40c5bf727f24d6847461386ca170b40f1269f527fbdcebac"}
Oct 30 19:15:21.295248 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.295054    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm"
Oct 30 19:15:21.295776 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:15:21.295752    4525 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-46385c-29889795-hbvv9_odh-ai-gateway-infra(e5149d31-2979-4353-9b41-df84f90cffbd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-46385c-29889795-hbvv9" podUID="e5149d31-2979-4353-9b41-df84f90cffbd"
Oct 30 19:15:21.363613 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.359444    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-68b466-29889795-fgfjm"]
Oct 30 19:15:21.378039 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.377128    4525 scope.go:122] "RemoveContainer" containerID="d5f34ddd28f492babd029a602d15f223482a69a95b92f58455cfd81508faad20"
Oct 30 19:15:21.434168 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.434076    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6f956-29889795-vgkfh"]
Oct 30 19:15:21.446518 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.446491    4525 scope.go:122] "RemoveContainer" containerID="c42d919b14615b93f20a7bb046bc087510d6aeb0edfe2ac016b32934b7999aa4"
Oct 30 19:15:21.449974 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.449631    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6f956-29889795-vgkfh"]
Oct 30 19:15:21.483552 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.473737    4525 scope.go:122] "RemoveContainer" containerID="2eb84195c4aa34b40939ff2bfec53476ecda6819602f7c8b20fba705102b3892"
Oct 30 19:15:21.488217 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.488179    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm"]
Oct 30 19:15:21.498506 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.498479    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-baa504-openshift-default-6944b9c4d5-qlfsm"]
Oct 30 19:15:21.509836 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.508899    4525 scope.go:122] "RemoveContainer" containerID="a761973cdcd815f36320a0d6d12e727b7c0c9fb85c8400f753926bb337f7e4e7"
Oct 30 19:15:21.516395 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.516370    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39bde3-29889795-rzh6s"]
Oct 30 19:15:21.517245 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.517223    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dcd326-29889795-zj7rm"
Oct 30 19:15:21.524732 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.524706    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-39bde3-29889795-rzh6s"]
Oct 30 19:15:21.561874 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.553824    4525 scope.go:122] "RemoveContainer" containerID="e5892e16af907e018d3182d0070ab093d9393d6bb952e3fbd502c922f26d0c0a"
Oct 30 19:15:21.575087 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.574541    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01d8e335-0163-44d7-b811-fb9b99cd285a" path="/var/lib/kubelet/pods/01d8e335-0163-44d7-b811-fb9b99cd285a/volumes"
Oct 30 19:15:21.576219 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.576173    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4849dbc2-0043-46f1-ab5e-a76c16f85db9" path="/var/lib/kubelet/pods/4849dbc2-0043-46f1-ab5e-a76c16f85db9/volumes"
Oct 30 19:15:21.577659 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.577636    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ccf27c7-f6d3-457b-9ff8-69b54f9b9287" path="/var/lib/kubelet/pods/6ccf27c7-f6d3-457b-9ff8-69b54f9b9287/volumes"
Oct 30 19:15:21.580343 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.580320    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75f454fd-6c51-42f1-a19d-24d1c739a88c" path="/var/lib/kubelet/pods/75f454fd-6c51-42f1-a19d-24d1c739a88c/volumes"
Oct 30 19:15:21.613880 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.613847    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfff552f-d7bb-414b-9a43-34922f58d2bd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfff552f-d7bb-414b-9a43-34922f58d2bd-maas-api-service-ca\") pod \"dfff552f-d7bb-414b-9a43-34922f58d2bd\" (UID: \"dfff552f-d7bb-414b-9a43-34922f58d2bd\") "
Oct 30 19:15:21.614085 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.614062    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfff552f-d7bb-414b-9a43-34922f58d2bd-kube-api-access-96f4m\" (UniqueName: \"kubernetes.io/projected/dfff552f-d7bb-414b-9a43-34922f58d2bd-kube-api-access-96f4m\") pod \"dfff552f-d7bb-414b-9a43-34922f58d2bd\" (UID: \"dfff552f-d7bb-414b-9a43-34922f58d2bd\") "
Oct 30 19:15:21.615491 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.615453    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfff552f-d7bb-414b-9a43-34922f58d2bd-maas-api-service-ca" pod "dfff552f-d7bb-414b-9a43-34922f58d2bd" (UID: "dfff552f-d7bb-414b-9a43-34922f58d2bd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:21.658740 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.658712    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-ef6b56-29889795-nl7wc"]
Oct 30 19:15:21.658915 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.658758    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f3d6a5-29889795-sfmsm"]
Oct 30 19:15:21.658915 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.658777    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f3d6a5-29889795-sfmsm"]
Oct 30 19:15:21.658915 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.658822    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-85cb41-29889795-9wpqw"]
Oct 30 19:15:21.658915 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.658859    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-85cb41-29889795-9wpqw"]
Oct 30 19:15:21.666078 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.666049    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ef6be7-29889795-pf2p2"
Oct 30 19:15:21.676507 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.676252    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfff552f-d7bb-414b-9a43-34922f58d2bd-kube-api-access-96f4m" pod "dfff552f-d7bb-414b-9a43-34922f58d2bd" (UID: "dfff552f-d7bb-414b-9a43-34922f58d2bd"). InnerVolumeSpecName "kube-api-access-96f4m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:21.686986 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.685996    4525 scope.go:122] "RemoveContainer" containerID="db2ce61338c1a1eded6961a0a0bba46a899de76f57513e2e54869927664e716f"
Oct 30 19:15:21.719322 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.719291    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96f4m\" (UniqueName: \"kubernetes.io/projected/dfff552f-d7bb-414b-9a43-34922f58d2bd-kube-api-access-96f4m\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:21.719456 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.719326    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfff552f-d7bb-414b-9a43-34922f58d2bd-maas-api-service-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:21.769187 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.768777    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-68b466-29889795-fgfjm"
Oct 30 19:15:21.769851 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.769147    4525 scope.go:122] "RemoveContainer" containerID="c8c24555076ec9b77fba843829b23ccbfde748b78f94fd62b04286465bb42423"
Oct 30 19:15:21.774081 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.774058    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ed012e-29889795-f5xmz"
Oct 30 19:15:21.797866 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.797840    4525 scope.go:122] "RemoveContainer" containerID="4b6ea5a7fbea7ec6dce81f393e119cc99c56ca9793b41137e4cba649ecf4f9d4"
Oct 30 19:15:21.815120 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.815099    4525 scope.go:122] "RemoveContainer" containerID="e4f00fabeb42c763cd05b704e39c68c4d896c14f5b0e0dfd0a847ccea5610a3b"
Oct 30 19:15:21.820352 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.820320    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8fa12c9-9b1d-4033-aebb-eabcd9340222-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8fa12c9-9b1d-4033-aebb-eabcd9340222-maas-api-service-ca\") pod \"e8fa12c9-9b1d-4033-aebb-eabcd9340222\" (UID: \"e8fa12c9-9b1d-4033-aebb-eabcd9340222\") "
Oct 30 19:15:21.820628 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.820576    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8fa12c9-9b1d-4033-aebb-eabcd9340222-kube-api-access-7shqk\" (UniqueName: \"kubernetes.io/projected/e8fa12c9-9b1d-4033-aebb-eabcd9340222-kube-api-access-7shqk\") pod \"e8fa12c9-9b1d-4033-aebb-eabcd9340222\" (UID: \"e8fa12c9-9b1d-4033-aebb-eabcd9340222\") "
Oct 30 19:15:21.822306 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.822282    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8fa12c9-9b1d-4033-aebb-eabcd9340222-maas-api-service-ca" pod "e8fa12c9-9b1d-4033-aebb-eabcd9340222" (UID: "e8fa12c9-9b1d-4033-aebb-eabcd9340222"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:21.836944 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.836914    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8fa12c9-9b1d-4033-aebb-eabcd9340222-kube-api-access-7shqk" pod "e8fa12c9-9b1d-4033-aebb-eabcd9340222" (UID: "e8fa12c9-9b1d-4033-aebb-eabcd9340222"). InnerVolumeSpecName "kube-api-access-7shqk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:21.838764 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.838743    4525 scope.go:122] "RemoveContainer" containerID="ebc7f76d96aafbd104dfa29a040cffffaf1790bdc6806aa0827284f5fe573d09"
Oct 30 19:15:21.851865 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.851840    4525 scope.go:122] "RemoveContainer" containerID="325128f3e14fb21cd06d956ada021baba82277fcd141493de35a31bccde645a0"
Oct 30 19:15:21.866561 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.866532    4525 scope.go:122] "RemoveContainer" containerID="ebc7f76d96aafbd104dfa29a040cffffaf1790bdc6806aa0827284f5fe573d09"
Oct 30 19:15:21.866939 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:15:21.866918    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebc7f76d96aafbd104dfa29a040cffffaf1790bdc6806aa0827284f5fe573d09\": container with ID starting with ebc7f76d96aafbd104dfa29a040cffffaf1790bdc6806aa0827284f5fe573d09 not found: ID does not exist" containerID="ebc7f76d96aafbd104dfa29a040cffffaf1790bdc6806aa0827284f5fe573d09"
Oct 30 19:15:21.867051 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.866950    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebc7f76d96aafbd104dfa29a040cffffaf1790bdc6806aa0827284f5fe573d09"} err="failed to get container status \"ebc7f76d96aafbd104dfa29a040cffffaf1790bdc6806aa0827284f5fe573d09\": rpc error: code = NotFound desc = could not find container \"ebc7f76d96aafbd104dfa29a040cffffaf1790bdc6806aa0827284f5fe573d09\": container with ID starting with ebc7f76d96aafbd104dfa29a040cffffaf1790bdc6806aa0827284f5fe573d09 not found: ID does not exist"
Oct 30 19:15:21.867051 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.866968    4525 scope.go:122] "RemoveContainer" containerID="325128f3e14fb21cd06d956ada021baba82277fcd141493de35a31bccde645a0"
Oct 30 19:15:21.867336 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:15:21.867303    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"325128f3e14fb21cd06d956ada021baba82277fcd141493de35a31bccde645a0\": container with ID starting with 325128f3e14fb21cd06d956ada021baba82277fcd141493de35a31bccde645a0 not found: ID does not exist" containerID="325128f3e14fb21cd06d956ada021baba82277fcd141493de35a31bccde645a0"
Oct 30 19:15:21.867499 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.867468    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"325128f3e14fb21cd06d956ada021baba82277fcd141493de35a31bccde645a0"} err="failed to get container status \"325128f3e14fb21cd06d956ada021baba82277fcd141493de35a31bccde645a0\": rpc error: code = NotFound desc = could not find container \"325128f3e14fb21cd06d956ada021baba82277fcd141493de35a31bccde645a0\": container with ID starting with 325128f3e14fb21cd06d956ada021baba82277fcd141493de35a31bccde645a0 not found: ID does not exist"
Oct 30 19:15:21.922361 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.922316    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfeff9d8-6996-4f87-a933-b41f97627004-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfeff9d8-6996-4f87-a933-b41f97627004-maas-api-service-ca\") pod \"bfeff9d8-6996-4f87-a933-b41f97627004\" (UID: \"bfeff9d8-6996-4f87-a933-b41f97627004\") "
Oct 30 19:15:21.922602 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.922557    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3aa2e02-e67a-4418-8fb7-3d805cb7ca7c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3aa2e02-e67a-4418-8fb7-3d805cb7ca7c-maas-api-service-ca\") pod \"d3aa2e02-e67a-4418-8fb7-3d805cb7ca7c\" (UID: \"d3aa2e02-e67a-4418-8fb7-3d805cb7ca7c\") "
Oct 30 19:15:21.922964 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.922868    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfeff9d8-6996-4f87-a933-b41f97627004-maas-api-service-ca" pod "bfeff9d8-6996-4f87-a933-b41f97627004" (UID: "bfeff9d8-6996-4f87-a933-b41f97627004"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:21.923123 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.923087    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfeff9d8-6996-4f87-a933-b41f97627004-kube-api-access-pgpvw\" (UniqueName: \"kubernetes.io/projected/bfeff9d8-6996-4f87-a933-b41f97627004-kube-api-access-pgpvw\") pod \"bfeff9d8-6996-4f87-a933-b41f97627004\" (UID: \"bfeff9d8-6996-4f87-a933-b41f97627004\") "
Oct 30 19:15:21.923253 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.923171    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3aa2e02-e67a-4418-8fb7-3d805cb7ca7c-kube-api-access-bqvgj\" (UniqueName: \"kubernetes.io/projected/d3aa2e02-e67a-4418-8fb7-3d805cb7ca7c-kube-api-access-bqvgj\") pod \"d3aa2e02-e67a-4418-8fb7-3d805cb7ca7c\" (UID: \"d3aa2e02-e67a-4418-8fb7-3d805cb7ca7c\") "
Oct 30 19:15:21.923253 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.923233    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3aa2e02-e67a-4418-8fb7-3d805cb7ca7c-maas-api-service-ca" pod "d3aa2e02-e67a-4418-8fb7-3d805cb7ca7c" (UID: "d3aa2e02-e67a-4418-8fb7-3d805cb7ca7c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:21.924034 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.924012    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfeff9d8-6996-4f87-a933-b41f97627004-maas-api-service-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:21.924127 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.924040    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3aa2e02-e67a-4418-8fb7-3d805cb7ca7c-maas-api-service-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:21.924127 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.924070    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7shqk\" (UniqueName: \"kubernetes.io/projected/e8fa12c9-9b1d-4033-aebb-eabcd9340222-kube-api-access-7shqk\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:21.924127 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.924086    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8fa12c9-9b1d-4033-aebb-eabcd9340222-maas-api-service-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:21.925368 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.925347    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3aa2e02-e67a-4418-8fb7-3d805cb7ca7c-kube-api-access-bqvgj" pod "d3aa2e02-e67a-4418-8fb7-3d805cb7ca7c" (UID: "d3aa2e02-e67a-4418-8fb7-3d805cb7ca7c"). InnerVolumeSpecName "kube-api-access-bqvgj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:21.925486 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:21.925469    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfeff9d8-6996-4f87-a933-b41f97627004-kube-api-access-pgpvw" pod "bfeff9d8-6996-4f87-a933-b41f97627004" (UID: "bfeff9d8-6996-4f87-a933-b41f97627004"). InnerVolumeSpecName "kube-api-access-pgpvw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:22.046771 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:22.046387    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgpvw\" (UniqueName: \"kubernetes.io/projected/bfeff9d8-6996-4f87-a933-b41f97627004-kube-api-access-pgpvw\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:22.046771 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:22.046429    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqvgj\" (UniqueName: \"kubernetes.io/projected/d3aa2e02-e67a-4418-8fb7-3d805cb7ca7c-kube-api-access-bqvgj\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:22.317353 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:22.316895    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9de872-29889795-c7whh"]
Oct 30 19:15:22.328110 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:22.328083    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a1d55d-29889795-bw7v2"]
Oct 30 19:15:22.329129 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:22.329103    4525 generic.go:356] "Generic (PLEG): container finished" podID="8d3ff890-4ac4-4395-b309-3483be728b66" containerID="28e04894d6d40bc263aa74f435a69fd6886115c0ebeac6425bc9f4cce1659983" exitCode=60
Oct 30 19:15:22.329212 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:22.329184    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-ef6b56-29889795-nl7wc" event={"ID":"8d3ff890-4ac4-4395-b309-3483be728b66","Type":"ContainerDied","Data":"28e04894d6d40bc263aa74f435a69fd6886115c0ebeac6425bc9f4cce1659983"}
Oct 30 19:15:22.329271 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:22.329234    4525 scope.go:122] "RemoveContainer" containerID="eca90751c5fcfd6ad9dd0ad5ef8f4ea898e4a9f3fbfc5cfa8218dc1a388ca81f"
Oct 30 19:15:22.334730 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:22.331134    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-68b466-29889795-fgfjm"
Oct 30 19:15:22.334730 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:22.331156    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-68b466-29889795-fgfjm" event={"ID":"bfeff9d8-6996-4f87-a933-b41f97627004","Type":"ContainerDied","Data":"067e08521281614add4d2b29c77d7e746281e0b9c10ef0b0a5505a8a738381b9"}
Oct 30 19:15:22.337667 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:22.337639    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ed012e-29889795-f5xmz"
Oct 30 19:15:22.337859 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:22.337658    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ed012e-29889795-f5xmz" event={"ID":"d3aa2e02-e67a-4418-8fb7-3d805cb7ca7c","Type":"ContainerDied","Data":"879f81e9f1e85325437817bc417ecc3ad4adb3bd0b7fdc74a2b5b7db7d6e020c"}
Oct 30 19:15:22.352958 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:22.352938    4525 scope.go:122] "RemoveContainer" containerID="9b1d63191a3a6857923331f5c32d8827e37d0cadc77b3dfd281d62e8119a672d"
Oct 30 19:15:22.353289 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:22.353149    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dcd326-29889795-zj7rm" event={"ID":"dfff552f-d7bb-414b-9a43-34922f58d2bd","Type":"ContainerDied","Data":"0eeacc2258bce752d44fe4cefeeac433c4ca468f0815cadec11a0ac94fd215ad"}
Oct 30 19:15:22.353396 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:22.353352    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dcd326-29889795-zj7rm"
Oct 30 19:15:22.369060 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:22.368399    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ef6be7-29889795-pf2p2" event={"ID":"e8fa12c9-9b1d-4033-aebb-eabcd9340222","Type":"ContainerDied","Data":"3dcdd4d795a879aacaf8883b61ecf9a49b4bd3204f9ea5726f456e99b9add72d"}
Oct 30 19:15:22.369060 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:22.368448    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ef6be7-29889795-pf2p2"
Oct 30 19:15:22.389711 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:22.389591    4525 scope.go:122] "RemoveContainer" containerID="6a2f621edf85a6c5a49b5d844078fc035bdf29dd61fea58b46d37425fec3b34a"
Oct 30 19:15:22.406445 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:22.406239    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ed012e-29889795-f5xmz"]
Oct 30 19:15:22.429018 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:22.424596    4525 scope.go:122] "RemoveContainer" containerID="1dfe7079ffb0e89ed989e66b35646feae6b4f2d34408ee276b16de581de82239"
Oct 30 19:15:22.435193 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:22.434509    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-ed012e-29889795-f5xmz"]
Oct 30 19:15:22.453175 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:22.453147    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-68b466-29889795-fgfjm"]
Oct 30 19:15:22.454141 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:22.454122    4525 scope.go:122] "RemoveContainer" containerID="6564aa36056e3bf97906ae42625916bc3c85107cd696a2e6bf1c5c3533bb47cb"
Oct 30 19:15:22.460958 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:22.460939    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-68b466-29889795-fgfjm"]
Oct 30 19:15:22.467025 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:22.466953    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2d1c71-29889795-p7txw"]
Oct 30 19:15:22.507884 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:22.507817    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ef6be7-29889795-pf2p2"]
Oct 30 19:15:22.515797 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:22.515770    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ef6be7-29889795-pf2p2"]
Oct 30 19:15:22.522619 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:22.522592    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dcd326-29889795-zj7rm"]
Oct 30 19:15:22.541769 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:22.541602    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dcd326-29889795-zj7rm"]
Oct 30 19:15:22.543650 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:22.543042    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-ef6b56-29889795-nl7wc"
Oct 30 19:15:22.578287 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:22.578209    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-46385c-29889795-hbvv9"]
Oct 30 19:15:22.666504 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:22.666189    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d3ff890-4ac4-4395-b309-3483be728b66-kube-api-access-9z7jt\" (UniqueName: \"kubernetes.io/projected/8d3ff890-4ac4-4395-b309-3483be728b66-kube-api-access-9z7jt\") pod \"8d3ff890-4ac4-4395-b309-3483be728b66\" (UID: \"8d3ff890-4ac4-4395-b309-3483be728b66\") "
Oct 30 19:15:22.666504 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:22.666448    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d3ff890-4ac4-4395-b309-3483be728b66-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d3ff890-4ac4-4395-b309-3483be728b66-maas-api-service-ca\") pod \"8d3ff890-4ac4-4395-b309-3483be728b66\" (UID: \"8d3ff890-4ac4-4395-b309-3483be728b66\") "
Oct 30 19:15:22.669889 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:22.669578    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d3ff890-4ac4-4395-b309-3483be728b66-maas-api-service-ca" pod "8d3ff890-4ac4-4395-b309-3483be728b66" (UID: "8d3ff890-4ac4-4395-b309-3483be728b66"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:22.701934 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:22.701863    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d3ff890-4ac4-4395-b309-3483be728b66-kube-api-access-9z7jt" pod "8d3ff890-4ac4-4395-b309-3483be728b66" (UID: "8d3ff890-4ac4-4395-b309-3483be728b66"). InnerVolumeSpecName "kube-api-access-9z7jt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:22.749948 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:22.749918    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9de872-29889795-c7whh"
Oct 30 19:15:22.752755 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:22.752674    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a1d55d-29889795-bw7v2"
Oct 30 19:15:22.768918 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:22.768885    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9z7jt\" (UniqueName: \"kubernetes.io/projected/8d3ff890-4ac4-4395-b309-3483be728b66-kube-api-access-9z7jt\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:22.769044 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:22.768923    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d3ff890-4ac4-4395-b309-3483be728b66-maas-api-service-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:22.795588 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:22.795560    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2d1c71-29889795-p7txw"
Oct 30 19:15:22.869752 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:22.869722    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e506709f-838a-4255-b3fb-498da9fb0697-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e506709f-838a-4255-b3fb-498da9fb0697-maas-api-service-ca\") pod \"e506709f-838a-4255-b3fb-498da9fb0697\" (UID: \"e506709f-838a-4255-b3fb-498da9fb0697\") "
Oct 30 19:15:22.869987 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:22.869965    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90e98322-b727-4a35-9d93-808e0b01d483-kube-api-access-6wvhv\" (UniqueName: \"kubernetes.io/projected/90e98322-b727-4a35-9d93-808e0b01d483-kube-api-access-6wvhv\") pod \"90e98322-b727-4a35-9d93-808e0b01d483\" (UID: \"90e98322-b727-4a35-9d93-808e0b01d483\") "
Oct 30 19:15:22.870102 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:22.870081    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e506709f-838a-4255-b3fb-498da9fb0697-maas-api-service-ca" pod "e506709f-838a-4255-b3fb-498da9fb0697" (UID: "e506709f-838a-4255-b3fb-498da9fb0697"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:22.870271 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:22.870253    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90e98322-b727-4a35-9d93-808e0b01d483-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90e98322-b727-4a35-9d93-808e0b01d483-maas-api-service-ca\") pod \"90e98322-b727-4a35-9d93-808e0b01d483\" (UID: \"90e98322-b727-4a35-9d93-808e0b01d483\") "
Oct 30 19:15:22.870557 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:22.870537    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e506709f-838a-4255-b3fb-498da9fb0697-kube-api-access-zjxcw\" (UniqueName: \"kubernetes.io/projected/e506709f-838a-4255-b3fb-498da9fb0697-kube-api-access-zjxcw\") pod \"e506709f-838a-4255-b3fb-498da9fb0697\" (UID: \"e506709f-838a-4255-b3fb-498da9fb0697\") "
Oct 30 19:15:22.870890 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:22.870863    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90e98322-b727-4a35-9d93-808e0b01d483-maas-api-service-ca" pod "90e98322-b727-4a35-9d93-808e0b01d483" (UID: "90e98322-b727-4a35-9d93-808e0b01d483"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:22.871398 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:22.871376    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e506709f-838a-4255-b3fb-498da9fb0697-maas-api-service-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:22.871481 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:22.871403    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90e98322-b727-4a35-9d93-808e0b01d483-maas-api-service-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:22.873283 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:22.873263    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90e98322-b727-4a35-9d93-808e0b01d483-kube-api-access-6wvhv" pod "90e98322-b727-4a35-9d93-808e0b01d483" (UID: "90e98322-b727-4a35-9d93-808e0b01d483"). InnerVolumeSpecName "kube-api-access-6wvhv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:22.875832 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:22.875810    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e506709f-838a-4255-b3fb-498da9fb0697-kube-api-access-zjxcw" pod "e506709f-838a-4255-b3fb-498da9fb0697" (UID: "e506709f-838a-4255-b3fb-498da9fb0697"). InnerVolumeSpecName "kube-api-access-zjxcw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:22.880554 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:22.880530    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-46385c-29889795-hbvv9"
Oct 30 19:15:22.973108 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:22.973071    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/403406c5-302f-4563-be3b-63ac8959d517-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/403406c5-302f-4563-be3b-63ac8959d517-maas-api-service-ca\") pod \"403406c5-302f-4563-be3b-63ac8959d517\" (UID: \"403406c5-302f-4563-be3b-63ac8959d517\") "
Oct 30 19:15:22.973620 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:22.973589    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/403406c5-302f-4563-be3b-63ac8959d517-kube-api-access-rqk7b\" (UniqueName: \"kubernetes.io/projected/403406c5-302f-4563-be3b-63ac8959d517-kube-api-access-rqk7b\") pod \"403406c5-302f-4563-be3b-63ac8959d517\" (UID: \"403406c5-302f-4563-be3b-63ac8959d517\") "
Oct 30 19:15:22.974735 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:22.974709    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjxcw\" (UniqueName: \"kubernetes.io/projected/e506709f-838a-4255-b3fb-498da9fb0697-kube-api-access-zjxcw\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:22.974873 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:22.974857    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wvhv\" (UniqueName: \"kubernetes.io/projected/90e98322-b727-4a35-9d93-808e0b01d483-kube-api-access-6wvhv\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:22.981942 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:22.981910    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/403406c5-302f-4563-be3b-63ac8959d517-maas-api-service-ca" pod "403406c5-302f-4563-be3b-63ac8959d517" (UID: "403406c5-302f-4563-be3b-63ac8959d517"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:22.989405 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:22.989323    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/403406c5-302f-4563-be3b-63ac8959d517-kube-api-access-rqk7b" pod "403406c5-302f-4563-be3b-63ac8959d517" (UID: "403406c5-302f-4563-be3b-63ac8959d517"). InnerVolumeSpecName "kube-api-access-rqk7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:23.076326 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:23.076291    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5149d31-2979-4353-9b41-df84f90cffbd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5149d31-2979-4353-9b41-df84f90cffbd-maas-api-service-ca\") pod \"e5149d31-2979-4353-9b41-df84f90cffbd\" (UID: \"e5149d31-2979-4353-9b41-df84f90cffbd\") "
Oct 30 19:15:23.076510 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:23.076352    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5149d31-2979-4353-9b41-df84f90cffbd-kube-api-access-mkjr6\" (UniqueName: \"kubernetes.io/projected/e5149d31-2979-4353-9b41-df84f90cffbd-kube-api-access-mkjr6\") pod \"e5149d31-2979-4353-9b41-df84f90cffbd\" (UID: \"e5149d31-2979-4353-9b41-df84f90cffbd\") "
Oct 30 19:15:23.076700 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:23.076655    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5149d31-2979-4353-9b41-df84f90cffbd-maas-api-service-ca" pod "e5149d31-2979-4353-9b41-df84f90cffbd" (UID: "e5149d31-2979-4353-9b41-df84f90cffbd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:23.076945 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:23.076922    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqk7b\" (UniqueName: \"kubernetes.io/projected/403406c5-302f-4563-be3b-63ac8959d517-kube-api-access-rqk7b\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:23.076945 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:23.076945    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/403406c5-302f-4563-be3b-63ac8959d517-maas-api-service-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:23.077102 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:23.076956    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5149d31-2979-4353-9b41-df84f90cffbd-maas-api-service-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:23.078662 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:23.078640    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5149d31-2979-4353-9b41-df84f90cffbd-kube-api-access-mkjr6" pod "e5149d31-2979-4353-9b41-df84f90cffbd" (UID: "e5149d31-2979-4353-9b41-df84f90cffbd"). InnerVolumeSpecName "kube-api-access-mkjr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:23.178318 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:23.178237    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkjr6\" (UniqueName: \"kubernetes.io/projected/e5149d31-2979-4353-9b41-df84f90cffbd-kube-api-access-mkjr6\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:23.406023 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:23.405983    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-ef6b56-29889795-nl7wc" event={"ID":"8d3ff890-4ac4-4395-b309-3483be728b66","Type":"ContainerDied","Data":"66dadb143e15595d94cdb01893bf2578b6caec9a9e67211295cc75d8c7a5d3d4"}
Oct 30 19:15:23.406476 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:23.405991    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-ef6b56-29889795-nl7wc"
Oct 30 19:15:23.406476 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:23.406091    4525 scope.go:122] "RemoveContainer" containerID="28e04894d6d40bc263aa74f435a69fd6886115c0ebeac6425bc9f4cce1659983"
Oct 30 19:15:23.410262 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:23.410232    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-46385c-29889795-hbvv9"
Oct 30 19:15:23.410589 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:23.410235    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-46385c-29889795-hbvv9" event={"ID":"e5149d31-2979-4353-9b41-df84f90cffbd","Type":"ContainerDied","Data":"c417b4934eb234f909be133b3e9d900d5fb25c41464df24babab393227f44ac3"}
Oct 30 19:15:23.413719 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:23.413661    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a1d55d-29889795-bw7v2"
Oct 30 19:15:23.413845 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:23.413786    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a1d55d-29889795-bw7v2" event={"ID":"90e98322-b727-4a35-9d93-808e0b01d483","Type":"ContainerDied","Data":"635d8514818f3ceaec5c108b24d37b903e0ea0b19677a5757e78e2263ae56448"}
Oct 30 19:15:23.419050 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:23.419030    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9de872-29889795-c7whh" event={"ID":"e506709f-838a-4255-b3fb-498da9fb0697","Type":"ContainerDied","Data":"745203aa516ddccab003daf04f27ca2d666d8dfd907df91f0db2ecdb9de0989b"}
Oct 30 19:15:23.419254 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:23.419105    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9de872-29889795-c7whh"
Oct 30 19:15:23.420496 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:23.420477    4525 scope.go:122] "RemoveContainer" containerID="50dae2a0619d5ea06ef3d80104186a43595bb131ec10c864da3e7d3394a2691a"
Oct 30 19:15:23.421549 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:23.421525    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2d1c71-29889795-p7txw" event={"ID":"403406c5-302f-4563-be3b-63ac8959d517","Type":"ContainerDied","Data":"6b623b63e028314a893df4dc17eafa6ce4f7d0bc19ab9d44c0552cdf5d19fba9"}
Oct 30 19:15:23.421641 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:23.421566    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2d1c71-29889795-p7txw"
Oct 30 19:15:23.435119 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:23.435095    4525 scope.go:122] "RemoveContainer" containerID="56f03a114cb5b7d194dd0e964a1c4aa77ad4f048a182ba64f291aee9b9705bc2"
Oct 30 19:15:23.435937 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:23.435916    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-ef6b56-29889795-nl7wc"]
Oct 30 19:15:23.441109 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:23.441090    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-ef6b56-29889795-nl7wc"]
Oct 30 19:15:23.446241 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:23.446223    4525 scope.go:122] "RemoveContainer" containerID="1aeaae8bc8126bbf16736be43accdfb188a626d8ece3315273da050766cd86e4"
Oct 30 19:15:23.455048 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:23.455029    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a1d55d-29889795-bw7v2"]
Oct 30 19:15:23.456662 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:23.456636    4525 scope.go:122] "RemoveContainer" containerID="f9e32a2808a4f22c8102aaec61db4a750f30c2d4ab5b20a279534913ad60f4b8"
Oct 30 19:15:23.464909 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:23.464889    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a1d55d-29889795-bw7v2"]
Oct 30 19:15:23.476390 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:23.476366    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-46385c-29889795-hbvv9"]
Oct 30 19:15:23.484346 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:23.484315    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-46385c-29889795-hbvv9"]
Oct 30 19:15:23.496104 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:23.496078    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2d1c71-29889795-p7txw"]
Oct 30 19:15:23.508162 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:23.508139    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2d1c71-29889795-p7txw"]
Oct 30 19:15:23.528457 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:23.528429    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9de872-29889795-c7whh"]
Oct 30 19:15:23.532795 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:23.532774    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9de872-29889795-c7whh"]
Oct 30 19:15:23.551303 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:23.551271    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3253644f-14ff-452e-851b-694d8724574f" path="/var/lib/kubelet/pods/3253644f-14ff-452e-851b-694d8724574f/volumes"
Oct 30 19:15:23.552076 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:23.552057    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="403406c5-302f-4563-be3b-63ac8959d517" path="/var/lib/kubelet/pods/403406c5-302f-4563-be3b-63ac8959d517/volumes"
Oct 30 19:15:23.552691 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:23.552659    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d3ff890-4ac4-4395-b309-3483be728b66" path="/var/lib/kubelet/pods/8d3ff890-4ac4-4395-b309-3483be728b66/volumes"
Oct 30 19:15:23.553715 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:23.553672    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90e98322-b727-4a35-9d93-808e0b01d483" path="/var/lib/kubelet/pods/90e98322-b727-4a35-9d93-808e0b01d483/volumes"
Oct 30 19:15:23.554305 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:23.554277    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfeff9d8-6996-4f87-a933-b41f97627004" path="/var/lib/kubelet/pods/bfeff9d8-6996-4f87-a933-b41f97627004/volumes"
Oct 30 19:15:23.554884 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:23.554865    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c67ac146-cadf-480c-b1a0-47e60b73d0e5" path="/var/lib/kubelet/pods/c67ac146-cadf-480c-b1a0-47e60b73d0e5/volumes"
Oct 30 19:15:23.555893 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:23.555873    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3aa2e02-e67a-4418-8fb7-3d805cb7ca7c" path="/var/lib/kubelet/pods/d3aa2e02-e67a-4418-8fb7-3d805cb7ca7c/volumes"
Oct 30 19:15:23.556447 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:23.556427    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfff552f-d7bb-414b-9a43-34922f58d2bd" path="/var/lib/kubelet/pods/dfff552f-d7bb-414b-9a43-34922f58d2bd/volumes"
Oct 30 19:15:23.557030 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:23.557009    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e506709f-838a-4255-b3fb-498da9fb0697" path="/var/lib/kubelet/pods/e506709f-838a-4255-b3fb-498da9fb0697/volumes"
Oct 30 19:15:23.558011 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:23.557981    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5149d31-2979-4353-9b41-df84f90cffbd" path="/var/lib/kubelet/pods/e5149d31-2979-4353-9b41-df84f90cffbd/volumes"
Oct 30 19:15:23.558572 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:23.558552    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8fa12c9-9b1d-4033-aebb-eabcd9340222" path="/var/lib/kubelet/pods/e8fa12c9-9b1d-4033-aebb-eabcd9340222/volumes"
Oct 30 19:15:25.262072 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:25.262039    4525 scope.go:122] "RemoveContainer" containerID="151aa2cef83566faae57d2fd168e9f29169251433ebed50ddd24f24c792e7b8f"
Oct 30 19:15:32.953258 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.953211    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c2e3f9-mgxxs"]
Oct 30 19:15:32.954932 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.954900    4525 cpu_manager.go:414] "containerMap: removing container" podUID="6ccf27c7-f6d3-457b-9ff8-69b54f9b9287" containerName="istio-proxy"
Oct 30 19:15:32.955102 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.955087    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ccf27c7-f6d3-457b-9ff8-69b54f9b9287" containerName="istio-proxy"
Oct 30 19:15:32.955212 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.955201    4525 cpu_manager.go:414] "containerMap: removing container" podUID="3253644f-14ff-452e-851b-694d8724574f" containerName="cleanup"
Oct 30 19:15:32.955297 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.955286    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3253644f-14ff-452e-851b-694d8724574f" containerName="cleanup"
Oct 30 19:15:32.955383 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.955371    4525 cpu_manager.go:414] "containerMap: removing container" podUID="e8fa12c9-9b1d-4033-aebb-eabcd9340222" containerName="cleanup"
Oct 30 19:15:32.955468 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.955458    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8fa12c9-9b1d-4033-aebb-eabcd9340222" containerName="cleanup"
Oct 30 19:15:32.955556 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.955546    4525 cpu_manager.go:414] "containerMap: removing container" podUID="403406c5-302f-4563-be3b-63ac8959d517" containerName="cleanup"
Oct 30 19:15:32.955640 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.955630    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="403406c5-302f-4563-be3b-63ac8959d517" containerName="cleanup"
Oct 30 19:15:32.955758 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.955746    4525 cpu_manager.go:414] "containerMap: removing container" podUID="a8e73ad5-17f6-4205-bc5f-8965afa49f57" containerName="istio-proxy"
Oct 30 19:15:32.955850 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.955839    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8e73ad5-17f6-4205-bc5f-8965afa49f57" containerName="istio-proxy"
Oct 30 19:15:32.955940 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.955930    4525 cpu_manager.go:414] "containerMap: removing container" podUID="865afd2d-c5e2-4ee5-bfdb-303102de6616" containerName="istio-proxy"
Oct 30 19:15:32.956020 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.956010    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="865afd2d-c5e2-4ee5-bfdb-303102de6616" containerName="istio-proxy"
Oct 30 19:15:32.956106 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.956096    4525 cpu_manager.go:414] "containerMap: removing container" podUID="4849dbc2-0043-46f1-ab5e-a76c16f85db9" containerName="cleanup"
Oct 30 19:15:32.956190 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.956180    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4849dbc2-0043-46f1-ab5e-a76c16f85db9" containerName="cleanup"
Oct 30 19:15:32.956270 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.956260    4525 cpu_manager.go:414] "containerMap: removing container" podUID="7590ba0b-baad-4a0a-ad6c-3309f2123c5c" containerName="cleanup"
Oct 30 19:15:32.956353 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.956343    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7590ba0b-baad-4a0a-ad6c-3309f2123c5c" containerName="cleanup"
Oct 30 19:15:32.956439 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.956429    4525 cpu_manager.go:414] "containerMap: removing container" podUID="c67ac146-cadf-480c-b1a0-47e60b73d0e5" containerName="cleanup"
Oct 30 19:15:32.956518 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.956508    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c67ac146-cadf-480c-b1a0-47e60b73d0e5" containerName="cleanup"
Oct 30 19:15:32.956613 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.956603    4525 cpu_manager.go:414] "containerMap: removing container" podUID="4849dbc2-0043-46f1-ab5e-a76c16f85db9" containerName="cleanup"
Oct 30 19:15:32.956715 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.956702    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4849dbc2-0043-46f1-ab5e-a76c16f85db9" containerName="cleanup"
Oct 30 19:15:32.956809 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.956799    4525 cpu_manager.go:414] "containerMap: removing container" podUID="bfeff9d8-6996-4f87-a933-b41f97627004" containerName="cleanup"
Oct 30 19:15:32.956889 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.956878    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfeff9d8-6996-4f87-a933-b41f97627004" containerName="cleanup"
Oct 30 19:15:32.956980 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.956970    4525 cpu_manager.go:414] "containerMap: removing container" podUID="90e98322-b727-4a35-9d93-808e0b01d483" containerName="cleanup"
Oct 30 19:15:32.957060 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.957050    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90e98322-b727-4a35-9d93-808e0b01d483" containerName="cleanup"
Oct 30 19:15:32.957144 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.957134    4525 cpu_manager.go:414] "containerMap: removing container" podUID="be16530b-c26d-4da7-b25e-f5246eff6c3d" containerName="istio-proxy"
Oct 30 19:15:32.957212 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.957202    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be16530b-c26d-4da7-b25e-f5246eff6c3d" containerName="istio-proxy"
Oct 30 19:15:32.957299 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.957290    4525 cpu_manager.go:414] "containerMap: removing container" podUID="8d3ff890-4ac4-4395-b309-3483be728b66" containerName="cleanup"
Oct 30 19:15:32.957376 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.957365    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d3ff890-4ac4-4395-b309-3483be728b66" containerName="cleanup"
Oct 30 19:15:32.957452 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.957442    4525 cpu_manager.go:414] "containerMap: removing container" podUID="75f454fd-6c51-42f1-a19d-24d1c739a88c" containerName="cleanup"
Oct 30 19:15:32.957539 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.957527    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75f454fd-6c51-42f1-a19d-24d1c739a88c" containerName="cleanup"
Oct 30 19:15:32.957622 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.957613    4525 cpu_manager.go:414] "containerMap: removing container" podUID="e5149d31-2979-4353-9b41-df84f90cffbd" containerName="cleanup"
Oct 30 19:15:32.957723 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.957711    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5149d31-2979-4353-9b41-df84f90cffbd" containerName="cleanup"
Oct 30 19:15:32.957817 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.957808    4525 cpu_manager.go:414] "containerMap: removing container" podUID="d3aa2e02-e67a-4418-8fb7-3d805cb7ca7c" containerName="cleanup"
Oct 30 19:15:32.957894 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.957884    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3aa2e02-e67a-4418-8fb7-3d805cb7ca7c" containerName="cleanup"
Oct 30 19:15:32.957974 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.957965    4525 cpu_manager.go:414] "containerMap: removing container" podUID="3253644f-14ff-452e-851b-694d8724574f" containerName="cleanup"
Oct 30 19:15:32.958062 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.958051    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3253644f-14ff-452e-851b-694d8724574f" containerName="cleanup"
Oct 30 19:15:32.958143 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.958133    4525 cpu_manager.go:414] "containerMap: removing container" podUID="bfeff9d8-6996-4f87-a933-b41f97627004" containerName="cleanup"
Oct 30 19:15:32.958224 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.958214    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfeff9d8-6996-4f87-a933-b41f97627004" containerName="cleanup"
Oct 30 19:15:32.958310 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.958299    4525 cpu_manager.go:414] "containerMap: removing container" podUID="77adecb1-af53-4dd9-bb52-d8e22ebb03e4" containerName="istio-proxy"
Oct 30 19:15:32.958398 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.958385    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77adecb1-af53-4dd9-bb52-d8e22ebb03e4" containerName="istio-proxy"
Oct 30 19:15:32.958478 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.958468    4525 cpu_manager.go:414] "containerMap: removing container" podUID="dfff552f-d7bb-414b-9a43-34922f58d2bd" containerName="cleanup"
Oct 30 19:15:32.958555 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.958545    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfff552f-d7bb-414b-9a43-34922f58d2bd" containerName="cleanup"
Oct 30 19:15:32.958643 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.958633    4525 cpu_manager.go:414] "containerMap: removing container" podUID="e506709f-838a-4255-b3fb-498da9fb0697" containerName="cleanup"
Oct 30 19:15:32.958755 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.958742    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e506709f-838a-4255-b3fb-498da9fb0697" containerName="cleanup"
Oct 30 19:15:32.958845 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.958836    4525 cpu_manager.go:414] "containerMap: removing container" podUID="a8e73ad5-17f6-4205-bc5f-8965afa49f57" containerName="istio-proxy"
Oct 30 19:15:32.958928 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.958918    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8e73ad5-17f6-4205-bc5f-8965afa49f57" containerName="istio-proxy"
Oct 30 19:15:32.959015 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.959006    4525 cpu_manager.go:414] "containerMap: removing container" podUID="01d8e335-0163-44d7-b811-fb9b99cd285a" containerName="cleanup"
Oct 30 19:15:32.959097 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.959087    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01d8e335-0163-44d7-b811-fb9b99cd285a" containerName="cleanup"
Oct 30 19:15:32.959181 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.959169    4525 cpu_manager.go:414] "containerMap: removing container" podUID="e5149d31-2979-4353-9b41-df84f90cffbd" containerName="cleanup"
Oct 30 19:15:32.959262 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.959252    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5149d31-2979-4353-9b41-df84f90cffbd" containerName="cleanup"
Oct 30 19:15:32.959345 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.959336    4525 cpu_manager.go:414] "containerMap: removing container" podUID="e506709f-838a-4255-b3fb-498da9fb0697" containerName="cleanup"
Oct 30 19:15:32.959419 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.959409    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e506709f-838a-4255-b3fb-498da9fb0697" containerName="cleanup"
Oct 30 19:15:32.959498 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.959488    4525 cpu_manager.go:414] "containerMap: removing container" podUID="dfff552f-d7bb-414b-9a43-34922f58d2bd" containerName="cleanup"
Oct 30 19:15:32.959573 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.959563    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfff552f-d7bb-414b-9a43-34922f58d2bd" containerName="cleanup"
Oct 30 19:15:32.959657 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.959647    4525 cpu_manager.go:414] "containerMap: removing container" podUID="be563a8d-e16a-4b6a-b82d-4d6c4847a527" containerName="istio-proxy"
Oct 30 19:15:32.959766 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.959753    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be563a8d-e16a-4b6a-b82d-4d6c4847a527" containerName="istio-proxy"
Oct 30 19:15:32.959853 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.959843    4525 cpu_manager.go:414] "containerMap: removing container" podUID="90e98322-b727-4a35-9d93-808e0b01d483" containerName="cleanup"
Oct 30 19:15:32.959931 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.959921    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90e98322-b727-4a35-9d93-808e0b01d483" containerName="cleanup"
Oct 30 19:15:32.960009 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.959999    4525 cpu_manager.go:414] "containerMap: removing container" podUID="01d8e335-0163-44d7-b811-fb9b99cd285a" containerName="cleanup"
Oct 30 19:15:32.960085 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.960074    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01d8e335-0163-44d7-b811-fb9b99cd285a" containerName="cleanup"
Oct 30 19:15:32.960185 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.960175    4525 cpu_manager.go:414] "containerMap: removing container" podUID="e5149d31-2979-4353-9b41-df84f90cffbd" containerName="cleanup"
Oct 30 19:15:32.960268 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.960258    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5149d31-2979-4353-9b41-df84f90cffbd" containerName="cleanup"
Oct 30 19:15:32.960349 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.960339    4525 cpu_manager.go:414] "containerMap: removing container" podUID="75f454fd-6c51-42f1-a19d-24d1c739a88c" containerName="cleanup"
Oct 30 19:15:32.960425 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.960415    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75f454fd-6c51-42f1-a19d-24d1c739a88c" containerName="cleanup"
Oct 30 19:15:32.960505 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.960496    4525 cpu_manager.go:414] "containerMap: removing container" podUID="8d3ff890-4ac4-4395-b309-3483be728b66" containerName="cleanup"
Oct 30 19:15:32.960575 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.960565    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d3ff890-4ac4-4395-b309-3483be728b66" containerName="cleanup"
Oct 30 19:15:32.960662 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.960652    4525 cpu_manager.go:414] "containerMap: removing container" podUID="c67ac146-cadf-480c-b1a0-47e60b73d0e5" containerName="cleanup"
Oct 30 19:15:32.960772 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.960760    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c67ac146-cadf-480c-b1a0-47e60b73d0e5" containerName="cleanup"
Oct 30 19:15:32.960863 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.960853    4525 cpu_manager.go:414] "containerMap: removing container" podUID="e8fa12c9-9b1d-4033-aebb-eabcd9340222" containerName="cleanup"
Oct 30 19:15:32.960938 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.960929    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8fa12c9-9b1d-4033-aebb-eabcd9340222" containerName="cleanup"
Oct 30 19:15:32.961017 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.961008    4525 cpu_manager.go:414] "containerMap: removing container" podUID="d3aa2e02-e67a-4418-8fb7-3d805cb7ca7c" containerName="cleanup"
Oct 30 19:15:32.961093 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.961084    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3aa2e02-e67a-4418-8fb7-3d805cb7ca7c" containerName="cleanup"
Oct 30 19:15:32.961175 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.961164    4525 cpu_manager.go:414] "containerMap: removing container" podUID="75f454fd-6c51-42f1-a19d-24d1c739a88c" containerName="cleanup"
Oct 30 19:15:32.961250 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.961240    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75f454fd-6c51-42f1-a19d-24d1c739a88c" containerName="cleanup"
Oct 30 19:15:32.961337 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.961327    4525 cpu_manager.go:414] "containerMap: removing container" podUID="dfff552f-d7bb-414b-9a43-34922f58d2bd" containerName="cleanup"
Oct 30 19:15:32.961412 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.961402    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfff552f-d7bb-414b-9a43-34922f58d2bd" containerName="cleanup"
Oct 30 19:15:32.961492 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.961483    4525 cpu_manager.go:414] "containerMap: removing container" podUID="d3aa2e02-e67a-4418-8fb7-3d805cb7ca7c" containerName="cleanup"
Oct 30 19:15:32.961575 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.961565    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3aa2e02-e67a-4418-8fb7-3d805cb7ca7c" containerName="cleanup"
Oct 30 19:15:32.961657 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.961648    4525 cpu_manager.go:414] "containerMap: removing container" podUID="8d3ff890-4ac4-4395-b309-3483be728b66" containerName="cleanup"
Oct 30 19:15:32.967868 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.967845    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d3ff890-4ac4-4395-b309-3483be728b66" containerName="cleanup"
Oct 30 19:15:32.967990 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.967978    4525 cpu_manager.go:414] "containerMap: removing container" podUID="90e98322-b727-4a35-9d93-808e0b01d483" containerName="cleanup"
Oct 30 19:15:32.968074 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.968063    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90e98322-b727-4a35-9d93-808e0b01d483" containerName="cleanup"
Oct 30 19:15:32.968155 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.968146    4525 cpu_manager.go:414] "containerMap: removing container" podUID="e9973f1f-61b0-4d38-a7c2-b9c6fde9da94" containerName="istio-proxy"
Oct 30 19:15:32.968232 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.968222    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9973f1f-61b0-4d38-a7c2-b9c6fde9da94" containerName="istio-proxy"
Oct 30 19:15:32.968309 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.968300    4525 cpu_manager.go:414] "containerMap: removing container" podUID="4849dbc2-0043-46f1-ab5e-a76c16f85db9" containerName="cleanup"
Oct 30 19:15:32.968391 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.968381    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4849dbc2-0043-46f1-ab5e-a76c16f85db9" containerName="cleanup"
Oct 30 19:15:32.977698 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.968461    4525 cpu_manager.go:414] "containerMap: removing container" podUID="403406c5-302f-4563-be3b-63ac8959d517" containerName="cleanup"
Oct 30 19:15:32.977698 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.968472    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="403406c5-302f-4563-be3b-63ac8959d517" containerName="cleanup"
Oct 30 19:15:32.977698 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.968488    4525 cpu_manager.go:414] "containerMap: removing container" podUID="403406c5-302f-4563-be3b-63ac8959d517" containerName="cleanup"
Oct 30 19:15:32.977698 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.968495    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="403406c5-302f-4563-be3b-63ac8959d517" containerName="cleanup"
Oct 30 19:15:32.977698 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.968504    4525 cpu_manager.go:414] "containerMap: removing container" podUID="77adecb1-af53-4dd9-bb52-d8e22ebb03e4" containerName="istio-proxy"
Oct 30 19:15:32.977698 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.968511    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77adecb1-af53-4dd9-bb52-d8e22ebb03e4" containerName="istio-proxy"
Oct 30 19:15:32.977698 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.968526    4525 cpu_manager.go:414] "containerMap: removing container" podUID="e506709f-838a-4255-b3fb-498da9fb0697" containerName="cleanup"
Oct 30 19:15:32.977698 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.968532    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e506709f-838a-4255-b3fb-498da9fb0697" containerName="cleanup"
Oct 30 19:15:32.977698 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.968791    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="403406c5-302f-4563-be3b-63ac8959d517" containerName="cleanup"
Oct 30 19:15:32.977698 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.968808    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="77adecb1-af53-4dd9-bb52-d8e22ebb03e4" containerName="istio-proxy"
Oct 30 19:15:32.977698 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.968818    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="e506709f-838a-4255-b3fb-498da9fb0697" containerName="cleanup"
Oct 30 19:15:32.977698 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.968828    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfeff9d8-6996-4f87-a933-b41f97627004" containerName="cleanup"
Oct 30 19:15:32.977698 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.968837    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ccf27c7-f6d3-457b-9ff8-69b54f9b9287" containerName="istio-proxy"
Oct 30 19:15:32.977698 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.968846    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="3253644f-14ff-452e-851b-694d8724574f" containerName="cleanup"
Oct 30 19:15:32.977698 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.968854    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8fa12c9-9b1d-4033-aebb-eabcd9340222" containerName="cleanup"
Oct 30 19:15:32.977698 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.968862    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="403406c5-302f-4563-be3b-63ac8959d517" containerName="cleanup"
Oct 30 19:15:32.977698 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.968871    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8e73ad5-17f6-4205-bc5f-8965afa49f57" containerName="istio-proxy"
Oct 30 19:15:32.977698 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.968879    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="865afd2d-c5e2-4ee5-bfdb-303102de6616" containerName="istio-proxy"
Oct 30 19:15:32.977698 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.968888    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="4849dbc2-0043-46f1-ab5e-a76c16f85db9" containerName="cleanup"
Oct 30 19:15:32.977698 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.968898    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="7590ba0b-baad-4a0a-ad6c-3309f2123c5c" containerName="cleanup"
Oct 30 19:15:32.977698 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.968905    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="c67ac146-cadf-480c-b1a0-47e60b73d0e5" containerName="cleanup"
Oct 30 19:15:32.977698 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.968916    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="4849dbc2-0043-46f1-ab5e-a76c16f85db9" containerName="cleanup"
Oct 30 19:15:32.977698 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.968924    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfeff9d8-6996-4f87-a933-b41f97627004" containerName="cleanup"
Oct 30 19:15:32.977698 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.968936    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="90e98322-b727-4a35-9d93-808e0b01d483" containerName="cleanup"
Oct 30 19:15:32.977698 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.968945    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="be16530b-c26d-4da7-b25e-f5246eff6c3d" containerName="istio-proxy"
Oct 30 19:15:32.977698 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.968955    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d3ff890-4ac4-4395-b309-3483be728b66" containerName="cleanup"
Oct 30 19:15:32.977698 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.968963    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="75f454fd-6c51-42f1-a19d-24d1c739a88c" containerName="cleanup"
Oct 30 19:15:32.977698 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.968971    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5149d31-2979-4353-9b41-df84f90cffbd" containerName="cleanup"
Oct 30 19:15:32.977698 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.968980    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3aa2e02-e67a-4418-8fb7-3d805cb7ca7c" containerName="cleanup"
Oct 30 19:15:32.977698 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.968989    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="3253644f-14ff-452e-851b-694d8724574f" containerName="cleanup"
Oct 30 19:15:32.977698 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.968996    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfeff9d8-6996-4f87-a933-b41f97627004" containerName="cleanup"
Oct 30 19:15:32.977698 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.969005    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfff552f-d7bb-414b-9a43-34922f58d2bd" containerName="cleanup"
Oct 30 19:15:32.977698 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.969014    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8fa12c9-9b1d-4033-aebb-eabcd9340222" containerName="cleanup"
Oct 30 19:15:32.977698 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.969022    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8e73ad5-17f6-4205-bc5f-8965afa49f57" containerName="istio-proxy"
Oct 30 19:15:32.977698 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.969032    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="01d8e335-0163-44d7-b811-fb9b99cd285a" containerName="cleanup"
Oct 30 19:15:32.977698 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.969039    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5149d31-2979-4353-9b41-df84f90cffbd" containerName="cleanup"
Oct 30 19:15:32.977698 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.969048    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="e506709f-838a-4255-b3fb-498da9fb0697" containerName="cleanup"
Oct 30 19:15:32.977698 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.969056    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="be563a8d-e16a-4b6a-b82d-4d6c4847a527" containerName="istio-proxy"
Oct 30 19:15:32.977698 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.969067    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="90e98322-b727-4a35-9d93-808e0b01d483" containerName="cleanup"
Oct 30 19:15:32.977698 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.969076    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="01d8e335-0163-44d7-b811-fb9b99cd285a" containerName="cleanup"
Oct 30 19:15:32.977698 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.969085    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5149d31-2979-4353-9b41-df84f90cffbd" containerName="cleanup"
Oct 30 19:15:32.977698 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.969095    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="75f454fd-6c51-42f1-a19d-24d1c739a88c" containerName="cleanup"
Oct 30 19:15:32.977698 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.969104    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d3ff890-4ac4-4395-b309-3483be728b66" containerName="cleanup"
Oct 30 19:15:32.977698 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.969115    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3aa2e02-e67a-4418-8fb7-3d805cb7ca7c" containerName="cleanup"
Oct 30 19:15:32.977698 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.969124    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="75f454fd-6c51-42f1-a19d-24d1c739a88c" containerName="cleanup"
Oct 30 19:15:32.977698 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.969133    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfff552f-d7bb-414b-9a43-34922f58d2bd" containerName="cleanup"
Oct 30 19:15:32.977698 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.969140    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3aa2e02-e67a-4418-8fb7-3d805cb7ca7c" containerName="cleanup"
Oct 30 19:15:32.977698 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.969151    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d3ff890-4ac4-4395-b309-3483be728b66" containerName="cleanup"
Oct 30 19:15:32.977698 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.969160    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="90e98322-b727-4a35-9d93-808e0b01d483" containerName="cleanup"
Oct 30 19:15:32.977698 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.969169    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9973f1f-61b0-4d38-a7c2-b9c6fde9da94" containerName="istio-proxy"
Oct 30 19:15:32.977698 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:32.969178    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="c67ac146-cadf-480c-b1a0-47e60b73d0e5" containerName="cleanup"
Oct 30 19:15:33.108944 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:33.108904    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c2e3f9-mgxxs"]
Oct 30 19:15:33.116910 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:33.116878    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c2e3f9-mgxxs"
Oct 30 19:15:33.127705 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:33.123610    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:15:33.127705 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:33.123840    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94hcn\""
Oct 30 19:15:33.131082 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:33.131046    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-ed012e-s6vzt"]
Oct 30 19:15:33.132081 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:33.132051    4525 cpu_manager.go:414] "containerMap: removing container" podUID="bfeff9d8-6996-4f87-a933-b41f97627004" containerName="cleanup"
Oct 30 19:15:33.132081 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:33.132080    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfeff9d8-6996-4f87-a933-b41f97627004" containerName="cleanup"
Oct 30 19:15:33.132276 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:33.132104    4525 cpu_manager.go:414] "containerMap: removing container" podUID="3253644f-14ff-452e-851b-694d8724574f" containerName="cleanup"
Oct 30 19:15:33.132276 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:33.132113    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3253644f-14ff-452e-851b-694d8724574f" containerName="cleanup"
Oct 30 19:15:33.132276 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:33.132146    4525 cpu_manager.go:414] "containerMap: removing container" podUID="01d8e335-0163-44d7-b811-fb9b99cd285a" containerName="cleanup"
Oct 30 19:15:33.132276 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:33.132155    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01d8e335-0163-44d7-b811-fb9b99cd285a" containerName="cleanup"
Oct 30 19:15:33.132276 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:33.132173    4525 cpu_manager.go:414] "containerMap: removing container" podUID="e8fa12c9-9b1d-4033-aebb-eabcd9340222" containerName="cleanup"
Oct 30 19:15:33.132276 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:33.132180    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8fa12c9-9b1d-4033-aebb-eabcd9340222" containerName="cleanup"
Oct 30 19:15:33.132276 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:33.132218    4525 cpu_manager.go:414] "containerMap: removing container" podUID="6ccf27c7-f6d3-457b-9ff8-69b54f9b9287" containerName="istio-proxy"
Oct 30 19:15:33.132276 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:33.132226    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ccf27c7-f6d3-457b-9ff8-69b54f9b9287" containerName="istio-proxy"
Oct 30 19:15:33.132276 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:33.132242    4525 cpu_manager.go:414] "containerMap: removing container" podUID="7590ba0b-baad-4a0a-ad6c-3309f2123c5c" containerName="cleanup"
Oct 30 19:15:33.132276 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:33.132249    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7590ba0b-baad-4a0a-ad6c-3309f2123c5c" containerName="cleanup"
Oct 30 19:15:33.132276 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:33.132260    4525 cpu_manager.go:414] "containerMap: removing container" podUID="c67ac146-cadf-480c-b1a0-47e60b73d0e5" containerName="cleanup"
Oct 30 19:15:33.132276 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:33.132267    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c67ac146-cadf-480c-b1a0-47e60b73d0e5" containerName="cleanup"
Oct 30 19:15:33.132967 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:33.132427    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="77adecb1-af53-4dd9-bb52-d8e22ebb03e4" containerName="istio-proxy"
Oct 30 19:15:33.132967 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:33.132443    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ccf27c7-f6d3-457b-9ff8-69b54f9b9287" containerName="istio-proxy"
Oct 30 19:15:33.132967 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:33.132452    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8fa12c9-9b1d-4033-aebb-eabcd9340222" containerName="cleanup"
Oct 30 19:15:33.132967 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:33.132463    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="403406c5-302f-4563-be3b-63ac8959d517" containerName="cleanup"
Oct 30 19:15:33.132967 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:33.132473    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="7590ba0b-baad-4a0a-ad6c-3309f2123c5c" containerName="cleanup"
Oct 30 19:15:33.132967 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:33.132480    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="c67ac146-cadf-480c-b1a0-47e60b73d0e5" containerName="cleanup"
Oct 30 19:15:33.132967 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:33.132488    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="4849dbc2-0043-46f1-ab5e-a76c16f85db9" containerName="cleanup"
Oct 30 19:15:33.132967 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:33.132500    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="3253644f-14ff-452e-851b-694d8724574f" containerName="cleanup"
Oct 30 19:15:33.132967 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:33.132511    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfff552f-d7bb-414b-9a43-34922f58d2bd" containerName="cleanup"
Oct 30 19:15:33.132967 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:33.132523    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="e506709f-838a-4255-b3fb-498da9fb0697" containerName="cleanup"
Oct 30 19:15:33.132967 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:33.132534    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="01d8e335-0163-44d7-b811-fb9b99cd285a" containerName="cleanup"
Oct 30 19:15:33.148886 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:33.148850    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-ed012e-s6vzt"
Oct 30 19:15:33.155781 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:33.155748    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-ed012e-s6vzt"]
Oct 30 19:15:33.334325 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:33.334233    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef28c15d-d04d-4186-ad97-5c88f5f89795-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-ed012e-s6vzt\" (UID: \"ef28c15d-d04d-4186-ad97-5c88f5f89795\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-ed012e-s6vzt"
Oct 30 19:15:33.334325 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:33.334287    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef28c15d-d04d-4186-ad97-5c88f5f89795-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-ed012e-s6vzt\" (UID: \"ef28c15d-d04d-4186-ad97-5c88f5f89795\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-ed012e-s6vzt"
Oct 30 19:15:33.334576 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:33.334356    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z4cg\" (UniqueName: \"kubernetes.io/projected/e80605e4-644b-42fc-b81d-3c1885a99496-kube-api-access-4z4cg\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-c2e3f9-mgxxs\" (UID: \"e80605e4-644b-42fc-b81d-3c1885a99496\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c2e3f9-mgxxs"
Oct 30 19:15:33.334576 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:33.334529    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e80605e4-644b-42fc-b81d-3c1885a99496-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-c2e3f9-mgxxs\" (UID: \"e80605e4-644b-42fc-b81d-3c1885a99496\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c2e3f9-mgxxs"
Oct 30 19:15:33.334746 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:33.334730    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhp5q\" (UniqueName: \"kubernetes.io/projected/ef28c15d-d04d-4186-ad97-5c88f5f89795-kube-api-access-hhp5q\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-ed012e-s6vzt\" (UID: \"ef28c15d-d04d-4186-ad97-5c88f5f89795\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-ed012e-s6vzt"
Oct 30 19:15:33.334792 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:33.334768    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e80605e4-644b-42fc-b81d-3c1885a99496-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-c2e3f9-mgxxs\" (UID: \"e80605e4-644b-42fc-b81d-3c1885a99496\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c2e3f9-mgxxs"
Oct 30 19:15:33.436235 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:33.436200    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhp5q\" (UniqueName: \"kubernetes.io/projected/ef28c15d-d04d-4186-ad97-5c88f5f89795-kube-api-access-hhp5q\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-ed012e-s6vzt\" (UID: \"ef28c15d-d04d-4186-ad97-5c88f5f89795\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-ed012e-s6vzt"
Oct 30 19:15:33.436445 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:33.436241    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e80605e4-644b-42fc-b81d-3c1885a99496-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-c2e3f9-mgxxs\" (UID: \"e80605e4-644b-42fc-b81d-3c1885a99496\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c2e3f9-mgxxs"
Oct 30 19:15:33.436445 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:33.436384    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef28c15d-d04d-4186-ad97-5c88f5f89795-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-ed012e-s6vzt\" (UID: \"ef28c15d-d04d-4186-ad97-5c88f5f89795\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-ed012e-s6vzt"
Oct 30 19:15:33.436445 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:33.436430    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef28c15d-d04d-4186-ad97-5c88f5f89795-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-ed012e-s6vzt\" (UID: \"ef28c15d-d04d-4186-ad97-5c88f5f89795\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-ed012e-s6vzt"
Oct 30 19:15:33.436649 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:33.436473    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z4cg\" (UniqueName: \"kubernetes.io/projected/e80605e4-644b-42fc-b81d-3c1885a99496-kube-api-access-4z4cg\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-c2e3f9-mgxxs\" (UID: \"e80605e4-644b-42fc-b81d-3c1885a99496\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c2e3f9-mgxxs"
Oct 30 19:15:33.436649 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:33.436563    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e80605e4-644b-42fc-b81d-3c1885a99496-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-c2e3f9-mgxxs\" (UID: \"e80605e4-644b-42fc-b81d-3c1885a99496\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c2e3f9-mgxxs"
Oct 30 19:15:33.437221 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:33.437190    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef28c15d-d04d-4186-ad97-5c88f5f89795-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-ed012e-s6vzt\" (UID: \"ef28c15d-d04d-4186-ad97-5c88f5f89795\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-ed012e-s6vzt"
Oct 30 19:15:33.437363 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:33.437312    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e80605e4-644b-42fc-b81d-3c1885a99496-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-c2e3f9-mgxxs\" (UID: \"e80605e4-644b-42fc-b81d-3c1885a99496\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c2e3f9-mgxxs"
Oct 30 19:15:33.439056 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:33.439033    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef28c15d-d04d-4186-ad97-5c88f5f89795-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-ed012e-s6vzt\" (UID: \"ef28c15d-d04d-4186-ad97-5c88f5f89795\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-ed012e-s6vzt"
Oct 30 19:15:33.439207 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:33.439070    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e80605e4-644b-42fc-b81d-3c1885a99496-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-c2e3f9-mgxxs\" (UID: \"e80605e4-644b-42fc-b81d-3c1885a99496\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c2e3f9-mgxxs"
Oct 30 19:15:33.444817 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:33.444786    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z4cg\" (UniqueName: \"kubernetes.io/projected/e80605e4-644b-42fc-b81d-3c1885a99496-kube-api-access-4z4cg\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-c2e3f9-mgxxs\" (UID: \"e80605e4-644b-42fc-b81d-3c1885a99496\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c2e3f9-mgxxs"
Oct 30 19:15:33.444915 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:33.444845    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhp5q\" (UniqueName: \"kubernetes.io/projected/ef28c15d-d04d-4186-ad97-5c88f5f89795-kube-api-access-hhp5q\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-ed012e-s6vzt\" (UID: \"ef28c15d-d04d-4186-ad97-5c88f5f89795\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-ed012e-s6vzt"
Oct 30 19:15:33.478151 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:33.478124    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c2e3f9-mgxxs"
Oct 30 19:15:33.499674 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:33.499652    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-ed012e-s6vzt"
Oct 30 19:15:33.773716 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:33.772914    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c2e3f9-mgxxs"]
Oct 30 19:15:33.776633 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:15:33.775907    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode80605e4_644b_42fc_b81d_3c1885a99496.slice/crio-9193f86333e194c7cd36478e60f5eb5a93e9e71d681f5fa8829e57c56ded696f WatchSource:0}: Error finding container 9193f86333e194c7cd36478e60f5eb5a93e9e71d681f5fa8829e57c56ded696f: Status 404 returned error can't find the container with id 9193f86333e194c7cd36478e60f5eb5a93e9e71d681f5fa8829e57c56ded696f
Oct 30 19:15:33.799414 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:15:33.799389    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef28c15d_d04d_4186_ad97_5c88f5f89795.slice/crio-361f0de4b88665f2da10db3f97d2e6e1903ec8e85f150fa467527a1f72aa155a WatchSource:0}: Error finding container 361f0de4b88665f2da10db3f97d2e6e1903ec8e85f150fa467527a1f72aa155a: Status 404 returned error can't find the container with id 361f0de4b88665f2da10db3f97d2e6e1903ec8e85f150fa467527a1f72aa155a
Oct 30 19:15:33.800324 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:33.800298    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-ed012e-s6vzt"]
Oct 30 19:15:34.747741 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:34.747695    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-ed012e-s6vzt" event={"ID":"ef28c15d-d04d-4186-ad97-5c88f5f89795","Type":"ContainerStarted","Data":"6359945997b3c810f4b94f990bbb77e393376667352d1e1e40b853b9bd39ad6e"}
Oct 30 19:15:34.748168 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:34.747754    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-ed012e-s6vzt" event={"ID":"ef28c15d-d04d-4186-ad97-5c88f5f89795","Type":"ContainerStarted","Data":"361f0de4b88665f2da10db3f97d2e6e1903ec8e85f150fa467527a1f72aa155a"}
Oct 30 19:15:34.750395 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:34.750365    4525 generic.go:356] "Generic (PLEG): container finished" podID="e80605e4-644b-42fc-b81d-3c1885a99496" containerID="7e6bea2706a6fee5263396d27bbf29dafa4bd33e7ad5c0dd7072b4d83525fb14" exitCode=0
Oct 30 19:15:34.750561 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:34.750451    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c2e3f9-mgxxs" event={"ID":"e80605e4-644b-42fc-b81d-3c1885a99496","Type":"ContainerDied","Data":"7e6bea2706a6fee5263396d27bbf29dafa4bd33e7ad5c0dd7072b4d83525fb14"}
Oct 30 19:15:34.750561 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:34.750508    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c2e3f9-mgxxs" event={"ID":"e80605e4-644b-42fc-b81d-3c1885a99496","Type":"ContainerStarted","Data":"9193f86333e194c7cd36478e60f5eb5a93e9e71d681f5fa8829e57c56ded696f"}
Oct 30 19:15:34.765739 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:34.765694    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-ed012e-s6vzt" podStartSLOduration=1.765667842 podStartE2EDuration="1.765667842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:34.763330404 +0000 UTC m=+2357.389000371" watchObservedRunningTime="2026-10-30 19:15:34.765667842 +0000 UTC m=+2357.391337806"
Oct 30 19:15:35.805297 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:35.805263    4525 generic.go:356] "Generic (PLEG): container finished" podID="ef28c15d-d04d-4186-ad97-5c88f5f89795" containerID="6359945997b3c810f4b94f990bbb77e393376667352d1e1e40b853b9bd39ad6e" exitCode=0
Oct 30 19:15:35.807034 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:35.806147    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-ed012e-s6vzt" event={"ID":"ef28c15d-d04d-4186-ad97-5c88f5f89795","Type":"ContainerDied","Data":"6359945997b3c810f4b94f990bbb77e393376667352d1e1e40b853b9bd39ad6e"}
Oct 30 19:15:36.004342 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:36.004314    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c2e3f9-mgxxs"
Oct 30 19:15:36.108429 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:36.108394    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e80605e4-644b-42fc-b81d-3c1885a99496-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e80605e4-644b-42fc-b81d-3c1885a99496-maas-api-openshift-ca\") pod \"e80605e4-644b-42fc-b81d-3c1885a99496\" (UID: \"e80605e4-644b-42fc-b81d-3c1885a99496\") "
Oct 30 19:15:36.108614 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:36.108459    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e80605e4-644b-42fc-b81d-3c1885a99496-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e80605e4-644b-42fc-b81d-3c1885a99496-maas-api-xks-ca\") pod \"e80605e4-644b-42fc-b81d-3c1885a99496\" (UID: \"e80605e4-644b-42fc-b81d-3c1885a99496\") "
Oct 30 19:15:36.108614 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:36.108587    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e80605e4-644b-42fc-b81d-3c1885a99496-kube-api-access-4z4cg\" (UniqueName: \"kubernetes.io/projected/e80605e4-644b-42fc-b81d-3c1885a99496-kube-api-access-4z4cg\") pod \"e80605e4-644b-42fc-b81d-3c1885a99496\" (UID: \"e80605e4-644b-42fc-b81d-3c1885a99496\") "
Oct 30 19:15:36.108797 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:36.108770    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e80605e4-644b-42fc-b81d-3c1885a99496-maas-api-openshift-ca" pod "e80605e4-644b-42fc-b81d-3c1885a99496" (UID: "e80605e4-644b-42fc-b81d-3c1885a99496"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:36.109186 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:36.109170    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e80605e4-644b-42fc-b81d-3c1885a99496-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:36.110715 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:36.110694    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e80605e4-644b-42fc-b81d-3c1885a99496-maas-api-xks-ca" pod "e80605e4-644b-42fc-b81d-3c1885a99496" (UID: "e80605e4-644b-42fc-b81d-3c1885a99496"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:15:36.110877 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:36.110850    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e80605e4-644b-42fc-b81d-3c1885a99496-kube-api-access-4z4cg" pod "e80605e4-644b-42fc-b81d-3c1885a99496" (UID: "e80605e4-644b-42fc-b81d-3c1885a99496"). InnerVolumeSpecName "kube-api-access-4z4cg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:36.210525 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:36.210494    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e80605e4-644b-42fc-b81d-3c1885a99496-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:36.210525 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:36.210522    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4z4cg\" (UniqueName: \"kubernetes.io/projected/e80605e4-644b-42fc-b81d-3c1885a99496-kube-api-access-4z4cg\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:36.819869 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:36.819835    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c2e3f9-mgxxs"
Oct 30 19:15:36.821127 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:36.819845    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c2e3f9-mgxxs" event={"ID":"e80605e4-644b-42fc-b81d-3c1885a99496","Type":"ContainerDied","Data":"9193f86333e194c7cd36478e60f5eb5a93e9e71d681f5fa8829e57c56ded696f"}
Oct 30 19:15:36.821273 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:36.821155    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9193f86333e194c7cd36478e60f5eb5a93e9e71d681f5fa8829e57c56ded696f"
Oct 30 19:15:36.972671 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:36.972644    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-ed012e-s6vzt"
Oct 30 19:15:37.053624 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:37.053593    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef28c15d-d04d-4186-ad97-5c88f5f89795-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef28c15d-d04d-4186-ad97-5c88f5f89795-maas-api-openshift-ca\") pod \"ef28c15d-d04d-4186-ad97-5c88f5f89795\" (UID: \"ef28c15d-d04d-4186-ad97-5c88f5f89795\") "
Oct 30 19:15:37.053883 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:37.053671    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef28c15d-d04d-4186-ad97-5c88f5f89795-kube-api-access-hhp5q\" (UniqueName: \"kubernetes.io/projected/ef28c15d-d04d-4186-ad97-5c88f5f89795-kube-api-access-hhp5q\") pod \"ef28c15d-d04d-4186-ad97-5c88f5f89795\" (UID: \"ef28c15d-d04d-4186-ad97-5c88f5f89795\") "
Oct 30 19:15:37.053883 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:37.053764    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef28c15d-d04d-4186-ad97-5c88f5f89795-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef28c15d-d04d-4186-ad97-5c88f5f89795-maas-api-xks-ca\") pod \"ef28c15d-d04d-4186-ad97-5c88f5f89795\" (UID: \"ef28c15d-d04d-4186-ad97-5c88f5f89795\") "
Oct 30 19:15:37.054109 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:37.054080    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef28c15d-d04d-4186-ad97-5c88f5f89795-maas-api-openshift-ca" pod "ef28c15d-d04d-4186-ad97-5c88f5f89795" (UID: "ef28c15d-d04d-4186-ad97-5c88f5f89795"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:37.054593 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:37.054570    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef28c15d-d04d-4186-ad97-5c88f5f89795-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:37.056078 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:37.056047    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef28c15d-d04d-4186-ad97-5c88f5f89795-maas-api-xks-ca" pod "ef28c15d-d04d-4186-ad97-5c88f5f89795" (UID: "ef28c15d-d04d-4186-ad97-5c88f5f89795"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:15:37.056197 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:37.056171    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef28c15d-d04d-4186-ad97-5c88f5f89795-kube-api-access-hhp5q" pod "ef28c15d-d04d-4186-ad97-5c88f5f89795" (UID: "ef28c15d-d04d-4186-ad97-5c88f5f89795"). InnerVolumeSpecName "kube-api-access-hhp5q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:37.156137 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:37.156107    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhp5q\" (UniqueName: \"kubernetes.io/projected/ef28c15d-d04d-4186-ad97-5c88f5f89795-kube-api-access-hhp5q\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:37.156137 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:37.156135    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef28c15d-d04d-4186-ad97-5c88f5f89795-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:37.835390 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:37.835258    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-ed012e-s6vzt" event={"ID":"ef28c15d-d04d-4186-ad97-5c88f5f89795","Type":"ContainerDied","Data":"361f0de4b88665f2da10db3f97d2e6e1903ec8e85f150fa467527a1f72aa155a"}
Oct 30 19:15:37.835390 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:37.835335    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="361f0de4b88665f2da10db3f97d2e6e1903ec8e85f150fa467527a1f72aa155a"
Oct 30 19:15:37.836058 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:37.835424    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-ed012e-s6vzt"
Oct 30 19:15:38.062439 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:38.062395    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-5hdmh"]
Oct 30 19:15:38.063054 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:38.063017    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-5hdmh" podUID="f7e914c3-d828-407a-bafb-f71a07d872c1" containerName="istio-proxy" containerID="cri-o://6c6e31cf364ed20bf0047898b173cf5e5f6e37039f70c57f5d094d2449541a4e" gracePeriod=30
Oct 30 19:15:38.073040 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:38.073012    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-ftq7b"]
Oct 30 19:15:38.073247 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:38.073221    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-ftq7b" podUID="e4cbde58-0ec2-4f0d-af09-048853d07d63" containerName="istio-proxy" containerID="cri-o://aa10dce9b5a287001fe76c4462e484a4bd2813cb8c89327e64aa82ea0959d438" gracePeriod=30
Oct 30 19:15:38.082421 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:38.082395    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45"]
Oct 30 19:15:38.082645 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:38.082618    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45" podUID="d2f9a313-f069-4e48-8cdc-f75c6a285a7c" containerName="istio-proxy" containerID="cri-o://5b8c6c1e4d153d4c504d2beac49d7d788e072865b13d8580aa8a54e737d1558b" gracePeriod=30
Oct 30 19:15:38.093475 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:38.093382    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-6crc4"]
Oct 30 19:15:38.093662 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:38.093622    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-6crc4" podUID="f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419" containerName="istio-proxy" containerID="cri-o://8ee5451b361b14e17fa190056169c7ac412528760bf7e91138bce835ef7d6ccb" gracePeriod=30
Oct 30 19:15:38.099749 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:38.099667    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2cf46"]
Oct 30 19:15:38.099937 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:38.099911    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2cf46" podUID="52a25631-2cfc-4155-a3c3-4a381f96bf8b" containerName="istio-proxy" containerID="cri-o://b520c9f45cc994c8fe9071082fad3019e873bfaac64681f87b690ca7d3c8714d" gracePeriod=30
Oct 30 19:15:38.140373 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:38.140336    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-68b466-openshift-default-57cb5457fc-jgkhq"]
Oct 30 19:15:38.140662 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:38.140631    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-68b466-openshift-default-57cb5457fc-jgkhq" podUID="19f00f47-605d-4b19-9a1c-664de3661507" containerName="istio-proxy" containerID="cri-o://3f123adeefacee438ae8ecf734e1bd9a72367e41eb105cf70ca97d5e30425608" gracePeriod=30
Oct 30 19:15:38.669064 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:38.669031    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4a477d-k4l9b"]
Oct 30 19:15:38.669622 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:38.669608    4525 cpu_manager.go:414] "containerMap: removing container" podUID="e80605e4-644b-42fc-b81d-3c1885a99496" containerName="revoke-keys"
Oct 30 19:15:38.669700 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:38.669623    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e80605e4-644b-42fc-b81d-3c1885a99496" containerName="revoke-keys"
Oct 30 19:15:38.669700 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:38.669651    4525 cpu_manager.go:414] "containerMap: removing container" podUID="ef28c15d-d04d-4186-ad97-5c88f5f89795" containerName="revoke-keys"
Oct 30 19:15:38.669700 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:38.669658    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef28c15d-d04d-4186-ad97-5c88f5f89795" containerName="revoke-keys"
Oct 30 19:15:38.669867 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:38.669774    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef28c15d-d04d-4186-ad97-5c88f5f89795" containerName="revoke-keys"
Oct 30 19:15:38.669867 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:38.669787    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="e80605e4-644b-42fc-b81d-3c1885a99496" containerName="revoke-keys"
Oct 30 19:15:38.673617 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:38.673599    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4a477d-k4l9b"
Oct 30 19:15:38.684593 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:38.684567    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4a477d-k4l9b"]
Oct 30 19:15:38.766836 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:38.766807    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:15:38.767135 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:38.767113    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94hcn\""
Oct 30 19:15:38.777008 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:38.776984    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9de872-xcgq7"]
Oct 30 19:15:38.781268 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:38.781249    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9de872-xcgq7"
Oct 30 19:15:38.794630 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:38.794602    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9de872-xcgq7"]
Oct 30 19:15:38.800648 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:38.800597    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7kpw\" (UniqueName: \"kubernetes.io/projected/a8fc9310-cf03-4629-be6c-4d184a067d93-kube-api-access-p7kpw\") pod \"maas-api-revoke-keys-e2e-extmod-b-4a477d-k4l9b\" (UID: \"a8fc9310-cf03-4629-be6c-4d184a067d93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4a477d-k4l9b"
Oct 30 19:15:38.800930 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:38.800904    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8fc9310-cf03-4629-be6c-4d184a067d93-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-4a477d-k4l9b\" (UID: \"a8fc9310-cf03-4629-be6c-4d184a067d93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4a477d-k4l9b"
Oct 30 19:15:38.801122 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:38.801103    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8fc9310-cf03-4629-be6c-4d184a067d93-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-4a477d-k4l9b\" (UID: \"a8fc9310-cf03-4629-be6c-4d184a067d93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4a477d-k4l9b"
Oct 30 19:15:38.903082 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:38.903050    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8fc9310-cf03-4629-be6c-4d184a067d93-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-4a477d-k4l9b\" (UID: \"a8fc9310-cf03-4629-be6c-4d184a067d93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4a477d-k4l9b"
Oct 30 19:15:38.903467 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:38.903115    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8fc9310-cf03-4629-be6c-4d184a067d93-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-4a477d-k4l9b\" (UID: \"a8fc9310-cf03-4629-be6c-4d184a067d93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4a477d-k4l9b"
Oct 30 19:15:38.903467 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:38.903167    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs6v6\" (UniqueName: \"kubernetes.io/projected/5c830b89-50cc-46ea-909d-5f1127534794-kube-api-access-bs6v6\") pod \"maas-api-revoke-keys-e2e-worker-w0-9de872-xcgq7\" (UID: \"5c830b89-50cc-46ea-909d-5f1127534794\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9de872-xcgq7"
Oct 30 19:15:38.903467 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:38.903212    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c830b89-50cc-46ea-909d-5f1127534794-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-9de872-xcgq7\" (UID: \"5c830b89-50cc-46ea-909d-5f1127534794\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9de872-xcgq7"
Oct 30 19:15:38.903467 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:38.903377    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c830b89-50cc-46ea-909d-5f1127534794-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-9de872-xcgq7\" (UID: \"5c830b89-50cc-46ea-909d-5f1127534794\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9de872-xcgq7"
Oct 30 19:15:38.903641 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:38.903584    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7kpw\" (UniqueName: \"kubernetes.io/projected/a8fc9310-cf03-4629-be6c-4d184a067d93-kube-api-access-p7kpw\") pod \"maas-api-revoke-keys-e2e-extmod-b-4a477d-k4l9b\" (UID: \"a8fc9310-cf03-4629-be6c-4d184a067d93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4a477d-k4l9b"
Oct 30 19:15:38.903863 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:38.903839    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8fc9310-cf03-4629-be6c-4d184a067d93-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-4a477d-k4l9b\" (UID: \"a8fc9310-cf03-4629-be6c-4d184a067d93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4a477d-k4l9b"
Oct 30 19:15:38.905772 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:38.905755    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8fc9310-cf03-4629-be6c-4d184a067d93-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-4a477d-k4l9b\" (UID: \"a8fc9310-cf03-4629-be6c-4d184a067d93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4a477d-k4l9b"
Oct 30 19:15:38.916856 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:38.916836    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7kpw\" (UniqueName: \"kubernetes.io/projected/a8fc9310-cf03-4629-be6c-4d184a067d93-kube-api-access-p7kpw\") pod \"maas-api-revoke-keys-e2e-extmod-b-4a477d-k4l9b\" (UID: \"a8fc9310-cf03-4629-be6c-4d184a067d93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4a477d-k4l9b"
Oct 30 19:15:38.990925 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:38.990838    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4a477d-k4l9b"
Oct 30 19:15:39.006555 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:39.006527    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c830b89-50cc-46ea-909d-5f1127534794-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-9de872-xcgq7\" (UID: \"5c830b89-50cc-46ea-909d-5f1127534794\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9de872-xcgq7"
Oct 30 19:15:39.006745 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:39.006654    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c830b89-50cc-46ea-909d-5f1127534794-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-9de872-xcgq7\" (UID: \"5c830b89-50cc-46ea-909d-5f1127534794\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9de872-xcgq7"
Oct 30 19:15:39.006977 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:39.006948    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs6v6\" (UniqueName: \"kubernetes.io/projected/5c830b89-50cc-46ea-909d-5f1127534794-kube-api-access-bs6v6\") pod \"maas-api-revoke-keys-e2e-worker-w0-9de872-xcgq7\" (UID: \"5c830b89-50cc-46ea-909d-5f1127534794\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9de872-xcgq7"
Oct 30 19:15:39.007444 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:39.007423    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c830b89-50cc-46ea-909d-5f1127534794-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-9de872-xcgq7\" (UID: \"5c830b89-50cc-46ea-909d-5f1127534794\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9de872-xcgq7"
Oct 30 19:15:39.009554 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:39.009530    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c830b89-50cc-46ea-909d-5f1127534794-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-9de872-xcgq7\" (UID: \"5c830b89-50cc-46ea-909d-5f1127534794\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9de872-xcgq7"
Oct 30 19:15:39.015980 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:39.015958    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs6v6\" (UniqueName: \"kubernetes.io/projected/5c830b89-50cc-46ea-909d-5f1127534794-kube-api-access-bs6v6\") pod \"maas-api-revoke-keys-e2e-worker-w0-9de872-xcgq7\" (UID: \"5c830b89-50cc-46ea-909d-5f1127534794\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9de872-xcgq7"
Oct 30 19:15:39.098979 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:39.098943    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9de872-xcgq7"
Oct 30 19:15:39.274574 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:15:39.274538    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8fc9310_cf03_4629_be6c_4d184a067d93.slice/crio-36003e115bd42034e7987c361e8278069637d6f25c087368c52af51817f74322 WatchSource:0}: Error finding container 36003e115bd42034e7987c361e8278069637d6f25c087368c52af51817f74322: Status 404 returned error can't find the container with id 36003e115bd42034e7987c361e8278069637d6f25c087368c52af51817f74322
Oct 30 19:15:39.275891 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:39.275866    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4a477d-k4l9b"]
Oct 30 19:15:39.405621 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:15:39.405589    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c830b89_50cc_46ea_909d_5f1127534794.slice/crio-9f228d38076e00069c4e0bc66ddb4b99d5a2965cfe33cc146e0f2b42827cf80e WatchSource:0}: Error finding container 9f228d38076e00069c4e0bc66ddb4b99d5a2965cfe33cc146e0f2b42827cf80e: Status 404 returned error can't find the container with id 9f228d38076e00069c4e0bc66ddb4b99d5a2965cfe33cc146e0f2b42827cf80e
Oct 30 19:15:39.407497 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:39.407418    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9de872-xcgq7"]
Oct 30 19:15:39.865610 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:39.865569    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4a477d-k4l9b" event={"ID":"a8fc9310-cf03-4629-be6c-4d184a067d93","Type":"ContainerStarted","Data":"913b4c46e3614ffc5ed1b385cb23a909399c634a3a1aeb50c0be5118f0c2a920"}
Oct 30 19:15:39.865817 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:39.865637    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4a477d-k4l9b" event={"ID":"a8fc9310-cf03-4629-be6c-4d184a067d93","Type":"ContainerStarted","Data":"36003e115bd42034e7987c361e8278069637d6f25c087368c52af51817f74322"}
Oct 30 19:15:39.867159 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:39.867120    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9de872-xcgq7" event={"ID":"5c830b89-50cc-46ea-909d-5f1127534794","Type":"ContainerStarted","Data":"9f228d38076e00069c4e0bc66ddb4b99d5a2965cfe33cc146e0f2b42827cf80e"}
Oct 30 19:15:39.885971 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:39.885925    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4a477d-k4l9b" podStartSLOduration=1.885912447 podStartE2EDuration="1.885912447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:39.881992773 +0000 UTC m=+2362.507662736" watchObservedRunningTime="2026-10-30 19:15:39.885912447 +0000 UTC m=+2362.511582452"
Oct 30 19:15:40.878230 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:40.878193    4525 generic.go:356] "Generic (PLEG): container finished" podID="5c830b89-50cc-46ea-909d-5f1127534794" containerID="75ea052af6fbb52e18e6579e4ec40fb52fa9b8b8d37520d26bbc533640767541" exitCode=0
Oct 30 19:15:40.878701 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:40.878318    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9de872-xcgq7" event={"ID":"5c830b89-50cc-46ea-909d-5f1127534794","Type":"ContainerDied","Data":"75ea052af6fbb52e18e6579e4ec40fb52fa9b8b8d37520d26bbc533640767541"}
Oct 30 19:15:40.880416 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:40.880386    4525 generic.go:356] "Generic (PLEG): container finished" podID="a8fc9310-cf03-4629-be6c-4d184a067d93" containerID="913b4c46e3614ffc5ed1b385cb23a909399c634a3a1aeb50c0be5118f0c2a920" exitCode=0
Oct 30 19:15:40.880540 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:40.880449    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4a477d-k4l9b" event={"ID":"a8fc9310-cf03-4629-be6c-4d184a067d93","Type":"ContainerDied","Data":"913b4c46e3614ffc5ed1b385cb23a909399c634a3a1aeb50c0be5118f0c2a920"}
Oct 30 19:15:41.016753 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:41.016664    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-d6f956-openshift-default-56888545df-6crc4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:15:41.016952 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:41.016780    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-6crc4" podUID="f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:15:41.149486 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:41.149388    4525 patch_prober.go:29] interesting pod/e2e-extmod-a-d6f956-openshift-default-56888545df-2cf46 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:15:41.149486 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:41.149457    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2cf46" podUID="52a25631-2cfc-4155-a3c3-4a381f96bf8b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:15:42.070411 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:42.070382    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9de872-xcgq7"
Oct 30 19:15:42.118178 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:42.118151    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4a477d-k4l9b"
Oct 30 19:15:42.197458 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:42.197370    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c830b89-50cc-46ea-909d-5f1127534794-kube-api-access-bs6v6\" (UniqueName: \"kubernetes.io/projected/5c830b89-50cc-46ea-909d-5f1127534794-kube-api-access-bs6v6\") pod \"5c830b89-50cc-46ea-909d-5f1127534794\" (UID: \"5c830b89-50cc-46ea-909d-5f1127534794\") "
Oct 30 19:15:42.197666 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:42.197645    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c830b89-50cc-46ea-909d-5f1127534794-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c830b89-50cc-46ea-909d-5f1127534794-maas-api-xks-ca\") pod \"5c830b89-50cc-46ea-909d-5f1127534794\" (UID: \"5c830b89-50cc-46ea-909d-5f1127534794\") "
Oct 30 19:15:42.197758 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:42.197711    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c830b89-50cc-46ea-909d-5f1127534794-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c830b89-50cc-46ea-909d-5f1127534794-maas-api-openshift-ca\") pod \"5c830b89-50cc-46ea-909d-5f1127534794\" (UID: \"5c830b89-50cc-46ea-909d-5f1127534794\") "
Oct 30 19:15:42.198226 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:42.198189    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c830b89-50cc-46ea-909d-5f1127534794-maas-api-openshift-ca" pod "5c830b89-50cc-46ea-909d-5f1127534794" (UID: "5c830b89-50cc-46ea-909d-5f1127534794"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:42.198412 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:42.198394    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c830b89-50cc-46ea-909d-5f1127534794-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:42.199839 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:42.199809    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c830b89-50cc-46ea-909d-5f1127534794-kube-api-access-bs6v6" pod "5c830b89-50cc-46ea-909d-5f1127534794" (UID: "5c830b89-50cc-46ea-909d-5f1127534794"). InnerVolumeSpecName "kube-api-access-bs6v6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:42.199937 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:42.199881    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c830b89-50cc-46ea-909d-5f1127534794-maas-api-xks-ca" pod "5c830b89-50cc-46ea-909d-5f1127534794" (UID: "5c830b89-50cc-46ea-909d-5f1127534794"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:15:42.303557 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:42.301859    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8fc9310-cf03-4629-be6c-4d184a067d93-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8fc9310-cf03-4629-be6c-4d184a067d93-maas-api-xks-ca\") pod \"a8fc9310-cf03-4629-be6c-4d184a067d93\" (UID: \"a8fc9310-cf03-4629-be6c-4d184a067d93\") "
Oct 30 19:15:42.303557 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:42.302126    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8fc9310-cf03-4629-be6c-4d184a067d93-kube-api-access-p7kpw\" (UniqueName: \"kubernetes.io/projected/a8fc9310-cf03-4629-be6c-4d184a067d93-kube-api-access-p7kpw\") pod \"a8fc9310-cf03-4629-be6c-4d184a067d93\" (UID: \"a8fc9310-cf03-4629-be6c-4d184a067d93\") "
Oct 30 19:15:42.303557 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:42.302310    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8fc9310-cf03-4629-be6c-4d184a067d93-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8fc9310-cf03-4629-be6c-4d184a067d93-maas-api-openshift-ca\") pod \"a8fc9310-cf03-4629-be6c-4d184a067d93\" (UID: \"a8fc9310-cf03-4629-be6c-4d184a067d93\") "
Oct 30 19:15:42.303557 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:42.303127    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bs6v6\" (UniqueName: \"kubernetes.io/projected/5c830b89-50cc-46ea-909d-5f1127534794-kube-api-access-bs6v6\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:42.303557 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:42.303154    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c830b89-50cc-46ea-909d-5f1127534794-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:42.310508 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:42.309940    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8fc9310-cf03-4629-be6c-4d184a067d93-maas-api-openshift-ca" pod "a8fc9310-cf03-4629-be6c-4d184a067d93" (UID: "a8fc9310-cf03-4629-be6c-4d184a067d93"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:42.313440 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:42.312611    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8fc9310-cf03-4629-be6c-4d184a067d93-maas-api-xks-ca" pod "a8fc9310-cf03-4629-be6c-4d184a067d93" (UID: "a8fc9310-cf03-4629-be6c-4d184a067d93"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:15:42.316975 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:42.316929    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8fc9310-cf03-4629-be6c-4d184a067d93-kube-api-access-p7kpw" pod "a8fc9310-cf03-4629-be6c-4d184a067d93" (UID: "a8fc9310-cf03-4629-be6c-4d184a067d93"). InnerVolumeSpecName "kube-api-access-p7kpw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:42.404718 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:42.404675    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7kpw\" (UniqueName: \"kubernetes.io/projected/a8fc9310-cf03-4629-be6c-4d184a067d93-kube-api-access-p7kpw\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:42.404718 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:42.404717    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8fc9310-cf03-4629-be6c-4d184a067d93-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:42.404943 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:42.404728    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8fc9310-cf03-4629-be6c-4d184a067d93-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:42.899041 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:42.899008    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9de872-xcgq7"
Oct 30 19:15:42.899223 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:42.899047    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9de872-xcgq7" event={"ID":"5c830b89-50cc-46ea-909d-5f1127534794","Type":"ContainerDied","Data":"9f228d38076e00069c4e0bc66ddb4b99d5a2965cfe33cc146e0f2b42827cf80e"}
Oct 30 19:15:42.899223 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:42.899098    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f228d38076e00069c4e0bc66ddb4b99d5a2965cfe33cc146e0f2b42827cf80e"
Oct 30 19:15:42.900812 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:42.900792    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4a477d-k4l9b"
Oct 30 19:15:42.900943 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:42.900815    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4a477d-k4l9b" event={"ID":"a8fc9310-cf03-4629-be6c-4d184a067d93","Type":"ContainerDied","Data":"36003e115bd42034e7987c361e8278069637d6f25c087368c52af51817f74322"}
Oct 30 19:15:42.900943 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:42.900859    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36003e115bd42034e7987c361e8278069637d6f25c087368c52af51817f74322"
Oct 30 19:15:43.729817 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.719087    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-5hdmh"
Oct 30 19:15:43.853445 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.852528    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f7e914c3-d828-407a-bafb-f71a07d872c1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7e914c3-d828-407a-bafb-f71a07d872c1-istio-podinfo\") pod \"f7e914c3-d828-407a-bafb-f71a07d872c1\" (UID: \"f7e914c3-d828-407a-bafb-f71a07d872c1\") "
Oct 30 19:15:43.853445 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.852605    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7e914c3-d828-407a-bafb-f71a07d872c1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7e914c3-d828-407a-bafb-f71a07d872c1-istiod-ca-cert\") pod \"f7e914c3-d828-407a-bafb-f71a07d872c1\" (UID: \"f7e914c3-d828-407a-bafb-f71a07d872c1\") "
Oct 30 19:15:43.853445 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.852716    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7e914c3-d828-407a-bafb-f71a07d872c1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7e914c3-d828-407a-bafb-f71a07d872c1-istio-data\") pod \"f7e914c3-d828-407a-bafb-f71a07d872c1\" (UID: \"f7e914c3-d828-407a-bafb-f71a07d872c1\") "
Oct 30 19:15:43.861171 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.860150    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f7e914c3-d828-407a-bafb-f71a07d872c1-istio-podinfo" pod "f7e914c3-d828-407a-bafb-f71a07d872c1" (UID: "f7e914c3-d828-407a-bafb-f71a07d872c1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:15:43.863194 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.863145    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7e914c3-d828-407a-bafb-f71a07d872c1-istiod-ca-cert" pod "f7e914c3-d828-407a-bafb-f71a07d872c1" (UID: "f7e914c3-d828-407a-bafb-f71a07d872c1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:43.866652 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.864911    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7e914c3-d828-407a-bafb-f71a07d872c1-istio-data" pod "f7e914c3-d828-407a-bafb-f71a07d872c1" (UID: "f7e914c3-d828-407a-bafb-f71a07d872c1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:43.869021 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.868148    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7e914c3-d828-407a-bafb-f71a07d872c1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7e914c3-d828-407a-bafb-f71a07d872c1-workload-certs\") pod \"f7e914c3-d828-407a-bafb-f71a07d872c1\" (UID: \"f7e914c3-d828-407a-bafb-f71a07d872c1\") "
Oct 30 19:15:43.869021 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.868241    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7e914c3-d828-407a-bafb-f71a07d872c1-kube-api-access-8klf6\" (UniqueName: \"kubernetes.io/projected/f7e914c3-d828-407a-bafb-f71a07d872c1-kube-api-access-8klf6\") pod \"f7e914c3-d828-407a-bafb-f71a07d872c1\" (UID: \"f7e914c3-d828-407a-bafb-f71a07d872c1\") "
Oct 30 19:15:43.869021 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.868337    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7e914c3-d828-407a-bafb-f71a07d872c1-istio-token\" (UniqueName: \"kubernetes.io/projected/f7e914c3-d828-407a-bafb-f71a07d872c1-istio-token\") pod \"f7e914c3-d828-407a-bafb-f71a07d872c1\" (UID: \"f7e914c3-d828-407a-bafb-f71a07d872c1\") "
Oct 30 19:15:43.869021 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.868408    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7e914c3-d828-407a-bafb-f71a07d872c1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7e914c3-d828-407a-bafb-f71a07d872c1-credential-socket\") pod \"f7e914c3-d828-407a-bafb-f71a07d872c1\" (UID: \"f7e914c3-d828-407a-bafb-f71a07d872c1\") "
Oct 30 19:15:43.869021 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.868579    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7e914c3-d828-407a-bafb-f71a07d872c1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7e914c3-d828-407a-bafb-f71a07d872c1-istio-envoy\") pod \"f7e914c3-d828-407a-bafb-f71a07d872c1\" (UID: \"f7e914c3-d828-407a-bafb-f71a07d872c1\") "
Oct 30 19:15:43.869021 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.868725    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7e914c3-d828-407a-bafb-f71a07d872c1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7e914c3-d828-407a-bafb-f71a07d872c1-workload-socket\") pod \"f7e914c3-d828-407a-bafb-f71a07d872c1\" (UID: \"f7e914c3-d828-407a-bafb-f71a07d872c1\") "
Oct 30 19:15:43.869523 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.869500    4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7e914c3-d828-407a-bafb-f71a07d872c1-istio-podinfo\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:43.869584 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.869527    4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7e914c3-d828-407a-bafb-f71a07d872c1-istiod-ca-cert\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:43.869584 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.869538    4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7e914c3-d828-407a-bafb-f71a07d872c1-istio-data\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:43.869950 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.869929    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7e914c3-d828-407a-bafb-f71a07d872c1-workload-certs" pod "f7e914c3-d828-407a-bafb-f71a07d872c1" (UID: "f7e914c3-d828-407a-bafb-f71a07d872c1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:43.870816 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.870174    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7e914c3-d828-407a-bafb-f71a07d872c1-workload-socket" pod "f7e914c3-d828-407a-bafb-f71a07d872c1" (UID: "f7e914c3-d828-407a-bafb-f71a07d872c1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:43.871303 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.871273    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7e914c3-d828-407a-bafb-f71a07d872c1-credential-socket" pod "f7e914c3-d828-407a-bafb-f71a07d872c1" (UID: "f7e914c3-d828-407a-bafb-f71a07d872c1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:43.880075 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.880027    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7e914c3-d828-407a-bafb-f71a07d872c1-kube-api-access-8klf6" pod "f7e914c3-d828-407a-bafb-f71a07d872c1" (UID: "f7e914c3-d828-407a-bafb-f71a07d872c1"). InnerVolumeSpecName "kube-api-access-8klf6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:43.900333 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.900285    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7e914c3-d828-407a-bafb-f71a07d872c1-istio-token" pod "f7e914c3-d828-407a-bafb-f71a07d872c1" (UID: "f7e914c3-d828-407a-bafb-f71a07d872c1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:43.902234 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.902203    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7e914c3-d828-407a-bafb-f71a07d872c1-istio-envoy" pod "f7e914c3-d828-407a-bafb-f71a07d872c1" (UID: "f7e914c3-d828-407a-bafb-f71a07d872c1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:43.908621 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.908110    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-6crc4"
Oct 30 19:15:43.914904 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.914223    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45"
Oct 30 19:15:43.919768 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.919405    4525 generic.go:356] "Generic (PLEG): container finished" podID="f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419" containerID="8ee5451b361b14e17fa190056169c7ac412528760bf7e91138bce835ef7d6ccb" exitCode=0
Oct 30 19:15:43.919768 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.919449    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-6crc4"
Oct 30 19:15:43.919768 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.919539    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-6crc4" event={"ID":"f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419","Type":"ContainerDied","Data":"8ee5451b361b14e17fa190056169c7ac412528760bf7e91138bce835ef7d6ccb"}
Oct 30 19:15:43.919768 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.919421    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-ftq7b"
Oct 30 19:15:43.919768 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.919590    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-6crc4" event={"ID":"f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419","Type":"ContainerDied","Data":"31b202142edd53065be9f6c0dfa3de2b7ef07bb363b0252d672140080691cbe0"}
Oct 30 19:15:43.919768 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.919621    4525 scope.go:122] "RemoveContainer" containerID="8ee5451b361b14e17fa190056169c7ac412528760bf7e91138bce835ef7d6ccb"
Oct 30 19:15:43.923585 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.923386    4525 generic.go:356] "Generic (PLEG): container finished" podID="19f00f47-605d-4b19-9a1c-664de3661507" containerID="3f123adeefacee438ae8ecf734e1bd9a72367e41eb105cf70ca97d5e30425608" exitCode=0
Oct 30 19:15:43.923585 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.923515    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-68b466-openshift-default-57cb5457fc-jgkhq" event={"ID":"19f00f47-605d-4b19-9a1c-664de3661507","Type":"ContainerDied","Data":"3f123adeefacee438ae8ecf734e1bd9a72367e41eb105cf70ca97d5e30425608"}
Oct 30 19:15:43.925950 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.925530    4525 generic.go:356] "Generic (PLEG): container finished" podID="e4cbde58-0ec2-4f0d-af09-048853d07d63" containerID="aa10dce9b5a287001fe76c4462e484a4bd2813cb8c89327e64aa82ea0959d438" exitCode=0
Oct 30 19:15:43.925950 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.925624    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-ftq7b" event={"ID":"e4cbde58-0ec2-4f0d-af09-048853d07d63","Type":"ContainerDied","Data":"aa10dce9b5a287001fe76c4462e484a4bd2813cb8c89327e64aa82ea0959d438"}
Oct 30 19:15:43.925950 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.925663    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-ftq7b" event={"ID":"e4cbde58-0ec2-4f0d-af09-048853d07d63","Type":"ContainerDied","Data":"2be7b29c1633f3d6b149e3984606eb558ec95e9083ff506e07ca312cf972b629"}
Oct 30 19:15:43.925950 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.925763    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-ftq7b"
Oct 30 19:15:43.930285 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.929666    4525 generic.go:356] "Generic (PLEG): container finished" podID="f7e914c3-d828-407a-bafb-f71a07d872c1" containerID="6c6e31cf364ed20bf0047898b173cf5e5f6e37039f70c57f5d094d2449541a4e" exitCode=0
Oct 30 19:15:43.931024 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.930453    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-5hdmh" event={"ID":"f7e914c3-d828-407a-bafb-f71a07d872c1","Type":"ContainerDied","Data":"6c6e31cf364ed20bf0047898b173cf5e5f6e37039f70c57f5d094d2449541a4e"}
Oct 30 19:15:43.931024 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.930494    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-5hdmh" event={"ID":"f7e914c3-d828-407a-bafb-f71a07d872c1","Type":"ContainerDied","Data":"43144da64b6a555285cc325fad5d4d5bfb434ddd04144d8545afc3477cdd802e"}
Oct 30 19:15:43.931024 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.930548    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-5hdmh"
Oct 30 19:15:43.945040 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.944319    4525 scope.go:122] "RemoveContainer" containerID="8ee5451b361b14e17fa190056169c7ac412528760bf7e91138bce835ef7d6ccb"
Oct 30 19:15:43.945040 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:15:43.944762    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ee5451b361b14e17fa190056169c7ac412528760bf7e91138bce835ef7d6ccb\": container with ID starting with 8ee5451b361b14e17fa190056169c7ac412528760bf7e91138bce835ef7d6ccb not found: ID does not exist" containerID="8ee5451b361b14e17fa190056169c7ac412528760bf7e91138bce835ef7d6ccb"
Oct 30 19:15:43.945040 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.944803    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ee5451b361b14e17fa190056169c7ac412528760bf7e91138bce835ef7d6ccb"} err="failed to get container status \"8ee5451b361b14e17fa190056169c7ac412528760bf7e91138bce835ef7d6ccb\": rpc error: code = NotFound desc = could not find container \"8ee5451b361b14e17fa190056169c7ac412528760bf7e91138bce835ef7d6ccb\": container with ID starting with 8ee5451b361b14e17fa190056169c7ac412528760bf7e91138bce835ef7d6ccb not found: ID does not exist"
Oct 30 19:15:43.945040 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.944826    4525 scope.go:122] "RemoveContainer" containerID="05032f3ae4130fe35addd2d7e6c00f8ff2d86f4825b9a8764db98d5829ad5434"
Oct 30 19:15:43.947075 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.947046    4525 generic.go:356] "Generic (PLEG): container finished" podID="d2f9a313-f069-4e48-8cdc-f75c6a285a7c" containerID="5b8c6c1e4d153d4c504d2beac49d7d788e072865b13d8580aa8a54e737d1558b" exitCode=0
Oct 30 19:15:43.947198 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.947178    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45" event={"ID":"d2f9a313-f069-4e48-8cdc-f75c6a285a7c","Type":"ContainerDied","Data":"5b8c6c1e4d153d4c504d2beac49d7d788e072865b13d8580aa8a54e737d1558b"}
Oct 30 19:15:43.947302 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.947230    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45" event={"ID":"d2f9a313-f069-4e48-8cdc-f75c6a285a7c","Type":"ContainerDied","Data":"2bd83764225aa9567618eb597150f429110d4ce37258e87d8c1b0fde256819e7"}
Oct 30 19:15:43.947401 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.947314    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45"
Oct 30 19:15:43.960190 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.960136    4525 generic.go:356] "Generic (PLEG): container finished" podID="52a25631-2cfc-4155-a3c3-4a381f96bf8b" containerID="b520c9f45cc994c8fe9071082fad3019e873bfaac64681f87b690ca7d3c8714d" exitCode=0
Oct 30 19:15:43.960440 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.960218    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2cf46" event={"ID":"52a25631-2cfc-4155-a3c3-4a381f96bf8b","Type":"ContainerDied","Data":"b520c9f45cc994c8fe9071082fad3019e873bfaac64681f87b690ca7d3c8714d"}
Oct 30 19:15:43.970331 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.970303    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4cbde58-0ec2-4f0d-af09-048853d07d63-istio-token\" (UniqueName: \"kubernetes.io/projected/e4cbde58-0ec2-4f0d-af09-048853d07d63-istio-token\") pod \"e4cbde58-0ec2-4f0d-af09-048853d07d63\" (UID: \"e4cbde58-0ec2-4f0d-af09-048853d07d63\") "
Oct 30 19:15:43.970441 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.970355    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2f9a313-f069-4e48-8cdc-f75c6a285a7c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2f9a313-f069-4e48-8cdc-f75c6a285a7c-istiod-ca-cert\") pod \"d2f9a313-f069-4e48-8cdc-f75c6a285a7c\" (UID: \"d2f9a313-f069-4e48-8cdc-f75c6a285a7c\") "
Oct 30 19:15:43.970441 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.970393    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2f9a313-f069-4e48-8cdc-f75c6a285a7c-kube-api-access-m8mst\" (UniqueName: \"kubernetes.io/projected/d2f9a313-f069-4e48-8cdc-f75c6a285a7c-kube-api-access-m8mst\") pod \"d2f9a313-f069-4e48-8cdc-f75c6a285a7c\" (UID: \"d2f9a313-f069-4e48-8cdc-f75c6a285a7c\") "
Oct 30 19:15:43.970534 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.970447    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2f9a313-f069-4e48-8cdc-f75c6a285a7c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2f9a313-f069-4e48-8cdc-f75c6a285a7c-workload-certs\") pod \"d2f9a313-f069-4e48-8cdc-f75c6a285a7c\" (UID: \"d2f9a313-f069-4e48-8cdc-f75c6a285a7c\") "
Oct 30 19:15:43.970534 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.970504    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d2f9a313-f069-4e48-8cdc-f75c6a285a7c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2f9a313-f069-4e48-8cdc-f75c6a285a7c-istio-podinfo\") pod \"d2f9a313-f069-4e48-8cdc-f75c6a285a7c\" (UID: \"d2f9a313-f069-4e48-8cdc-f75c6a285a7c\") "
Oct 30 19:15:43.970606 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.970533    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419-istiod-ca-cert\") pod \"f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419\" (UID: \"f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419\") "
Oct 30 19:15:43.970606 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.970588    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419-kube-api-access-zcr65\" (UniqueName: \"kubernetes.io/projected/f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419-kube-api-access-zcr65\") pod \"f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419\" (UID: \"f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419\") "
Oct 30 19:15:43.970701 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.970627    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4cbde58-0ec2-4f0d-af09-048853d07d63-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4cbde58-0ec2-4f0d-af09-048853d07d63-istio-data\") pod \"e4cbde58-0ec2-4f0d-af09-048853d07d63\" (UID: \"e4cbde58-0ec2-4f0d-af09-048853d07d63\") "
Oct 30 19:15:43.970701 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.970664    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4cbde58-0ec2-4f0d-af09-048853d07d63-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4cbde58-0ec2-4f0d-af09-048853d07d63-workload-socket\") pod \"e4cbde58-0ec2-4f0d-af09-048853d07d63\" (UID: \"e4cbde58-0ec2-4f0d-af09-048853d07d63\") "
Oct 30 19:15:43.971185 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.971160    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-68b466-openshift-default-57cb5457fc-jgkhq"
Oct 30 19:15:43.972376 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.972355    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2f9a313-f069-4e48-8cdc-f75c6a285a7c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2f9a313-f069-4e48-8cdc-f75c6a285a7c-credential-socket\") pod \"d2f9a313-f069-4e48-8cdc-f75c6a285a7c\" (UID: \"d2f9a313-f069-4e48-8cdc-f75c6a285a7c\") "
Oct 30 19:15:43.972553 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.972516    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4cbde58-0ec2-4f0d-af09-048853d07d63-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4cbde58-0ec2-4f0d-af09-048853d07d63-workload-certs\") pod \"e4cbde58-0ec2-4f0d-af09-048853d07d63\" (UID: \"e4cbde58-0ec2-4f0d-af09-048853d07d63\") "
Oct 30 19:15:43.972804 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.972782    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2f9a313-f069-4e48-8cdc-f75c6a285a7c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2f9a313-f069-4e48-8cdc-f75c6a285a7c-istio-envoy\") pod \"d2f9a313-f069-4e48-8cdc-f75c6a285a7c\" (UID: \"d2f9a313-f069-4e48-8cdc-f75c6a285a7c\") "
Oct 30 19:15:43.972913 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.972887    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2f9a313-f069-4e48-8cdc-f75c6a285a7c-istiod-ca-cert" pod "d2f9a313-f069-4e48-8cdc-f75c6a285a7c" (UID: "d2f9a313-f069-4e48-8cdc-f75c6a285a7c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:43.973051 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.973033    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419-istio-token\" (UniqueName: \"kubernetes.io/projected/f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419-istio-token\") pod \"f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419\" (UID: \"f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419\") "
Oct 30 19:15:43.973133 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.973085    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419-istio-podinfo\") pod \"f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419\" (UID: \"f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419\") "
Oct 30 19:15:43.973190 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.973173    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419-istio-data\") pod \"f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419\" (UID: \"f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419\") "
Oct 30 19:15:43.973255 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.973225    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e4cbde58-0ec2-4f0d-af09-048853d07d63-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4cbde58-0ec2-4f0d-af09-048853d07d63-istio-podinfo\") pod \"e4cbde58-0ec2-4f0d-af09-048853d07d63\" (UID: \"e4cbde58-0ec2-4f0d-af09-048853d07d63\") "
Oct 30 19:15:43.973350 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.973293    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4cbde58-0ec2-4f0d-af09-048853d07d63-kube-api-access-ks47v\" (UniqueName: \"kubernetes.io/projected/e4cbde58-0ec2-4f0d-af09-048853d07d63-kube-api-access-ks47v\") pod \"e4cbde58-0ec2-4f0d-af09-048853d07d63\" (UID: \"e4cbde58-0ec2-4f0d-af09-048853d07d63\") "
Oct 30 19:15:43.973413 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.973351    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2f9a313-f069-4e48-8cdc-f75c6a285a7c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2f9a313-f069-4e48-8cdc-f75c6a285a7c-istio-data\") pod \"d2f9a313-f069-4e48-8cdc-f75c6a285a7c\" (UID: \"d2f9a313-f069-4e48-8cdc-f75c6a285a7c\") "
Oct 30 19:15:43.973475 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.973418    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4cbde58-0ec2-4f0d-af09-048853d07d63-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4cbde58-0ec2-4f0d-af09-048853d07d63-istiod-ca-cert\") pod \"e4cbde58-0ec2-4f0d-af09-048853d07d63\" (UID: \"e4cbde58-0ec2-4f0d-af09-048853d07d63\") "
Oct 30 19:15:43.973539 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.973470    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4cbde58-0ec2-4f0d-af09-048853d07d63-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4cbde58-0ec2-4f0d-af09-048853d07d63-istio-envoy\") pod \"e4cbde58-0ec2-4f0d-af09-048853d07d63\" (UID: \"e4cbde58-0ec2-4f0d-af09-048853d07d63\") "
Oct 30 19:15:43.973539 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.973509    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419-workload-certs\") pod \"f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419\" (UID: \"f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419\") "
Oct 30 19:15:43.973640 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.973594    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419-credential-socket\") pod \"f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419\" (UID: \"f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419\") "
Oct 30 19:15:43.973640 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.973625    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4cbde58-0ec2-4f0d-af09-048853d07d63-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4cbde58-0ec2-4f0d-af09-048853d07d63-credential-socket\") pod \"e4cbde58-0ec2-4f0d-af09-048853d07d63\" (UID: \"e4cbde58-0ec2-4f0d-af09-048853d07d63\") "
Oct 30 19:15:43.973927 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.973759    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419-istio-envoy\") pod \"f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419\" (UID: \"f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419\") "
Oct 30 19:15:43.973927 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.973818    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2f9a313-f069-4e48-8cdc-f75c6a285a7c-istio-token\" (UniqueName: \"kubernetes.io/projected/d2f9a313-f069-4e48-8cdc-f75c6a285a7c-istio-token\") pod \"d2f9a313-f069-4e48-8cdc-f75c6a285a7c\" (UID: \"d2f9a313-f069-4e48-8cdc-f75c6a285a7c\") "
Oct 30 19:15:43.973927 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.973858    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419-workload-socket\") pod \"f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419\" (UID: \"f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419\") "
Oct 30 19:15:43.973927 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.973923    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2f9a313-f069-4e48-8cdc-f75c6a285a7c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2f9a313-f069-4e48-8cdc-f75c6a285a7c-workload-socket\") pod \"d2f9a313-f069-4e48-8cdc-f75c6a285a7c\" (UID: \"d2f9a313-f069-4e48-8cdc-f75c6a285a7c\") "
Oct 30 19:15:43.975243 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.974327    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419-istiod-ca-cert" pod "f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419" (UID: "f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:43.975443 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.975412    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d2f9a313-f069-4e48-8cdc-f75c6a285a7c-istio-podinfo" pod "d2f9a313-f069-4e48-8cdc-f75c6a285a7c" (UID: "d2f9a313-f069-4e48-8cdc-f75c6a285a7c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:15:43.976213 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.976176    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4cbde58-0ec2-4f0d-af09-048853d07d63-istio-token" pod "e4cbde58-0ec2-4f0d-af09-048853d07d63" (UID: "e4cbde58-0ec2-4f0d-af09-048853d07d63"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:43.976213 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.976191    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4cbde58-0ec2-4f0d-af09-048853d07d63-istiod-ca-cert" pod "e4cbde58-0ec2-4f0d-af09-048853d07d63" (UID: "e4cbde58-0ec2-4f0d-af09-048853d07d63"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:43.976497 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.976445    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2f9a313-f069-4e48-8cdc-f75c6a285a7c-istio-data" pod "d2f9a313-f069-4e48-8cdc-f75c6a285a7c" (UID: "d2f9a313-f069-4e48-8cdc-f75c6a285a7c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:43.977660 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.977074    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419-workload-certs" pod "f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419" (UID: "f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:43.977660 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.977331    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419-workload-socket" pod "f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419" (UID: "f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:43.977660 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.977399    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4cbde58-0ec2-4f0d-af09-048853d07d63-credential-socket" pod "e4cbde58-0ec2-4f0d-af09-048853d07d63" (UID: "e4cbde58-0ec2-4f0d-af09-048853d07d63"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:43.980945 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.980278    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419-credential-socket" pod "f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419" (UID: "f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:43.980945 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.980817    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2f9a313-f069-4e48-8cdc-f75c6a285a7c-workload-socket" pod "d2f9a313-f069-4e48-8cdc-f75c6a285a7c" (UID: "d2f9a313-f069-4e48-8cdc-f75c6a285a7c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:43.982672 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.981167    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4cbde58-0ec2-4f0d-af09-048853d07d63-workload-socket" pod "e4cbde58-0ec2-4f0d-af09-048853d07d63" (UID: "e4cbde58-0ec2-4f0d-af09-048853d07d63"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:43.982672 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.981218    4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7e914c3-d828-407a-bafb-f71a07d872c1-istio-token\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:43.982672 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.981242    4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7e914c3-d828-407a-bafb-f71a07d872c1-credential-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:43.982672 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.981259    4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419-workload-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:43.982672 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.981273    4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2f9a313-f069-4e48-8cdc-f75c6a285a7c-workload-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:43.982672 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.981287    4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4cbde58-0ec2-4f0d-af09-048853d07d63-istio-token\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:43.982672 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.981302    4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7e914c3-d828-407a-bafb-f71a07d872c1-istio-envoy\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:43.982672 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.981316    4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2f9a313-f069-4e48-8cdc-f75c6a285a7c-istiod-ca-cert\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:43.982672 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.981329    4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2f9a313-f069-4e48-8cdc-f75c6a285a7c-istio-podinfo\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:43.982672 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.981345    4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7e914c3-d828-407a-bafb-f71a07d872c1-workload-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:43.982672 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.981365    4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419-istiod-ca-cert\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:43.982672 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.981398    4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2f9a313-f069-4e48-8cdc-f75c6a285a7c-istio-data\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:43.982672 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.981419    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4cbde58-0ec2-4f0d-af09-048853d07d63-istio-data" pod "e4cbde58-0ec2-4f0d-af09-048853d07d63" (UID: "e4cbde58-0ec2-4f0d-af09-048853d07d63"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:43.982672 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.981586    4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4cbde58-0ec2-4f0d-af09-048853d07d63-istiod-ca-cert\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:43.982672 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.981608    4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7e914c3-d828-407a-bafb-f71a07d872c1-workload-certs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:43.982672 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.981623    4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419-workload-certs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:43.982672 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.981638    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8klf6\" (UniqueName: \"kubernetes.io/projected/f7e914c3-d828-407a-bafb-f71a07d872c1-kube-api-access-8klf6\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:43.982672 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.981656    4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419-credential-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:43.982672 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.981671    4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4cbde58-0ec2-4f0d-af09-048853d07d63-credential-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:43.982672 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.981747    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419-istio-data" pod "f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419" (UID: "f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:43.982672 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.981857    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4cbde58-0ec2-4f0d-af09-048853d07d63-workload-certs" pod "e4cbde58-0ec2-4f0d-af09-048853d07d63" (UID: "e4cbde58-0ec2-4f0d-af09-048853d07d63"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:43.982672 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.982516    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4cbde58-0ec2-4f0d-af09-048853d07d63-istio-envoy" pod "e4cbde58-0ec2-4f0d-af09-048853d07d63" (UID: "e4cbde58-0ec2-4f0d-af09-048853d07d63"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:43.982672 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.982655    4525 scope.go:122] "RemoveContainer" containerID="aa10dce9b5a287001fe76c4462e484a4bd2813cb8c89327e64aa82ea0959d438"
Oct 30 19:15:43.985191 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.982708    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2f9a313-f069-4e48-8cdc-f75c6a285a7c-kube-api-access-m8mst" pod "d2f9a313-f069-4e48-8cdc-f75c6a285a7c" (UID: "d2f9a313-f069-4e48-8cdc-f75c6a285a7c"). InnerVolumeSpecName "kube-api-access-m8mst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:43.985191 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.982780    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2f9a313-f069-4e48-8cdc-f75c6a285a7c-istio-token" pod "d2f9a313-f069-4e48-8cdc-f75c6a285a7c" (UID: "d2f9a313-f069-4e48-8cdc-f75c6a285a7c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:43.985191 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.983330    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2f9a313-f069-4e48-8cdc-f75c6a285a7c-istio-envoy" pod "d2f9a313-f069-4e48-8cdc-f75c6a285a7c" (UID: "d2f9a313-f069-4e48-8cdc-f75c6a285a7c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:43.985191 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.983805    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2f9a313-f069-4e48-8cdc-f75c6a285a7c-credential-socket" pod "d2f9a313-f069-4e48-8cdc-f75c6a285a7c" (UID: "d2f9a313-f069-4e48-8cdc-f75c6a285a7c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:43.985191 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:43.984592    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2f9a313-f069-4e48-8cdc-f75c6a285a7c-workload-certs" pod "d2f9a313-f069-4e48-8cdc-f75c6a285a7c" (UID: "d2f9a313-f069-4e48-8cdc-f75c6a285a7c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:44.009705 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.008966    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419-kube-api-access-zcr65" pod "f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419" (UID: "f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419"). InnerVolumeSpecName "kube-api-access-zcr65". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:44.009705 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.009098    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419-istio-token" pod "f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419" (UID: "f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:44.009705 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.009158    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419-istio-podinfo" pod "f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419" (UID: "f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:15:44.038247 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.016013    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4cbde58-0ec2-4f0d-af09-048853d07d63-kube-api-access-ks47v" pod "e4cbde58-0ec2-4f0d-af09-048853d07d63" (UID: "e4cbde58-0ec2-4f0d-af09-048853d07d63"). InnerVolumeSpecName "kube-api-access-ks47v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:44.038247 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.016127    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e4cbde58-0ec2-4f0d-af09-048853d07d63-istio-podinfo" pod "e4cbde58-0ec2-4f0d-af09-048853d07d63" (UID: "e4cbde58-0ec2-4f0d-af09-048853d07d63"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:15:44.038247 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.022850    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2cf46"
Oct 30 19:15:44.087733 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.081149    4525 scope.go:122] "RemoveContainer" containerID="aa10dce9b5a287001fe76c4462e484a4bd2813cb8c89327e64aa82ea0959d438"
Oct 30 19:15:44.087733 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.081143    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419-istio-envoy" pod "f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419" (UID: "f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:44.087733 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.083773    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52a25631-2cfc-4155-a3c3-4a381f96bf8b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52a25631-2cfc-4155-a3c3-4a381f96bf8b-workload-socket\") pod \"52a25631-2cfc-4155-a3c3-4a381f96bf8b\" (UID: \"52a25631-2cfc-4155-a3c3-4a381f96bf8b\") "
Oct 30 19:15:44.087733 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.083845    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52a25631-2cfc-4155-a3c3-4a381f96bf8b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52a25631-2cfc-4155-a3c3-4a381f96bf8b-istiod-ca-cert\") pod \"52a25631-2cfc-4155-a3c3-4a381f96bf8b\" (UID: \"52a25631-2cfc-4155-a3c3-4a381f96bf8b\") "
Oct 30 19:15:44.087733 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.083886    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19f00f47-605d-4b19-9a1c-664de3661507-istio-token\" (UniqueName: \"kubernetes.io/projected/19f00f47-605d-4b19-9a1c-664de3661507-istio-token\") pod \"19f00f47-605d-4b19-9a1c-664de3661507\" (UID: \"19f00f47-605d-4b19-9a1c-664de3661507\") "
Oct 30 19:15:44.087733 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.083910    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52a25631-2cfc-4155-a3c3-4a381f96bf8b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52a25631-2cfc-4155-a3c3-4a381f96bf8b-istio-data\") pod \"52a25631-2cfc-4155-a3c3-4a381f96bf8b\" (UID: \"52a25631-2cfc-4155-a3c3-4a381f96bf8b\") "
Oct 30 19:15:44.087733 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.083932    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52a25631-2cfc-4155-a3c3-4a381f96bf8b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52a25631-2cfc-4155-a3c3-4a381f96bf8b-credential-socket\") pod \"52a25631-2cfc-4155-a3c3-4a381f96bf8b\" (UID: \"52a25631-2cfc-4155-a3c3-4a381f96bf8b\") "
Oct 30 19:15:44.087733 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.083978    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19f00f47-605d-4b19-9a1c-664de3661507-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19f00f47-605d-4b19-9a1c-664de3661507-workload-socket\") pod \"19f00f47-605d-4b19-9a1c-664de3661507\" (UID: \"19f00f47-605d-4b19-9a1c-664de3661507\") "
Oct 30 19:15:44.087733 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.084037    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52a25631-2cfc-4155-a3c3-4a381f96bf8b-istio-token\" (UniqueName: \"kubernetes.io/projected/52a25631-2cfc-4155-a3c3-4a381f96bf8b-istio-token\") pod \"52a25631-2cfc-4155-a3c3-4a381f96bf8b\" (UID: \"52a25631-2cfc-4155-a3c3-4a381f96bf8b\") "
Oct 30 19:15:44.087733 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.084054    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52a25631-2cfc-4155-a3c3-4a381f96bf8b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52a25631-2cfc-4155-a3c3-4a381f96bf8b-workload-certs\") pod \"52a25631-2cfc-4155-a3c3-4a381f96bf8b\" (UID: \"52a25631-2cfc-4155-a3c3-4a381f96bf8b\") "
Oct 30 19:15:44.087733 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.084085    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19f00f47-605d-4b19-9a1c-664de3661507-kube-api-access-prt54\" (UniqueName: \"kubernetes.io/projected/19f00f47-605d-4b19-9a1c-664de3661507-kube-api-access-prt54\") pod \"19f00f47-605d-4b19-9a1c-664de3661507\" (UID: \"19f00f47-605d-4b19-9a1c-664de3661507\") "
Oct 30 19:15:44.087733 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.084123    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/52a25631-2cfc-4155-a3c3-4a381f96bf8b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52a25631-2cfc-4155-a3c3-4a381f96bf8b-istio-podinfo\") pod \"52a25631-2cfc-4155-a3c3-4a381f96bf8b\" (UID: \"52a25631-2cfc-4155-a3c3-4a381f96bf8b\") "
Oct 30 19:15:44.087733 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.084163    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52a25631-2cfc-4155-a3c3-4a381f96bf8b-kube-api-access-hnvc4\" (UniqueName: \"kubernetes.io/projected/52a25631-2cfc-4155-a3c3-4a381f96bf8b-kube-api-access-hnvc4\") pod \"52a25631-2cfc-4155-a3c3-4a381f96bf8b\" (UID: \"52a25631-2cfc-4155-a3c3-4a381f96bf8b\") "
Oct 30 19:15:44.087733 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.084219    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19f00f47-605d-4b19-9a1c-664de3661507-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19f00f47-605d-4b19-9a1c-664de3661507-credential-socket\") pod \"19f00f47-605d-4b19-9a1c-664de3661507\" (UID: \"19f00f47-605d-4b19-9a1c-664de3661507\") "
Oct 30 19:15:44.087733 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.084266    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19f00f47-605d-4b19-9a1c-664de3661507-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19f00f47-605d-4b19-9a1c-664de3661507-istiod-ca-cert\") pod \"19f00f47-605d-4b19-9a1c-664de3661507\" (UID: \"19f00f47-605d-4b19-9a1c-664de3661507\") "
Oct 30 19:15:44.087733 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.084311    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/19f00f47-605d-4b19-9a1c-664de3661507-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19f00f47-605d-4b19-9a1c-664de3661507-istio-podinfo\") pod \"19f00f47-605d-4b19-9a1c-664de3661507\" (UID: \"19f00f47-605d-4b19-9a1c-664de3661507\") "
Oct 30 19:15:44.087733 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.084366    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19f00f47-605d-4b19-9a1c-664de3661507-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19f00f47-605d-4b19-9a1c-664de3661507-istio-envoy\") pod \"19f00f47-605d-4b19-9a1c-664de3661507\" (UID: \"19f00f47-605d-4b19-9a1c-664de3661507\") "
Oct 30 19:15:44.087733 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.084384    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52a25631-2cfc-4155-a3c3-4a381f96bf8b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52a25631-2cfc-4155-a3c3-4a381f96bf8b-istio-envoy\") pod \"52a25631-2cfc-4155-a3c3-4a381f96bf8b\" (UID: \"52a25631-2cfc-4155-a3c3-4a381f96bf8b\") "
Oct 30 19:15:44.087733 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.084437    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19f00f47-605d-4b19-9a1c-664de3661507-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19f00f47-605d-4b19-9a1c-664de3661507-workload-certs\") pod \"19f00f47-605d-4b19-9a1c-664de3661507\" (UID: \"19f00f47-605d-4b19-9a1c-664de3661507\") "
Oct 30 19:15:44.087733 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.084479    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19f00f47-605d-4b19-9a1c-664de3661507-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19f00f47-605d-4b19-9a1c-664de3661507-istio-data\") pod \"19f00f47-605d-4b19-9a1c-664de3661507\" (UID: \"19f00f47-605d-4b19-9a1c-664de3661507\") "
Oct 30 19:15:44.087733 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.085380    4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419-istio-data\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:44.087733 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.085400    4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4cbde58-0ec2-4f0d-af09-048853d07d63-istio-podinfo\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:44.087733 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.085411    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ks47v\" (UniqueName: \"kubernetes.io/projected/e4cbde58-0ec2-4f0d-af09-048853d07d63-kube-api-access-ks47v\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:44.087733 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.085421    4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4cbde58-0ec2-4f0d-af09-048853d07d63-istio-envoy\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:44.087733 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.085431    4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419-istio-envoy\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:44.087733 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.085439    4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2f9a313-f069-4e48-8cdc-f75c6a285a7c-istio-token\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:44.087733 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.085449    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8mst\" (UniqueName: \"kubernetes.io/projected/d2f9a313-f069-4e48-8cdc-f75c6a285a7c-kube-api-access-m8mst\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:44.087733 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.085458    4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2f9a313-f069-4e48-8cdc-f75c6a285a7c-workload-certs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:44.087733 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.085468    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcr65\" (UniqueName: \"kubernetes.io/projected/f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419-kube-api-access-zcr65\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:44.087733 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.085476    4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4cbde58-0ec2-4f0d-af09-048853d07d63-istio-data\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:44.087733 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.085485    4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4cbde58-0ec2-4f0d-af09-048853d07d63-workload-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:44.087733 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.085505    4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2f9a313-f069-4e48-8cdc-f75c6a285a7c-credential-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:44.087733 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.085513    4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4cbde58-0ec2-4f0d-af09-048853d07d63-workload-certs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:44.087733 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.085521    4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2f9a313-f069-4e48-8cdc-f75c6a285a7c-istio-envoy\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:44.087733 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.085541    4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419-istio-token\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:44.087733 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.085550    4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419-istio-podinfo\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:44.087733 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.085794    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19f00f47-605d-4b19-9a1c-664de3661507-istio-data" pod "19f00f47-605d-4b19-9a1c-664de3661507" (UID: "19f00f47-605d-4b19-9a1c-664de3661507"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:44.087733 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.087466    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19f00f47-605d-4b19-9a1c-664de3661507-kube-api-access-prt54" pod "19f00f47-605d-4b19-9a1c-664de3661507" (UID: "19f00f47-605d-4b19-9a1c-664de3661507"). InnerVolumeSpecName "kube-api-access-prt54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:44.091010 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.087782    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52a25631-2cfc-4155-a3c3-4a381f96bf8b-workload-socket" pod "52a25631-2cfc-4155-a3c3-4a381f96bf8b" (UID: "52a25631-2cfc-4155-a3c3-4a381f96bf8b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:44.091010 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.088043    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52a25631-2cfc-4155-a3c3-4a381f96bf8b-istiod-ca-cert" pod "52a25631-2cfc-4155-a3c3-4a381f96bf8b" (UID: "52a25631-2cfc-4155-a3c3-4a381f96bf8b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:44.091010 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.090836    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19f00f47-605d-4b19-9a1c-664de3661507-istio-token" pod "19f00f47-605d-4b19-9a1c-664de3661507" (UID: "19f00f47-605d-4b19-9a1c-664de3661507"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:44.091010 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.090976    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52a25631-2cfc-4155-a3c3-4a381f96bf8b-istio-data" pod "52a25631-2cfc-4155-a3c3-4a381f96bf8b" (UID: "52a25631-2cfc-4155-a3c3-4a381f96bf8b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:44.091350 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.091088    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52a25631-2cfc-4155-a3c3-4a381f96bf8b-credential-socket" pod "52a25631-2cfc-4155-a3c3-4a381f96bf8b" (UID: "52a25631-2cfc-4155-a3c3-4a381f96bf8b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:44.091350 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.091201    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19f00f47-605d-4b19-9a1c-664de3661507-workload-socket" pod "19f00f47-605d-4b19-9a1c-664de3661507" (UID: "19f00f47-605d-4b19-9a1c-664de3661507"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:44.092801 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:15:44.091574    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa10dce9b5a287001fe76c4462e484a4bd2813cb8c89327e64aa82ea0959d438\": container with ID starting with aa10dce9b5a287001fe76c4462e484a4bd2813cb8c89327e64aa82ea0959d438 not found: ID does not exist" containerID="aa10dce9b5a287001fe76c4462e484a4bd2813cb8c89327e64aa82ea0959d438"
Oct 30 19:15:44.092801 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.091612    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa10dce9b5a287001fe76c4462e484a4bd2813cb8c89327e64aa82ea0959d438"} err="failed to get container status \"aa10dce9b5a287001fe76c4462e484a4bd2813cb8c89327e64aa82ea0959d438\": rpc error: code = NotFound desc = could not find container \"aa10dce9b5a287001fe76c4462e484a4bd2813cb8c89327e64aa82ea0959d438\": container with ID starting with aa10dce9b5a287001fe76c4462e484a4bd2813cb8c89327e64aa82ea0959d438 not found: ID does not exist"
Oct 30 19:15:44.092801 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.091637    4525 scope.go:122] "RemoveContainer" containerID="6c6e31cf364ed20bf0047898b173cf5e5f6e37039f70c57f5d094d2449541a4e"
Oct 30 19:15:44.093516 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.093106    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52a25631-2cfc-4155-a3c3-4a381f96bf8b-istio-token" pod "52a25631-2cfc-4155-a3c3-4a381f96bf8b" (UID: "52a25631-2cfc-4155-a3c3-4a381f96bf8b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:44.093516 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.093151    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52a25631-2cfc-4155-a3c3-4a381f96bf8b-workload-certs" pod "52a25631-2cfc-4155-a3c3-4a381f96bf8b" (UID: "52a25631-2cfc-4155-a3c3-4a381f96bf8b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:44.093516 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.093218    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-5hdmh"]
Oct 30 19:15:44.094053 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.093927    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19f00f47-605d-4b19-9a1c-664de3661507-workload-certs" pod "19f00f47-605d-4b19-9a1c-664de3661507" (UID: "19f00f47-605d-4b19-9a1c-664de3661507"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:44.094053 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.094018    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19f00f47-605d-4b19-9a1c-664de3661507-credential-socket" pod "19f00f47-605d-4b19-9a1c-664de3661507" (UID: "19f00f47-605d-4b19-9a1c-664de3661507"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:44.094357 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.094335    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19f00f47-605d-4b19-9a1c-664de3661507-istiod-ca-cert" pod "19f00f47-605d-4b19-9a1c-664de3661507" (UID: "19f00f47-605d-4b19-9a1c-664de3661507"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:44.097016 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.096992    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19f00f47-605d-4b19-9a1c-664de3661507-istio-envoy" pod "19f00f47-605d-4b19-9a1c-664de3661507" (UID: "19f00f47-605d-4b19-9a1c-664de3661507"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:44.103127 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.103075    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/19f00f47-605d-4b19-9a1c-664de3661507-istio-podinfo" pod "19f00f47-605d-4b19-9a1c-664de3661507" (UID: "19f00f47-605d-4b19-9a1c-664de3661507"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:15:44.103127 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.103085    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52a25631-2cfc-4155-a3c3-4a381f96bf8b-istio-envoy" pod "52a25631-2cfc-4155-a3c3-4a381f96bf8b" (UID: "52a25631-2cfc-4155-a3c3-4a381f96bf8b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:44.103299 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.103246    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/52a25631-2cfc-4155-a3c3-4a381f96bf8b-istio-podinfo" pod "52a25631-2cfc-4155-a3c3-4a381f96bf8b" (UID: "52a25631-2cfc-4155-a3c3-4a381f96bf8b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:15:44.104428 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.104397    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-5hdmh"]
Oct 30 19:15:44.104873 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.104845    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52a25631-2cfc-4155-a3c3-4a381f96bf8b-kube-api-access-hnvc4" pod "52a25631-2cfc-4155-a3c3-4a381f96bf8b" (UID: "52a25631-2cfc-4155-a3c3-4a381f96bf8b"). InnerVolumeSpecName "kube-api-access-hnvc4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:44.141548 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.141204    4525 scope.go:122] "RemoveContainer" containerID="6c6e31cf364ed20bf0047898b173cf5e5f6e37039f70c57f5d094d2449541a4e"
Oct 30 19:15:44.142210 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:15:44.141801    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c6e31cf364ed20bf0047898b173cf5e5f6e37039f70c57f5d094d2449541a4e\": container with ID starting with 6c6e31cf364ed20bf0047898b173cf5e5f6e37039f70c57f5d094d2449541a4e not found: ID does not exist" containerID="6c6e31cf364ed20bf0047898b173cf5e5f6e37039f70c57f5d094d2449541a4e"
Oct 30 19:15:44.142210 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.141843    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c6e31cf364ed20bf0047898b173cf5e5f6e37039f70c57f5d094d2449541a4e"} err="failed to get container status \"6c6e31cf364ed20bf0047898b173cf5e5f6e37039f70c57f5d094d2449541a4e\": rpc error: code = NotFound desc = could not find container \"6c6e31cf364ed20bf0047898b173cf5e5f6e37039f70c57f5d094d2449541a4e\": container with ID starting with 6c6e31cf364ed20bf0047898b173cf5e5f6e37039f70c57f5d094d2449541a4e not found: ID does not exist"
Oct 30 19:15:44.142210 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.141869    4525 scope.go:122] "RemoveContainer" containerID="5b8c6c1e4d153d4c504d2beac49d7d788e072865b13d8580aa8a54e737d1558b"
Oct 30 19:15:44.157010 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.156943    4525 scope.go:122] "RemoveContainer" containerID="4e0c7213e1372f284371fd7df3e88f0b367fecefa141b4f62b24c7e75b267acf"
Oct 30 19:15:44.175176 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.174473    4525 scope.go:122] "RemoveContainer" containerID="5b8c6c1e4d153d4c504d2beac49d7d788e072865b13d8580aa8a54e737d1558b"
Oct 30 19:15:44.175176 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:15:44.174878    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b8c6c1e4d153d4c504d2beac49d7d788e072865b13d8580aa8a54e737d1558b\": container with ID starting with 5b8c6c1e4d153d4c504d2beac49d7d788e072865b13d8580aa8a54e737d1558b not found: ID does not exist" containerID="5b8c6c1e4d153d4c504d2beac49d7d788e072865b13d8580aa8a54e737d1558b"
Oct 30 19:15:44.175176 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.174911    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b8c6c1e4d153d4c504d2beac49d7d788e072865b13d8580aa8a54e737d1558b"} err="failed to get container status \"5b8c6c1e4d153d4c504d2beac49d7d788e072865b13d8580aa8a54e737d1558b\": rpc error: code = NotFound desc = could not find container \"5b8c6c1e4d153d4c504d2beac49d7d788e072865b13d8580aa8a54e737d1558b\": container with ID starting with 5b8c6c1e4d153d4c504d2beac49d7d788e072865b13d8580aa8a54e737d1558b not found: ID does not exist"
Oct 30 19:15:44.175176 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.174936    4525 scope.go:122] "RemoveContainer" containerID="4e0c7213e1372f284371fd7df3e88f0b367fecefa141b4f62b24c7e75b267acf"
Oct 30 19:15:44.175571 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:15:44.175267    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e0c7213e1372f284371fd7df3e88f0b367fecefa141b4f62b24c7e75b267acf\": container with ID starting with 4e0c7213e1372f284371fd7df3e88f0b367fecefa141b4f62b24c7e75b267acf not found: ID does not exist" containerID="4e0c7213e1372f284371fd7df3e88f0b367fecefa141b4f62b24c7e75b267acf"
Oct 30 19:15:44.175571 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.175293    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e0c7213e1372f284371fd7df3e88f0b367fecefa141b4f62b24c7e75b267acf"} err="failed to get container status \"4e0c7213e1372f284371fd7df3e88f0b367fecefa141b4f62b24c7e75b267acf\": rpc error: code = NotFound desc = could not find container \"4e0c7213e1372f284371fd7df3e88f0b367fecefa141b4f62b24c7e75b267acf\": container with ID starting with 4e0c7213e1372f284371fd7df3e88f0b367fecefa141b4f62b24c7e75b267acf not found: ID does not exist"
Oct 30 19:15:44.187061 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.187032    4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19f00f47-605d-4b19-9a1c-664de3661507-workload-certs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:44.187191 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.187064    4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19f00f47-605d-4b19-9a1c-664de3661507-istio-data\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:44.187191 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.187079    4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52a25631-2cfc-4155-a3c3-4a381f96bf8b-workload-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:44.187191 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.187088    4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52a25631-2cfc-4155-a3c3-4a381f96bf8b-istiod-ca-cert\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:44.187191 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.187097    4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19f00f47-605d-4b19-9a1c-664de3661507-istio-token\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:44.187191 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.187107    4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52a25631-2cfc-4155-a3c3-4a381f96bf8b-istio-data\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:44.187191 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.187115    4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52a25631-2cfc-4155-a3c3-4a381f96bf8b-credential-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:44.187191 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.187125    4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19f00f47-605d-4b19-9a1c-664de3661507-workload-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:44.187191 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.187135    4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52a25631-2cfc-4155-a3c3-4a381f96bf8b-istio-token\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:44.187191 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.187143    4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52a25631-2cfc-4155-a3c3-4a381f96bf8b-workload-certs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:44.187191 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.187152    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prt54\" (UniqueName: \"kubernetes.io/projected/19f00f47-605d-4b19-9a1c-664de3661507-kube-api-access-prt54\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:44.187191 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.187160    4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52a25631-2cfc-4155-a3c3-4a381f96bf8b-istio-podinfo\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:44.187191 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.187169    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnvc4\" (UniqueName: \"kubernetes.io/projected/52a25631-2cfc-4155-a3c3-4a381f96bf8b-kube-api-access-hnvc4\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:44.187191 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.187177    4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19f00f47-605d-4b19-9a1c-664de3661507-credential-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:44.187191 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.187185    4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19f00f47-605d-4b19-9a1c-664de3661507-istiod-ca-cert\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:44.187191 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.187192    4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19f00f47-605d-4b19-9a1c-664de3661507-istio-podinfo\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:44.187191 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.187202    4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19f00f47-605d-4b19-9a1c-664de3661507-istio-envoy\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:44.188154 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.187210    4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52a25631-2cfc-4155-a3c3-4a381f96bf8b-istio-envoy\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:44.249523 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.249493    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-6crc4"]
Oct 30 19:15:44.263730 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.263697    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-6crc4"]
Oct 30 19:15:44.272458 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.272428    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-ftq7b"]
Oct 30 19:15:44.278307 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.278274    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-ftq7b"]
Oct 30 19:15:44.285874 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.285835    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45"]
Oct 30 19:15:44.290596 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.290571    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2xr45"]
Oct 30 19:15:44.969581 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.969547    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2cf46"
Oct 30 19:15:44.970105 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.969579    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2cf46" event={"ID":"52a25631-2cfc-4155-a3c3-4a381f96bf8b","Type":"ContainerDied","Data":"da703547836a8016d2af8bbc05dd14b2720547bf506c0364bf5d6e5e6631be73"}
Oct 30 19:15:44.970105 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.969658    4525 scope.go:122] "RemoveContainer" containerID="b520c9f45cc994c8fe9071082fad3019e873bfaac64681f87b690ca7d3c8714d"
Oct 30 19:15:44.973300 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.973282    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-68b466-openshift-default-57cb5457fc-jgkhq"
Oct 30 19:15:44.973439 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.973324    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-68b466-openshift-default-57cb5457fc-jgkhq" event={"ID":"19f00f47-605d-4b19-9a1c-664de3661507","Type":"ContainerDied","Data":"fb49545060e773d6c39767f3d47f21ef1a4543762041a99e0e17c37c098eec3c"}
Oct 30 19:15:44.982698 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:44.982663    4525 scope.go:122] "RemoveContainer" containerID="3f123adeefacee438ae8ecf734e1bd9a72367e41eb105cf70ca97d5e30425608"
Oct 30 19:15:45.051854 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:45.051813    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-68b466-openshift-default-57cb5457fc-jgkhq"]
Oct 30 19:15:45.059876 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:45.059845    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-68b466-openshift-default-57cb5457fc-jgkhq"]
Oct 30 19:15:45.066630 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:45.066606    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2cf46"]
Oct 30 19:15:45.097239 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:45.097211    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-d6f956-openshift-default-56888545df-2cf46"]
Oct 30 19:15:45.545232 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:45.545200    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19f00f47-605d-4b19-9a1c-664de3661507" path="/var/lib/kubelet/pods/19f00f47-605d-4b19-9a1c-664de3661507/volumes"
Oct 30 19:15:45.545785 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:45.545763    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52a25631-2cfc-4155-a3c3-4a381f96bf8b" path="/var/lib/kubelet/pods/52a25631-2cfc-4155-a3c3-4a381f96bf8b/volumes"
Oct 30 19:15:45.546315 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:45.546300    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2f9a313-f069-4e48-8cdc-f75c6a285a7c" path="/var/lib/kubelet/pods/d2f9a313-f069-4e48-8cdc-f75c6a285a7c/volumes"
Oct 30 19:15:45.547162 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:45.547133    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4cbde58-0ec2-4f0d-af09-048853d07d63" path="/var/lib/kubelet/pods/e4cbde58-0ec2-4f0d-af09-048853d07d63/volumes"
Oct 30 19:15:45.547585 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:45.547571    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7e914c3-d828-407a-bafb-f71a07d872c1" path="/var/lib/kubelet/pods/f7e914c3-d828-407a-bafb-f71a07d872c1/volumes"
Oct 30 19:15:45.548014 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:45.548000    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419" path="/var/lib/kubelet/pods/f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419/volumes"
Oct 30 19:15:47.401157 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:15:47.401114    4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d49l" containerName="extractor"
Oct 30 19:15:53.004744 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:53.003395    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c2e3f9-mgxxs"]
Oct 30 19:15:53.015107 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:53.015061    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c2e3f9-mgxxs"]
Oct 30 19:15:53.439457 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:53.439424    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb4b9fc4-x5727"]
Oct 30 19:15:53.440022 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:53.440008    4525 cpu_manager.go:414] "containerMap: removing container" podUID="5c830b89-50cc-46ea-909d-5f1127534794" containerName="revoke-keys"
Oct 30 19:15:53.440067 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:53.440024    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c830b89-50cc-46ea-909d-5f1127534794" containerName="revoke-keys"
Oct 30 19:15:53.440067 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:53.440035    4525 cpu_manager.go:414] "containerMap: removing container" podUID="19f00f47-605d-4b19-9a1c-664de3661507" containerName="istio-proxy"
Oct 30 19:15:53.440067 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:53.440039    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19f00f47-605d-4b19-9a1c-664de3661507" containerName="istio-proxy"
Oct 30 19:15:53.440067 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:53.440061    4525 cpu_manager.go:414] "containerMap: removing container" podUID="d2f9a313-f069-4e48-8cdc-f75c6a285a7c" containerName="istio-proxy"
Oct 30 19:15:53.440067 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:53.440066    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2f9a313-f069-4e48-8cdc-f75c6a285a7c" containerName="istio-proxy"
Oct 30 19:15:53.440224 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:53.440072    4525 cpu_manager.go:414] "containerMap: removing container" podUID="a8fc9310-cf03-4629-be6c-4d184a067d93" containerName="revoke-keys"
Oct 30 19:15:53.440224 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:53.440077    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8fc9310-cf03-4629-be6c-4d184a067d93" containerName="revoke-keys"
Oct 30 19:15:53.440224 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:53.440084    4525 cpu_manager.go:414] "containerMap: removing container" podUID="52a25631-2cfc-4155-a3c3-4a381f96bf8b" containerName="istio-proxy"
Oct 30 19:15:53.440224 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:53.440089    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52a25631-2cfc-4155-a3c3-4a381f96bf8b" containerName="istio-proxy"
Oct 30 19:15:53.440224 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:53.440096    4525 cpu_manager.go:414] "containerMap: removing container" podUID="f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419" containerName="istio-proxy"
Oct 30 19:15:53.440224 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:53.440100    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419" containerName="istio-proxy"
Oct 30 19:15:53.440224 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:53.440111    4525 cpu_manager.go:414] "containerMap: removing container" podUID="d2f9a313-f069-4e48-8cdc-f75c6a285a7c" containerName="istio-proxy"
Oct 30 19:15:53.440224 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:53.440115    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2f9a313-f069-4e48-8cdc-f75c6a285a7c" containerName="istio-proxy"
Oct 30 19:15:53.440224 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:53.440123    4525 cpu_manager.go:414] "containerMap: removing container" podUID="e4cbde58-0ec2-4f0d-af09-048853d07d63" containerName="istio-proxy"
Oct 30 19:15:53.440224 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:53.440128    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4cbde58-0ec2-4f0d-af09-048853d07d63" containerName="istio-proxy"
Oct 30 19:15:53.440224 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:53.440136    4525 cpu_manager.go:414] "containerMap: removing container" podUID="f7e914c3-d828-407a-bafb-f71a07d872c1" containerName="istio-proxy"
Oct 30 19:15:53.440224 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:53.440140    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7e914c3-d828-407a-bafb-f71a07d872c1" containerName="istio-proxy"
Oct 30 19:15:53.440601 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:53.440239    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c830b89-50cc-46ea-909d-5f1127534794" containerName="revoke-keys"
Oct 30 19:15:53.440601 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:53.440247    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="19f00f47-605d-4b19-9a1c-664de3661507" containerName="istio-proxy"
Oct 30 19:15:53.440601 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:53.440256    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2f9a313-f069-4e48-8cdc-f75c6a285a7c" containerName="istio-proxy"
Oct 30 19:15:53.440601 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:53.440263    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8fc9310-cf03-4629-be6c-4d184a067d93" containerName="revoke-keys"
Oct 30 19:15:53.440601 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:53.440269    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="52a25631-2cfc-4155-a3c3-4a381f96bf8b" containerName="istio-proxy"
Oct 30 19:15:53.440601 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:53.440275    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8bb77e5-3e3a-4bc7-ae77-06b5eb01d419" containerName="istio-proxy"
Oct 30 19:15:53.440601 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:53.440280    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2f9a313-f069-4e48-8cdc-f75c6a285a7c" containerName="istio-proxy"
Oct 30 19:15:53.440601 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:53.440284    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4cbde58-0ec2-4f0d-af09-048853d07d63" containerName="istio-proxy"
Oct 30 19:15:53.440601 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:53.440291    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7e914c3-d828-407a-bafb-f71a07d872c1" containerName="istio-proxy"
Oct 30 19:15:53.445585 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:53.445565    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb4b9fc4-x5727"
Oct 30 19:15:53.452244 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:53.452219    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb4b9fc4-x5727"]
Oct 30 19:15:53.528783 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:53.528749    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:15:53.528939 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:53.528795    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94hcn\""
Oct 30 19:15:53.538350 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:53.538327    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a2b88cf-5484-4765-ac9e-87917569f9f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb4b9fc4-x5727\" (UID: \"0a2b88cf-5484-4765-ac9e-87917569f9f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb4b9fc4-x5727"
Oct 30 19:15:53.538536 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:53.538519    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a2b88cf-5484-4765-ac9e-87917569f9f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb4b9fc4-x5727\" (UID: \"0a2b88cf-5484-4765-ac9e-87917569f9f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb4b9fc4-x5727"
Oct 30 19:15:53.538587 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:53.538558    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hdr8\" (UniqueName: \"kubernetes.io/projected/0a2b88cf-5484-4765-ac9e-87917569f9f4-kube-api-access-9hdr8\") pod \"maas-api-revoke-sub-fb4b9fc4-x5727\" (UID: \"0a2b88cf-5484-4765-ac9e-87917569f9f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb4b9fc4-x5727"
Oct 30 19:15:53.549844 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:53.549812    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e80605e4-644b-42fc-b81d-3c1885a99496" path="/var/lib/kubelet/pods/e80605e4-644b-42fc-b81d-3c1885a99496/volumes"
Oct 30 19:15:53.639955 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:53.639916    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hdr8\" (UniqueName: \"kubernetes.io/projected/0a2b88cf-5484-4765-ac9e-87917569f9f4-kube-api-access-9hdr8\") pod \"maas-api-revoke-sub-fb4b9fc4-x5727\" (UID: \"0a2b88cf-5484-4765-ac9e-87917569f9f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb4b9fc4-x5727"
Oct 30 19:15:53.640115 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:53.640011    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a2b88cf-5484-4765-ac9e-87917569f9f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb4b9fc4-x5727\" (UID: \"0a2b88cf-5484-4765-ac9e-87917569f9f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb4b9fc4-x5727"
Oct 30 19:15:53.640199 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:53.640188    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a2b88cf-5484-4765-ac9e-87917569f9f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb4b9fc4-x5727\" (UID: \"0a2b88cf-5484-4765-ac9e-87917569f9f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb4b9fc4-x5727"
Oct 30 19:15:53.640880 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:53.640854    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a2b88cf-5484-4765-ac9e-87917569f9f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb4b9fc4-x5727\" (UID: \"0a2b88cf-5484-4765-ac9e-87917569f9f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb4b9fc4-x5727"
Oct 30 19:15:53.642780 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:53.642752    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a2b88cf-5484-4765-ac9e-87917569f9f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb4b9fc4-x5727\" (UID: \"0a2b88cf-5484-4765-ac9e-87917569f9f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb4b9fc4-x5727"
Oct 30 19:15:53.648365 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:53.648340    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hdr8\" (UniqueName: \"kubernetes.io/projected/0a2b88cf-5484-4765-ac9e-87917569f9f4-kube-api-access-9hdr8\") pod \"maas-api-revoke-sub-fb4b9fc4-x5727\" (UID: \"0a2b88cf-5484-4765-ac9e-87917569f9f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb4b9fc4-x5727"
Oct 30 19:15:53.763480 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:53.763385    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb4b9fc4-x5727"
Oct 30 19:15:54.019212 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:54.019102    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-ed012e-s6vzt"]
Oct 30 19:15:54.026193 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:54.026163    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-ed012e-s6vzt"]
Oct 30 19:15:54.172542 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:54.172502    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb4b9fc4-x5727"]
Oct 30 19:15:55.273200 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:55.273165    4525 generic.go:356] "Generic (PLEG): container finished" podID="0a2b88cf-5484-4765-ac9e-87917569f9f4" containerID="3a1727c52a930b45fe6ec46ed51cc8fb02a00e7523caebde8967242c03a4be62" exitCode=0
Oct 30 19:15:55.273632 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:55.273250    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb4b9fc4-x5727" event={"ID":"0a2b88cf-5484-4765-ac9e-87917569f9f4","Type":"ContainerDied","Data":"3a1727c52a930b45fe6ec46ed51cc8fb02a00e7523caebde8967242c03a4be62"}
Oct 30 19:15:55.273632 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:55.273308    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb4b9fc4-x5727" event={"ID":"0a2b88cf-5484-4765-ac9e-87917569f9f4","Type":"ContainerStarted","Data":"c6a27af83646bc3ad05dae3aaca82a64b3e08b51ba12c24664745da83d3f41dd"}
Oct 30 19:15:55.546540 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:55.546506    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef28c15d-d04d-4186-ad97-5c88f5f89795" path="/var/lib/kubelet/pods/ef28c15d-d04d-4186-ad97-5c88f5f89795/volumes"
Oct 30 19:15:56.468046 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:56.467417    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb4b9fc4-x5727"
Oct 30 19:15:56.504220 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:56.504185    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a2b88cf-5484-4765-ac9e-87917569f9f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a2b88cf-5484-4765-ac9e-87917569f9f4-maas-api-xks-ca\") pod \"0a2b88cf-5484-4765-ac9e-87917569f9f4\" (UID: \"0a2b88cf-5484-4765-ac9e-87917569f9f4\") "
Oct 30 19:15:56.504418 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:56.504300    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a2b88cf-5484-4765-ac9e-87917569f9f4-kube-api-access-9hdr8\" (UniqueName: \"kubernetes.io/projected/0a2b88cf-5484-4765-ac9e-87917569f9f4-kube-api-access-9hdr8\") pod \"0a2b88cf-5484-4765-ac9e-87917569f9f4\" (UID: \"0a2b88cf-5484-4765-ac9e-87917569f9f4\") "
Oct 30 19:15:56.504500 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:56.504465    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a2b88cf-5484-4765-ac9e-87917569f9f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a2b88cf-5484-4765-ac9e-87917569f9f4-maas-api-openshift-ca\") pod \"0a2b88cf-5484-4765-ac9e-87917569f9f4\" (UID: \"0a2b88cf-5484-4765-ac9e-87917569f9f4\") "
Oct 30 19:15:56.505178 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:56.505144    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a2b88cf-5484-4765-ac9e-87917569f9f4-maas-api-openshift-ca" pod "0a2b88cf-5484-4765-ac9e-87917569f9f4" (UID: "0a2b88cf-5484-4765-ac9e-87917569f9f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:56.506588 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:56.506561    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a2b88cf-5484-4765-ac9e-87917569f9f4-maas-api-xks-ca" pod "0a2b88cf-5484-4765-ac9e-87917569f9f4" (UID: "0a2b88cf-5484-4765-ac9e-87917569f9f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:15:56.506588 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:56.506583    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a2b88cf-5484-4765-ac9e-87917569f9f4-kube-api-access-9hdr8" pod "0a2b88cf-5484-4765-ac9e-87917569f9f4" (UID: "0a2b88cf-5484-4765-ac9e-87917569f9f4"). InnerVolumeSpecName "kube-api-access-9hdr8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:56.606810 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:56.606785    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a2b88cf-5484-4765-ac9e-87917569f9f4-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:56.606810 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:56.606809    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hdr8\" (UniqueName: \"kubernetes.io/projected/0a2b88cf-5484-4765-ac9e-87917569f9f4-kube-api-access-9hdr8\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:56.606999 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:56.606821    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a2b88cf-5484-4765-ac9e-87917569f9f4-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:57.290466 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:57.290437    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb4b9fc4-x5727"
Oct 30 19:15:57.290466 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:57.290458    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb4b9fc4-x5727" event={"ID":"0a2b88cf-5484-4765-ac9e-87917569f9f4","Type":"ContainerDied","Data":"c6a27af83646bc3ad05dae3aaca82a64b3e08b51ba12c24664745da83d3f41dd"}
Oct 30 19:15:57.290750 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:57.290499    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6a27af83646bc3ad05dae3aaca82a64b3e08b51ba12c24664745da83d3f41dd"
Oct 30 19:15:58.166547 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:58.166509    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5e80b74-9sj46"]
Oct 30 19:15:58.167125 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:58.167112    4525 cpu_manager.go:414] "containerMap: removing container" podUID="19f00f47-605d-4b19-9a1c-664de3661507" containerName="istio-proxy"
Oct 30 19:15:58.167187 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:58.167126    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19f00f47-605d-4b19-9a1c-664de3661507" containerName="istio-proxy"
Oct 30 19:15:58.167187 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:58.167162    4525 cpu_manager.go:414] "containerMap: removing container" podUID="0a2b88cf-5484-4765-ac9e-87917569f9f4" containerName="revoke-keys"
Oct 30 19:15:58.167187 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:58.167167    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a2b88cf-5484-4765-ac9e-87917569f9f4" containerName="revoke-keys"
Oct 30 19:15:58.167286 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:58.167266    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="19f00f47-605d-4b19-9a1c-664de3661507" containerName="istio-proxy"
Oct 30 19:15:58.167286 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:58.167276    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a2b88cf-5484-4765-ac9e-87917569f9f4" containerName="revoke-keys"
Oct 30 19:15:58.170724 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:58.170666    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5e80b74-9sj46"
Oct 30 19:15:58.174504 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:58.174477    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f3a3b99-57p29"]
Oct 30 19:15:58.177792 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:58.177582    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:15:58.180730 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:58.179760    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f3a3b99-57p29"
Oct 30 19:15:58.183731 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:58.182789    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94hcn\""
Oct 30 19:15:58.186488 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:58.186468    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5e80b74-9sj46"]
Oct 30 19:15:58.206108 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:58.206079    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f3a3b99-57p29"]
Oct 30 19:15:58.228930 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:58.228896    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70641967-e202-48ef-bdbd-d03939412444-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5e80b74-9sj46\" (UID: \"70641967-e202-48ef-bdbd-d03939412444\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5e80b74-9sj46"
Oct 30 19:15:58.229096 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:58.228972    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eda0d3b8-cfbd-44bf-940d-200d449dec9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f3a3b99-57p29\" (UID: \"eda0d3b8-cfbd-44bf-940d-200d449dec9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f3a3b99-57p29"
Oct 30 19:15:58.229140 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:58.229092    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpr2k\" (UniqueName: \"kubernetes.io/projected/70641967-e202-48ef-bdbd-d03939412444-kube-api-access-lpr2k\") pod \"maas-api-revoke-sub-a5e80b74-9sj46\" (UID: \"70641967-e202-48ef-bdbd-d03939412444\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5e80b74-9sj46"
Oct 30 19:15:58.229182 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:58.229143    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vh6qp\" (UniqueName: \"kubernetes.io/projected/eda0d3b8-cfbd-44bf-940d-200d449dec9a-kube-api-access-vh6qp\") pod \"maas-api-revoke-sub-2f3a3b99-57p29\" (UID: \"eda0d3b8-cfbd-44bf-940d-200d449dec9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f3a3b99-57p29"
Oct 30 19:15:58.229238 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:58.229223    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70641967-e202-48ef-bdbd-d03939412444-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5e80b74-9sj46\" (UID: \"70641967-e202-48ef-bdbd-d03939412444\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5e80b74-9sj46"
Oct 30 19:15:58.229291 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:58.229272    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eda0d3b8-cfbd-44bf-940d-200d449dec9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f3a3b99-57p29\" (UID: \"eda0d3b8-cfbd-44bf-940d-200d449dec9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f3a3b99-57p29"
Oct 30 19:15:58.331148 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:58.331040    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eda0d3b8-cfbd-44bf-940d-200d449dec9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f3a3b99-57p29\" (UID: \"eda0d3b8-cfbd-44bf-940d-200d449dec9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f3a3b99-57p29"
Oct 30 19:15:58.331340 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:58.331230    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70641967-e202-48ef-bdbd-d03939412444-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5e80b74-9sj46\" (UID: \"70641967-e202-48ef-bdbd-d03939412444\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5e80b74-9sj46"
Oct 30 19:15:58.331340 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:58.331308    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eda0d3b8-cfbd-44bf-940d-200d449dec9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f3a3b99-57p29\" (UID: \"eda0d3b8-cfbd-44bf-940d-200d449dec9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f3a3b99-57p29"
Oct 30 19:15:58.331480 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:58.331401    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpr2k\" (UniqueName: \"kubernetes.io/projected/70641967-e202-48ef-bdbd-d03939412444-kube-api-access-lpr2k\") pod \"maas-api-revoke-sub-a5e80b74-9sj46\" (UID: \"70641967-e202-48ef-bdbd-d03939412444\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5e80b74-9sj46"
Oct 30 19:15:58.331480 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:58.331470    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh6qp\" (UniqueName: \"kubernetes.io/projected/eda0d3b8-cfbd-44bf-940d-200d449dec9a-kube-api-access-vh6qp\") pod \"maas-api-revoke-sub-2f3a3b99-57p29\" (UID: \"eda0d3b8-cfbd-44bf-940d-200d449dec9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f3a3b99-57p29"
Oct 30 19:15:58.331594 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:58.331517    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70641967-e202-48ef-bdbd-d03939412444-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5e80b74-9sj46\" (UID: \"70641967-e202-48ef-bdbd-d03939412444\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5e80b74-9sj46"
Oct 30 19:15:58.331936 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:58.331904    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eda0d3b8-cfbd-44bf-940d-200d449dec9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f3a3b99-57p29\" (UID: \"eda0d3b8-cfbd-44bf-940d-200d449dec9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f3a3b99-57p29"
Oct 30 19:15:58.332218 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:58.332198    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70641967-e202-48ef-bdbd-d03939412444-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5e80b74-9sj46\" (UID: \"70641967-e202-48ef-bdbd-d03939412444\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5e80b74-9sj46"
Oct 30 19:15:58.333823 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:58.333803    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70641967-e202-48ef-bdbd-d03939412444-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5e80b74-9sj46\" (UID: \"70641967-e202-48ef-bdbd-d03939412444\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5e80b74-9sj46"
Oct 30 19:15:58.334013 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:58.333986    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eda0d3b8-cfbd-44bf-940d-200d449dec9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f3a3b99-57p29\" (UID: \"eda0d3b8-cfbd-44bf-940d-200d449dec9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f3a3b99-57p29"
Oct 30 19:15:58.343117 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:58.343090    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vh6qp\" (UniqueName: \"kubernetes.io/projected/eda0d3b8-cfbd-44bf-940d-200d449dec9a-kube-api-access-vh6qp\") pod \"maas-api-revoke-sub-2f3a3b99-57p29\" (UID: \"eda0d3b8-cfbd-44bf-940d-200d449dec9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f3a3b99-57p29"
Oct 30 19:15:58.343246 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:58.343221    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpr2k\" (UniqueName: \"kubernetes.io/projected/70641967-e202-48ef-bdbd-d03939412444-kube-api-access-lpr2k\") pod \"maas-api-revoke-sub-a5e80b74-9sj46\" (UID: \"70641967-e202-48ef-bdbd-d03939412444\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5e80b74-9sj46"
Oct 30 19:15:58.505757 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:58.505487    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5e80b74-9sj46"
Oct 30 19:15:58.516741 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:58.510878    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f3a3b99-57p29"
Oct 30 19:15:58.643927 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:58.643903    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4a477d-k4l9b"]
Oct 30 19:15:58.654083 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:58.654050    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4a477d-k4l9b"]
Oct 30 19:15:58.958704 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:58.957861    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5e80b74-9sj46"]
Oct 30 19:15:59.035496 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:59.019648    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f3a3b99-57p29"]
Oct 30 19:15:59.045454 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:15:59.045356    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeda0d3b8_cfbd_44bf_940d_200d449dec9a.slice/crio-b18c897c4a29de38fee894e5553fd8e1ce9afecbee20e8111f2be60015c68c00 WatchSource:0}: Error finding container b18c897c4a29de38fee894e5553fd8e1ce9afecbee20e8111f2be60015c68c00: Status 404 returned error can't find the container with id b18c897c4a29de38fee894e5553fd8e1ce9afecbee20e8111f2be60015c68c00
Oct 30 19:15:59.319790 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:59.319749    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5e80b74-9sj46" event={"ID":"70641967-e202-48ef-bdbd-d03939412444","Type":"ContainerStarted","Data":"18577161c074e765320c240cf75a68973ff9659e8306e13787db44751880ace4"}
Oct 30 19:15:59.321605 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:59.321571    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f3a3b99-57p29" event={"ID":"eda0d3b8-cfbd-44bf-940d-200d449dec9a","Type":"ContainerStarted","Data":"b18c897c4a29de38fee894e5553fd8e1ce9afecbee20e8111f2be60015c68c00"}
Oct 30 19:15:59.546360 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:15:59.546328    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8fc9310-cf03-4629-be6c-4d184a067d93" path="/var/lib/kubelet/pods/a8fc9310-cf03-4629-be6c-4d184a067d93/volumes"
Oct 30 19:16:00.336612 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:00.336571    4525 generic.go:356] "Generic (PLEG): container finished" podID="70641967-e202-48ef-bdbd-d03939412444" containerID="82cfcc0bf11edbe45ff4045073e37f739aed3c69742a7cc54d9baad1a891946e" exitCode=0
Oct 30 19:16:00.337152 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:00.336735    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5e80b74-9sj46" event={"ID":"70641967-e202-48ef-bdbd-d03939412444","Type":"ContainerDied","Data":"82cfcc0bf11edbe45ff4045073e37f739aed3c69742a7cc54d9baad1a891946e"}
Oct 30 19:16:00.339731 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:00.339625    4525 generic.go:356] "Generic (PLEG): container finished" podID="eda0d3b8-cfbd-44bf-940d-200d449dec9a" containerID="b093f00d8ee0c06a008657ffb1cbe21d853ba9e57cc1ad9df19995dcc651f400" exitCode=0
Oct 30 19:16:00.339872 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:00.339801    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f3a3b99-57p29" event={"ID":"eda0d3b8-cfbd-44bf-940d-200d449dec9a","Type":"ContainerDied","Data":"b093f00d8ee0c06a008657ffb1cbe21d853ba9e57cc1ad9df19995dcc651f400"}
Oct 30 19:16:00.358619 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:00.358586    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-clvlk"]
Oct 30 19:16:00.358884 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:00.358853    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-clvlk" podUID="e14cca94-de16-4d65-b71a-9ece97424174" containerName="istio-proxy" containerID="cri-o://51099112565d75045a8d9cb6d3d3c02280322816fd0cdf6169c2a059fadb714a" gracePeriod=30
Oct 30 19:16:00.962706 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:00.949905    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2d1c71-2mg9x"]
Oct 30 19:16:01.303904 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:01.303502    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2d1c71-2mg9x"]
Oct 30 19:16:01.303904 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:01.303638    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2d1c71-2mg9x"
Oct 30 19:16:01.421411 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:01.415209    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsxkj\" (UniqueName: \"kubernetes.io/projected/80950513-d45b-4a63-a6b0-91d6a6ef112e-kube-api-access-qsxkj\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-2d1c71-2mg9x\" (UID: \"80950513-d45b-4a63-a6b0-91d6a6ef112e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2d1c71-2mg9x"
Oct 30 19:16:01.421411 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:01.418729    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80950513-d45b-4a63-a6b0-91d6a6ef112e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-2d1c71-2mg9x\" (UID: \"80950513-d45b-4a63-a6b0-91d6a6ef112e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2d1c71-2mg9x"
Oct 30 19:16:01.421411 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:01.418870    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80950513-d45b-4a63-a6b0-91d6a6ef112e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-2d1c71-2mg9x\" (UID: \"80950513-d45b-4a63-a6b0-91d6a6ef112e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2d1c71-2mg9x"
Oct 30 19:16:01.533810 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:01.532953    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80950513-d45b-4a63-a6b0-91d6a6ef112e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-2d1c71-2mg9x\" (UID: \"80950513-d45b-4a63-a6b0-91d6a6ef112e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2d1c71-2mg9x"
Oct 30 19:16:01.533810 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:01.533049    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsxkj\" (UniqueName: \"kubernetes.io/projected/80950513-d45b-4a63-a6b0-91d6a6ef112e-kube-api-access-qsxkj\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-2d1c71-2mg9x\" (UID: \"80950513-d45b-4a63-a6b0-91d6a6ef112e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2d1c71-2mg9x"
Oct 30 19:16:01.533810 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:01.533295    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80950513-d45b-4a63-a6b0-91d6a6ef112e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-2d1c71-2mg9x\" (UID: \"80950513-d45b-4a63-a6b0-91d6a6ef112e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2d1c71-2mg9x"
Oct 30 19:16:01.534566 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:01.534256    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80950513-d45b-4a63-a6b0-91d6a6ef112e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-2d1c71-2mg9x\" (UID: \"80950513-d45b-4a63-a6b0-91d6a6ef112e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2d1c71-2mg9x"
Oct 30 19:16:01.548233 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:01.537333    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80950513-d45b-4a63-a6b0-91d6a6ef112e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-2d1c71-2mg9x\" (UID: \"80950513-d45b-4a63-a6b0-91d6a6ef112e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2d1c71-2mg9x"
Oct 30 19:16:01.584925 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:01.582666    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsxkj\" (UniqueName: \"kubernetes.io/projected/80950513-d45b-4a63-a6b0-91d6a6ef112e-kube-api-access-qsxkj\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-2d1c71-2mg9x\" (UID: \"80950513-d45b-4a63-a6b0-91d6a6ef112e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2d1c71-2mg9x"
Oct 30 19:16:01.648315 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:01.648284    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5e80b74-9sj46"
Oct 30 19:16:01.677386 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:01.677032    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f3a3b99-57p29"
Oct 30 19:16:01.747051 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:01.747015    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eda0d3b8-cfbd-44bf-940d-200d449dec9a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eda0d3b8-cfbd-44bf-940d-200d449dec9a-maas-api-openshift-ca\") pod \"eda0d3b8-cfbd-44bf-940d-200d449dec9a\" (UID: \"eda0d3b8-cfbd-44bf-940d-200d449dec9a\") "
Oct 30 19:16:01.747267 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:01.747199    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eda0d3b8-cfbd-44bf-940d-200d449dec9a-kube-api-access-vh6qp\" (UniqueName: \"kubernetes.io/projected/eda0d3b8-cfbd-44bf-940d-200d449dec9a-kube-api-access-vh6qp\") pod \"eda0d3b8-cfbd-44bf-940d-200d449dec9a\" (UID: \"eda0d3b8-cfbd-44bf-940d-200d449dec9a\") "
Oct 30 19:16:01.747267 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:01.747232    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eda0d3b8-cfbd-44bf-940d-200d449dec9a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eda0d3b8-cfbd-44bf-940d-200d449dec9a-maas-api-xks-ca\") pod \"eda0d3b8-cfbd-44bf-940d-200d449dec9a\" (UID: \"eda0d3b8-cfbd-44bf-940d-200d449dec9a\") "
Oct 30 19:16:01.747267 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:01.747265    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70641967-e202-48ef-bdbd-d03939412444-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70641967-e202-48ef-bdbd-d03939412444-maas-api-openshift-ca\") pod \"70641967-e202-48ef-bdbd-d03939412444\" (UID: \"70641967-e202-48ef-bdbd-d03939412444\") "
Oct 30 19:16:01.748071 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:01.747364    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70641967-e202-48ef-bdbd-d03939412444-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70641967-e202-48ef-bdbd-d03939412444-maas-api-xks-ca\") pod \"70641967-e202-48ef-bdbd-d03939412444\" (UID: \"70641967-e202-48ef-bdbd-d03939412444\") "
Oct 30 19:16:01.748071 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:01.747480    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70641967-e202-48ef-bdbd-d03939412444-kube-api-access-lpr2k\" (UniqueName: \"kubernetes.io/projected/70641967-e202-48ef-bdbd-d03939412444-kube-api-access-lpr2k\") pod \"70641967-e202-48ef-bdbd-d03939412444\" (UID: \"70641967-e202-48ef-bdbd-d03939412444\") "
Oct 30 19:16:01.748071 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:01.747763    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70641967-e202-48ef-bdbd-d03939412444-maas-api-openshift-ca" pod "70641967-e202-48ef-bdbd-d03939412444" (UID: "70641967-e202-48ef-bdbd-d03939412444"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:16:01.748071 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:01.747820    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eda0d3b8-cfbd-44bf-940d-200d449dec9a-maas-api-openshift-ca" pod "eda0d3b8-cfbd-44bf-940d-200d449dec9a" (UID: "eda0d3b8-cfbd-44bf-940d-200d449dec9a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:16:01.748370 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:01.748307    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eda0d3b8-cfbd-44bf-940d-200d449dec9a-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:01.748438 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:01.748379    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70641967-e202-48ef-bdbd-d03939412444-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:01.750288 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:01.750256    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70641967-e202-48ef-bdbd-d03939412444-maas-api-xks-ca" pod "70641967-e202-48ef-bdbd-d03939412444" (UID: "70641967-e202-48ef-bdbd-d03939412444"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:16:01.750288 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:01.750269    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eda0d3b8-cfbd-44bf-940d-200d449dec9a-maas-api-xks-ca" pod "eda0d3b8-cfbd-44bf-940d-200d449dec9a" (UID: "eda0d3b8-cfbd-44bf-940d-200d449dec9a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:16:01.750444 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:01.750328    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eda0d3b8-cfbd-44bf-940d-200d449dec9a-kube-api-access-vh6qp" pod "eda0d3b8-cfbd-44bf-940d-200d449dec9a" (UID: "eda0d3b8-cfbd-44bf-940d-200d449dec9a"). InnerVolumeSpecName "kube-api-access-vh6qp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:01.750444 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:01.750365    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70641967-e202-48ef-bdbd-d03939412444-kube-api-access-lpr2k" pod "70641967-e202-48ef-bdbd-d03939412444" (UID: "70641967-e202-48ef-bdbd-d03939412444"). InnerVolumeSpecName "kube-api-access-lpr2k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:01.754419 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:01.754395    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2d1c71-2mg9x"
Oct 30 19:16:01.849959 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:01.849828    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vh6qp\" (UniqueName: \"kubernetes.io/projected/eda0d3b8-cfbd-44bf-940d-200d449dec9a-kube-api-access-vh6qp\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:01.849959 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:01.849863    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eda0d3b8-cfbd-44bf-940d-200d449dec9a-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:01.849959 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:01.849877    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70641967-e202-48ef-bdbd-d03939412444-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:01.849959 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:01.849889    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpr2k\" (UniqueName: \"kubernetes.io/projected/70641967-e202-48ef-bdbd-d03939412444-kube-api-access-lpr2k\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:02.045946 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:02.045911    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2d1c71-2mg9x"]
Oct 30 19:16:02.046112 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:16:02.045968    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80950513_d45b_4a63_a6b0_91d6a6ef112e.slice/crio-26caaa6e9fc6323477314228cf62ba0140edaa620b3f9f7cc266b1bbf58f19f7 WatchSource:0}: Error finding container 26caaa6e9fc6323477314228cf62ba0140edaa620b3f9f7cc266b1bbf58f19f7: Status 404 returned error can't find the container with id 26caaa6e9fc6323477314228cf62ba0140edaa620b3f9f7cc266b1bbf58f19f7
Oct 30 19:16:02.410642 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:02.410615    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f3a3b99-57p29"
Oct 30 19:16:02.410917 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:02.410644    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f3a3b99-57p29" event={"ID":"eda0d3b8-cfbd-44bf-940d-200d449dec9a","Type":"ContainerDied","Data":"b18c897c4a29de38fee894e5553fd8e1ce9afecbee20e8111f2be60015c68c00"}
Oct 30 19:16:02.410917 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:02.410712    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b18c897c4a29de38fee894e5553fd8e1ce9afecbee20e8111f2be60015c68c00"
Oct 30 19:16:02.416870 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:02.414961    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5e80b74-9sj46"
Oct 30 19:16:02.416870 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:02.414963    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5e80b74-9sj46" event={"ID":"70641967-e202-48ef-bdbd-d03939412444","Type":"ContainerDied","Data":"18577161c074e765320c240cf75a68973ff9659e8306e13787db44751880ace4"}
Oct 30 19:16:02.416870 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:02.415116    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18577161c074e765320c240cf75a68973ff9659e8306e13787db44751880ace4"
Oct 30 19:16:02.422147 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:02.422121    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2d1c71-2mg9x" event={"ID":"80950513-d45b-4a63-a6b0-91d6a6ef112e","Type":"ContainerStarted","Data":"544bf66ac646e0898fc6433cf46280e4443e74716d70fecd8977f26fc718a20a"}
Oct 30 19:16:02.422767 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:02.422744    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2d1c71-2mg9x" event={"ID":"80950513-d45b-4a63-a6b0-91d6a6ef112e","Type":"ContainerStarted","Data":"26caaa6e9fc6323477314228cf62ba0140edaa620b3f9f7cc266b1bbf58f19f7"}
Oct 30 19:16:02.468667 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:02.468594    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944"]
Oct 30 19:16:02.469525 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:02.468904    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944" podUID="92a5cb37-4cd6-466c-bf3b-740d3573da07" containerName="istio-proxy" containerID="cri-o://afda9a783e450c5c8cd9b7ccfde31be7702833962128f3da7b465318d4d5c377" gracePeriod=30
Oct 30 19:16:02.479747 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:02.478126    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2d1c71-2mg9x" podStartSLOduration=2.478107326 podStartE2EDuration="2.478107326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:16:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:02.455923478 +0000 UTC m=+2385.081593439" watchObservedRunningTime="2026-10-30 19:16:02.478107326 +0000 UTC m=+2385.103777291"
Oct 30 19:16:03.126293 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:03.126254    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0458df-qqlcw"]
Oct 30 19:16:03.126942 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:03.126924    4525 cpu_manager.go:414] "containerMap: removing container" podUID="eda0d3b8-cfbd-44bf-940d-200d449dec9a" containerName="revoke-keys"
Oct 30 19:16:03.127006 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:03.126944    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eda0d3b8-cfbd-44bf-940d-200d449dec9a" containerName="revoke-keys"
Oct 30 19:16:03.127006 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:03.126968    4525 cpu_manager.go:414] "containerMap: removing container" podUID="70641967-e202-48ef-bdbd-d03939412444" containerName="revoke-keys"
Oct 30 19:16:03.127006 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:03.126974    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70641967-e202-48ef-bdbd-d03939412444" containerName="revoke-keys"
Oct 30 19:16:03.127115 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:03.127102    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="70641967-e202-48ef-bdbd-d03939412444" containerName="revoke-keys"
Oct 30 19:16:03.127174 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:03.127125    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="eda0d3b8-cfbd-44bf-940d-200d449dec9a" containerName="revoke-keys"
Oct 30 19:16:03.131190 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:03.131154    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0458df-qqlcw"
Oct 30 19:16:03.139237 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:03.139204    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0458df-qqlcw"]
Oct 30 19:16:03.253377 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:03.249983    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb03ccc2-1675-482f-a64b-fa7f9bef180f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-0458df-qqlcw\" (UID: \"bb03ccc2-1675-482f-a64b-fa7f9bef180f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0458df-qqlcw"
Oct 30 19:16:03.253377 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:03.250131    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-957t2\" (UniqueName: \"kubernetes.io/projected/bb03ccc2-1675-482f-a64b-fa7f9bef180f-kube-api-access-957t2\") pod \"maas-api-revoke-keys-e2e-worker-w4-0458df-qqlcw\" (UID: \"bb03ccc2-1675-482f-a64b-fa7f9bef180f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0458df-qqlcw"
Oct 30 19:16:03.253377 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:03.250248    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb03ccc2-1675-482f-a64b-fa7f9bef180f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-0458df-qqlcw\" (UID: \"bb03ccc2-1675-482f-a64b-fa7f9bef180f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0458df-qqlcw"
Oct 30 19:16:03.351982 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:03.351885    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb03ccc2-1675-482f-a64b-fa7f9bef180f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-0458df-qqlcw\" (UID: \"bb03ccc2-1675-482f-a64b-fa7f9bef180f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0458df-qqlcw"
Oct 30 19:16:03.352163 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:03.351991    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-957t2\" (UniqueName: \"kubernetes.io/projected/bb03ccc2-1675-482f-a64b-fa7f9bef180f-kube-api-access-957t2\") pod \"maas-api-revoke-keys-e2e-worker-w4-0458df-qqlcw\" (UID: \"bb03ccc2-1675-482f-a64b-fa7f9bef180f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0458df-qqlcw"
Oct 30 19:16:03.352163 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:03.352082    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb03ccc2-1675-482f-a64b-fa7f9bef180f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-0458df-qqlcw\" (UID: \"bb03ccc2-1675-482f-a64b-fa7f9bef180f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0458df-qqlcw"
Oct 30 19:16:03.352712 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:03.352669    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb03ccc2-1675-482f-a64b-fa7f9bef180f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-0458df-qqlcw\" (UID: \"bb03ccc2-1675-482f-a64b-fa7f9bef180f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0458df-qqlcw"
Oct 30 19:16:03.354459 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:03.354436    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb03ccc2-1675-482f-a64b-fa7f9bef180f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-0458df-qqlcw\" (UID: \"bb03ccc2-1675-482f-a64b-fa7f9bef180f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0458df-qqlcw"
Oct 30 19:16:03.359993 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:03.359968    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-957t2\" (UniqueName: \"kubernetes.io/projected/bb03ccc2-1675-482f-a64b-fa7f9bef180f-kube-api-access-957t2\") pod \"maas-api-revoke-keys-e2e-worker-w4-0458df-qqlcw\" (UID: \"bb03ccc2-1675-482f-a64b-fa7f9bef180f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0458df-qqlcw"
Oct 30 19:16:03.432264 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:03.432232    4525 generic.go:356] "Generic (PLEG): container finished" podID="80950513-d45b-4a63-a6b0-91d6a6ef112e" containerID="544bf66ac646e0898fc6433cf46280e4443e74716d70fecd8977f26fc718a20a" exitCode=0
Oct 30 19:16:03.432652 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:03.432314    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2d1c71-2mg9x" event={"ID":"80950513-d45b-4a63-a6b0-91d6a6ef112e","Type":"ContainerDied","Data":"544bf66ac646e0898fc6433cf46280e4443e74716d70fecd8977f26fc718a20a"}
Oct 30 19:16:03.464352 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:03.464319    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0458df-qqlcw"
Oct 30 19:16:03.748593 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:16:03.748562    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb03ccc2_1675_482f_a64b_fa7f9bef180f.slice/crio-b916bd6bffa7cf5fb62c7294de22c0cc6425ca23a67d964034a1ab1ac296bc6b WatchSource:0}: Error finding container b916bd6bffa7cf5fb62c7294de22c0cc6425ca23a67d964034a1ab1ac296bc6b: Status 404 returned error can't find the container with id b916bd6bffa7cf5fb62c7294de22c0cc6425ca23a67d964034a1ab1ac296bc6b
Oct 30 19:16:03.748773 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:03.748712    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0458df-qqlcw"]
Oct 30 19:16:04.148396 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:04.148368    4525 patch_prober.go:29] interesting pod/e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:16:04.148535 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:04.148413    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944" podUID="92a5cb37-4cd6-466c-bf3b-740d3573da07" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:16:04.448654 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:04.448616    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0458df-qqlcw" event={"ID":"bb03ccc2-1675-482f-a64b-fa7f9bef180f","Type":"ContainerStarted","Data":"c87fe44480ae3adb2193a73c22b8ef45b9b214867c218c0dab0bdd579b95a5f1"}
Oct 30 19:16:04.449086 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:04.448672    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0458df-qqlcw" event={"ID":"bb03ccc2-1675-482f-a64b-fa7f9bef180f","Type":"ContainerStarted","Data":"b916bd6bffa7cf5fb62c7294de22c0cc6425ca23a67d964034a1ab1ac296bc6b"}
Oct 30 19:16:04.468232 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:04.468172    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0458df-qqlcw" podStartSLOduration=1.468151684 podStartE2EDuration="1.468151684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:16:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:04.463957521 +0000 UTC m=+2387.089627483" watchObservedRunningTime="2026-10-30 19:16:04.468151684 +0000 UTC m=+2387.093821641"
Oct 30 19:16:04.617204 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:04.617178    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2d1c71-2mg9x"
Oct 30 19:16:04.775758 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:04.773290    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80950513-d45b-4a63-a6b0-91d6a6ef112e-kube-api-access-qsxkj\" (UniqueName: \"kubernetes.io/projected/80950513-d45b-4a63-a6b0-91d6a6ef112e-kube-api-access-qsxkj\") pod \"80950513-d45b-4a63-a6b0-91d6a6ef112e\" (UID: \"80950513-d45b-4a63-a6b0-91d6a6ef112e\") "
Oct 30 19:16:04.775758 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:04.773511    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80950513-d45b-4a63-a6b0-91d6a6ef112e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80950513-d45b-4a63-a6b0-91d6a6ef112e-maas-api-xks-ca\") pod \"80950513-d45b-4a63-a6b0-91d6a6ef112e\" (UID: \"80950513-d45b-4a63-a6b0-91d6a6ef112e\") "
Oct 30 19:16:04.775758 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:04.773603    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80950513-d45b-4a63-a6b0-91d6a6ef112e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80950513-d45b-4a63-a6b0-91d6a6ef112e-maas-api-openshift-ca\") pod \"80950513-d45b-4a63-a6b0-91d6a6ef112e\" (UID: \"80950513-d45b-4a63-a6b0-91d6a6ef112e\") "
Oct 30 19:16:04.775758 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:04.774792    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80950513-d45b-4a63-a6b0-91d6a6ef112e-maas-api-openshift-ca" pod "80950513-d45b-4a63-a6b0-91d6a6ef112e" (UID: "80950513-d45b-4a63-a6b0-91d6a6ef112e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:16:04.778857 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:04.778826    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80950513-d45b-4a63-a6b0-91d6a6ef112e-maas-api-xks-ca" pod "80950513-d45b-4a63-a6b0-91d6a6ef112e" (UID: "80950513-d45b-4a63-a6b0-91d6a6ef112e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:16:04.781511 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:04.781459    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80950513-d45b-4a63-a6b0-91d6a6ef112e-kube-api-access-qsxkj" pod "80950513-d45b-4a63-a6b0-91d6a6ef112e" (UID: "80950513-d45b-4a63-a6b0-91d6a6ef112e"). InnerVolumeSpecName "kube-api-access-qsxkj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:04.875236 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:04.875206    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80950513-d45b-4a63-a6b0-91d6a6ef112e-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:04.875236 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:04.875234    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80950513-d45b-4a63-a6b0-91d6a6ef112e-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:04.875236 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:04.875243    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsxkj\" (UniqueName: \"kubernetes.io/projected/80950513-d45b-4a63-a6b0-91d6a6ef112e-kube-api-access-qsxkj\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:05.459248 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:05.459214    4525 generic.go:356] "Generic (PLEG): container finished" podID="bb03ccc2-1675-482f-a64b-fa7f9bef180f" containerID="c87fe44480ae3adb2193a73c22b8ef45b9b214867c218c0dab0bdd579b95a5f1" exitCode=0
Oct 30 19:16:05.459639 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:05.459291    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0458df-qqlcw" event={"ID":"bb03ccc2-1675-482f-a64b-fa7f9bef180f","Type":"ContainerDied","Data":"c87fe44480ae3adb2193a73c22b8ef45b9b214867c218c0dab0bdd579b95a5f1"}
Oct 30 19:16:05.460827 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:05.460805    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2d1c71-2mg9x" event={"ID":"80950513-d45b-4a63-a6b0-91d6a6ef112e","Type":"ContainerDied","Data":"26caaa6e9fc6323477314228cf62ba0140edaa620b3f9f7cc266b1bbf58f19f7"}
Oct 30 19:16:05.460983 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:05.460839    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2d1c71-2mg9x"
Oct 30 19:16:05.460983 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:05.460843    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26caaa6e9fc6323477314228cf62ba0140edaa620b3f9f7cc266b1bbf58f19f7"
Oct 30 19:16:05.680047 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:05.680024    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-clvlk"
Oct 30 19:16:05.716436 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:05.716402    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e14cca94-de16-4d65-b71a-9ece97424174-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e14cca94-de16-4d65-b71a-9ece97424174-workload-socket\") pod \"e14cca94-de16-4d65-b71a-9ece97424174\" (UID: \"e14cca94-de16-4d65-b71a-9ece97424174\") "
Oct 30 19:16:05.716624 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:05.716487    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e14cca94-de16-4d65-b71a-9ece97424174-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e14cca94-de16-4d65-b71a-9ece97424174-istio-podinfo\") pod \"e14cca94-de16-4d65-b71a-9ece97424174\" (UID: \"e14cca94-de16-4d65-b71a-9ece97424174\") "
Oct 30 19:16:05.716624 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:05.716577    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e14cca94-de16-4d65-b71a-9ece97424174-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e14cca94-de16-4d65-b71a-9ece97424174-istio-envoy\") pod \"e14cca94-de16-4d65-b71a-9ece97424174\" (UID: \"e14cca94-de16-4d65-b71a-9ece97424174\") "
Oct 30 19:16:05.716624 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:05.716610    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e14cca94-de16-4d65-b71a-9ece97424174-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e14cca94-de16-4d65-b71a-9ece97424174-istio-data\") pod \"e14cca94-de16-4d65-b71a-9ece97424174\" (UID: \"e14cca94-de16-4d65-b71a-9ece97424174\") "
Oct 30 19:16:05.716834 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:05.716663    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e14cca94-de16-4d65-b71a-9ece97424174-kube-api-access-qpp2h\" (UniqueName: \"kubernetes.io/projected/e14cca94-de16-4d65-b71a-9ece97424174-kube-api-access-qpp2h\") pod \"e14cca94-de16-4d65-b71a-9ece97424174\" (UID: \"e14cca94-de16-4d65-b71a-9ece97424174\") "
Oct 30 19:16:05.716834 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:05.716670    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e14cca94-de16-4d65-b71a-9ece97424174-workload-socket" pod "e14cca94-de16-4d65-b71a-9ece97424174" (UID: "e14cca94-de16-4d65-b71a-9ece97424174"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:05.716834 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:05.716755    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e14cca94-de16-4d65-b71a-9ece97424174-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e14cca94-de16-4d65-b71a-9ece97424174-credential-socket\") pod \"e14cca94-de16-4d65-b71a-9ece97424174\" (UID: \"e14cca94-de16-4d65-b71a-9ece97424174\") "
Oct 30 19:16:05.716834 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:05.716826    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e14cca94-de16-4d65-b71a-9ece97424174-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e14cca94-de16-4d65-b71a-9ece97424174-workload-certs\") pod \"e14cca94-de16-4d65-b71a-9ece97424174\" (UID: \"e14cca94-de16-4d65-b71a-9ece97424174\") "
Oct 30 19:16:05.717054 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:05.716856    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e14cca94-de16-4d65-b71a-9ece97424174-istio-token\" (UniqueName: \"kubernetes.io/projected/e14cca94-de16-4d65-b71a-9ece97424174-istio-token\") pod \"e14cca94-de16-4d65-b71a-9ece97424174\" (UID: \"e14cca94-de16-4d65-b71a-9ece97424174\") "
Oct 30 19:16:05.717054 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:05.716911    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e14cca94-de16-4d65-b71a-9ece97424174-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e14cca94-de16-4d65-b71a-9ece97424174-istiod-ca-cert\") pod \"e14cca94-de16-4d65-b71a-9ece97424174\" (UID: \"e14cca94-de16-4d65-b71a-9ece97424174\") "
Oct 30 19:16:05.717180 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:05.717159    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e14cca94-de16-4d65-b71a-9ece97424174-workload-certs" pod "e14cca94-de16-4d65-b71a-9ece97424174" (UID: "e14cca94-de16-4d65-b71a-9ece97424174"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:05.717263 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:05.717238    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e14cca94-de16-4d65-b71a-9ece97424174-credential-socket" pod "e14cca94-de16-4d65-b71a-9ece97424174" (UID: "e14cca94-de16-4d65-b71a-9ece97424174"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:05.717339 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:05.717315    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e14cca94-de16-4d65-b71a-9ece97424174-istio-data" pod "e14cca94-de16-4d65-b71a-9ece97424174" (UID: "e14cca94-de16-4d65-b71a-9ece97424174"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:05.717991 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:05.717600    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e14cca94-de16-4d65-b71a-9ece97424174-istiod-ca-cert" pod "e14cca94-de16-4d65-b71a-9ece97424174" (UID: "e14cca94-de16-4d65-b71a-9ece97424174"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:16:05.717991 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:05.717904    4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e14cca94-de16-4d65-b71a-9ece97424174-workload-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:05.717991 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:05.717926    4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e14cca94-de16-4d65-b71a-9ece97424174-istio-data\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:05.717991 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:05.717941    4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e14cca94-de16-4d65-b71a-9ece97424174-credential-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:05.717991 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:05.717954    4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e14cca94-de16-4d65-b71a-9ece97424174-workload-certs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:05.717991 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:05.717968    4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e14cca94-de16-4d65-b71a-9ece97424174-istiod-ca-cert\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:05.720313 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:05.720283    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e14cca94-de16-4d65-b71a-9ece97424174-istio-envoy" pod "e14cca94-de16-4d65-b71a-9ece97424174" (UID: "e14cca94-de16-4d65-b71a-9ece97424174"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:05.720425 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:05.720360    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e14cca94-de16-4d65-b71a-9ece97424174-istio-token" pod "e14cca94-de16-4d65-b71a-9ece97424174" (UID: "e14cca94-de16-4d65-b71a-9ece97424174"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:05.720532 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:05.720509    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e14cca94-de16-4d65-b71a-9ece97424174-istio-podinfo" pod "e14cca94-de16-4d65-b71a-9ece97424174" (UID: "e14cca94-de16-4d65-b71a-9ece97424174"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:16:05.720607 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:05.720553    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e14cca94-de16-4d65-b71a-9ece97424174-kube-api-access-qpp2h" pod "e14cca94-de16-4d65-b71a-9ece97424174" (UID: "e14cca94-de16-4d65-b71a-9ece97424174"). InnerVolumeSpecName "kube-api-access-qpp2h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:05.847203 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:05.847091    4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e14cca94-de16-4d65-b71a-9ece97424174-istio-token\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:05.847203 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:05.847131    4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e14cca94-de16-4d65-b71a-9ece97424174-istio-podinfo\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:05.847203 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:05.847148    4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e14cca94-de16-4d65-b71a-9ece97424174-istio-envoy\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:05.847203 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:05.847174    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpp2h\" (UniqueName: \"kubernetes.io/projected/e14cca94-de16-4d65-b71a-9ece97424174-kube-api-access-qpp2h\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:06.339408 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:06.339372    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4"]
Oct 30 19:16:06.339755 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:06.339709    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4" podUID="0650ae0c-36c1-4b6e-9212-12264a92c920" containerName="istio-proxy" containerID="cri-o://0bc4d925a818f8e98248567d5c9c1915c6cd0b32a9fb2e98b6959553ed535f14" gracePeriod=30
Oct 30 19:16:06.568653 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:06.568616    4525 generic.go:356] "Generic (PLEG): container finished" podID="e14cca94-de16-4d65-b71a-9ece97424174" containerID="51099112565d75045a8d9cb6d3d3c02280322816fd0cdf6169c2a059fadb714a" exitCode=0
Oct 30 19:16:06.570095 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:06.570066    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-clvlk"
Oct 30 19:16:06.584292 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:06.584252    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-clvlk" event={"ID":"e14cca94-de16-4d65-b71a-9ece97424174","Type":"ContainerDied","Data":"51099112565d75045a8d9cb6d3d3c02280322816fd0cdf6169c2a059fadb714a"}
Oct 30 19:16:06.585245 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:06.584519    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-clvlk" event={"ID":"e14cca94-de16-4d65-b71a-9ece97424174","Type":"ContainerDied","Data":"0075f3b7c884c36045ca13159bc0b9f395bdcd346ceb7206da73f9037134521c"}
Oct 30 19:16:06.585245 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:06.584561    4525 scope.go:122] "RemoveContainer" containerID="51099112565d75045a8d9cb6d3d3c02280322816fd0cdf6169c2a059fadb714a"
Oct 30 19:16:06.655568 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:06.655545    4525 scope.go:122] "RemoveContainer" containerID="60d23ddc31d508eaa5eb17f954bf9625d80aeae99677e0445494225b7535eb60"
Oct 30 19:16:06.659836 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:06.659815    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-clvlk"]
Oct 30 19:16:06.667077 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:06.667051    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-c2e3f9-openshift-default-697cdcbdfc-clvlk"]
Oct 30 19:16:06.672895 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:06.672878    4525 scope.go:122] "RemoveContainer" containerID="51099112565d75045a8d9cb6d3d3c02280322816fd0cdf6169c2a059fadb714a"
Oct 30 19:16:06.673237 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:16:06.673218    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51099112565d75045a8d9cb6d3d3c02280322816fd0cdf6169c2a059fadb714a\": container with ID starting with 51099112565d75045a8d9cb6d3d3c02280322816fd0cdf6169c2a059fadb714a not found: ID does not exist" containerID="51099112565d75045a8d9cb6d3d3c02280322816fd0cdf6169c2a059fadb714a"
Oct 30 19:16:06.673327 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:06.673244    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51099112565d75045a8d9cb6d3d3c02280322816fd0cdf6169c2a059fadb714a"} err="failed to get container status \"51099112565d75045a8d9cb6d3d3c02280322816fd0cdf6169c2a059fadb714a\": rpc error: code = NotFound desc = could not find container \"51099112565d75045a8d9cb6d3d3c02280322816fd0cdf6169c2a059fadb714a\": container with ID starting with 51099112565d75045a8d9cb6d3d3c02280322816fd0cdf6169c2a059fadb714a not found: ID does not exist"
Oct 30 19:16:06.673327 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:06.673260    4525 scope.go:122] "RemoveContainer" containerID="60d23ddc31d508eaa5eb17f954bf9625d80aeae99677e0445494225b7535eb60"
Oct 30 19:16:06.674069 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:16:06.673632    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60d23ddc31d508eaa5eb17f954bf9625d80aeae99677e0445494225b7535eb60\": container with ID starting with 60d23ddc31d508eaa5eb17f954bf9625d80aeae99677e0445494225b7535eb60 not found: ID does not exist" containerID="60d23ddc31d508eaa5eb17f954bf9625d80aeae99677e0445494225b7535eb60"
Oct 30 19:16:06.674069 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:06.673669    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60d23ddc31d508eaa5eb17f954bf9625d80aeae99677e0445494225b7535eb60"} err="failed to get container status \"60d23ddc31d508eaa5eb17f954bf9625d80aeae99677e0445494225b7535eb60\": rpc error: code = NotFound desc = could not find container \"60d23ddc31d508eaa5eb17f954bf9625d80aeae99677e0445494225b7535eb60\": container with ID starting with 60d23ddc31d508eaa5eb17f954bf9625d80aeae99677e0445494225b7535eb60 not found: ID does not exist"
Oct 30 19:16:06.801182 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:06.801157    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0458df-qqlcw"
Oct 30 19:16:06.840226 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:06.840187    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a1d55d-sz5qn"]
Oct 30 19:16:06.844933 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:06.841149    4525 cpu_manager.go:414] "containerMap: removing container" podUID="80950513-d45b-4a63-a6b0-91d6a6ef112e" containerName="revoke-keys"
Oct 30 19:16:06.844933 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:06.841171    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80950513-d45b-4a63-a6b0-91d6a6ef112e" containerName="revoke-keys"
Oct 30 19:16:06.844933 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:06.841198    4525 cpu_manager.go:414] "containerMap: removing container" podUID="e14cca94-de16-4d65-b71a-9ece97424174" containerName="istio-proxy"
Oct 30 19:16:06.844933 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:06.841206    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e14cca94-de16-4d65-b71a-9ece97424174" containerName="istio-proxy"
Oct 30 19:16:06.844933 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:06.841292    4525 cpu_manager.go:414] "containerMap: removing container" podUID="e14cca94-de16-4d65-b71a-9ece97424174" containerName="istio-proxy"
Oct 30 19:16:06.844933 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:06.841300    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e14cca94-de16-4d65-b71a-9ece97424174" containerName="istio-proxy"
Oct 30 19:16:06.844933 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:06.841315    4525 cpu_manager.go:414] "containerMap: removing container" podUID="bb03ccc2-1675-482f-a64b-fa7f9bef180f" containerName="revoke-keys"
Oct 30 19:16:06.844933 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:06.841323    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb03ccc2-1675-482f-a64b-fa7f9bef180f" containerName="revoke-keys"
Oct 30 19:16:06.844933 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:06.841446    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="e14cca94-de16-4d65-b71a-9ece97424174" containerName="istio-proxy"
Oct 30 19:16:06.844933 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:06.841456    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb03ccc2-1675-482f-a64b-fa7f9bef180f" containerName="revoke-keys"
Oct 30 19:16:06.844933 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:06.841471    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="80950513-d45b-4a63-a6b0-91d6a6ef112e" containerName="revoke-keys"
Oct 30 19:16:06.844933 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:06.841484    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="e14cca94-de16-4d65-b71a-9ece97424174" containerName="istio-proxy"
Oct 30 19:16:06.859983 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:06.859935    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a1d55d-sz5qn"]
Oct 30 19:16:06.860156 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:06.860092    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a1d55d-sz5qn"
Oct 30 19:16:06.889955 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:06.889925    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb03ccc2-1675-482f-a64b-fa7f9bef180f-kube-api-access-957t2\" (UniqueName: \"kubernetes.io/projected/bb03ccc2-1675-482f-a64b-fa7f9bef180f-kube-api-access-957t2\") pod \"bb03ccc2-1675-482f-a64b-fa7f9bef180f\" (UID: \"bb03ccc2-1675-482f-a64b-fa7f9bef180f\") "
Oct 30 19:16:06.890184 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:06.890007    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb03ccc2-1675-482f-a64b-fa7f9bef180f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb03ccc2-1675-482f-a64b-fa7f9bef180f-maas-api-xks-ca\") pod \"bb03ccc2-1675-482f-a64b-fa7f9bef180f\" (UID: \"bb03ccc2-1675-482f-a64b-fa7f9bef180f\") "
Oct 30 19:16:06.890184 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:06.890156    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb03ccc2-1675-482f-a64b-fa7f9bef180f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb03ccc2-1675-482f-a64b-fa7f9bef180f-maas-api-openshift-ca\") pod \"bb03ccc2-1675-482f-a64b-fa7f9bef180f\" (UID: \"bb03ccc2-1675-482f-a64b-fa7f9bef180f\") "
Oct 30 19:16:06.890536 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:06.890508    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d779\" (UniqueName: \"kubernetes.io/projected/4d396afe-ec50-4729-bcbb-20256faa39e0-kube-api-access-8d779\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-a1d55d-sz5qn\" (UID: \"4d396afe-ec50-4729-bcbb-20256faa39e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a1d55d-sz5qn"
Oct 30 19:16:06.890740 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:06.890711    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb03ccc2-1675-482f-a64b-fa7f9bef180f-maas-api-openshift-ca" pod "bb03ccc2-1675-482f-a64b-fa7f9bef180f" (UID: "bb03ccc2-1675-482f-a64b-fa7f9bef180f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:16:06.890838 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:06.890798    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d396afe-ec50-4729-bcbb-20256faa39e0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-a1d55d-sz5qn\" (UID: \"4d396afe-ec50-4729-bcbb-20256faa39e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a1d55d-sz5qn"
Oct 30 19:16:06.891086 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:06.891060    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d396afe-ec50-4729-bcbb-20256faa39e0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-a1d55d-sz5qn\" (UID: \"4d396afe-ec50-4729-bcbb-20256faa39e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a1d55d-sz5qn"
Oct 30 19:16:06.891253 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:06.891234    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb03ccc2-1675-482f-a64b-fa7f9bef180f-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:06.892964 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:06.892942    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb03ccc2-1675-482f-a64b-fa7f9bef180f-kube-api-access-957t2" pod "bb03ccc2-1675-482f-a64b-fa7f9bef180f" (UID: "bb03ccc2-1675-482f-a64b-fa7f9bef180f"). InnerVolumeSpecName "kube-api-access-957t2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:06.893891 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:06.893867    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb03ccc2-1675-482f-a64b-fa7f9bef180f-maas-api-xks-ca" pod "bb03ccc2-1675-482f-a64b-fa7f9bef180f" (UID: "bb03ccc2-1675-482f-a64b-fa7f9bef180f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:16:06.992712 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:06.992643    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d779\" (UniqueName: \"kubernetes.io/projected/4d396afe-ec50-4729-bcbb-20256faa39e0-kube-api-access-8d779\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-a1d55d-sz5qn\" (UID: \"4d396afe-ec50-4729-bcbb-20256faa39e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a1d55d-sz5qn"
Oct 30 19:16:06.992903 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:06.992786    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d396afe-ec50-4729-bcbb-20256faa39e0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-a1d55d-sz5qn\" (UID: \"4d396afe-ec50-4729-bcbb-20256faa39e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a1d55d-sz5qn"
Oct 30 19:16:06.992970 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:06.992953    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d396afe-ec50-4729-bcbb-20256faa39e0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-a1d55d-sz5qn\" (UID: \"4d396afe-ec50-4729-bcbb-20256faa39e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a1d55d-sz5qn"
Oct 30 19:16:06.993055 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:06.993042    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-957t2\" (UniqueName: \"kubernetes.io/projected/bb03ccc2-1675-482f-a64b-fa7f9bef180f-kube-api-access-957t2\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:06.993092 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:06.993060    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb03ccc2-1675-482f-a64b-fa7f9bef180f-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:06.993421 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:06.993392    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d396afe-ec50-4729-bcbb-20256faa39e0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-a1d55d-sz5qn\" (UID: \"4d396afe-ec50-4729-bcbb-20256faa39e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a1d55d-sz5qn"
Oct 30 19:16:06.995454 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:06.995432    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d396afe-ec50-4729-bcbb-20256faa39e0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-a1d55d-sz5qn\" (UID: \"4d396afe-ec50-4729-bcbb-20256faa39e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a1d55d-sz5qn"
Oct 30 19:16:07.000926 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:07.000907    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d779\" (UniqueName: \"kubernetes.io/projected/4d396afe-ec50-4729-bcbb-20256faa39e0-kube-api-access-8d779\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-a1d55d-sz5qn\" (UID: \"4d396afe-ec50-4729-bcbb-20256faa39e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a1d55d-sz5qn"
Oct 30 19:16:07.177753 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:07.177628    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a1d55d-sz5qn"
Oct 30 19:16:07.464797 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:16:07.464768    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d396afe_ec50_4729_bcbb_20256faa39e0.slice/crio-0a9d59de5d0e82fa09f98c0908f2da13deb71fae97c45ac602ccd704b62bceae WatchSource:0}: Error finding container 0a9d59de5d0e82fa09f98c0908f2da13deb71fae97c45ac602ccd704b62bceae: Status 404 returned error can't find the container with id 0a9d59de5d0e82fa09f98c0908f2da13deb71fae97c45ac602ccd704b62bceae
Oct 30 19:16:07.464942 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:07.464910    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a1d55d-sz5qn"]
Oct 30 19:16:07.547996 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:07.547961    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e14cca94-de16-4d65-b71a-9ece97424174" path="/var/lib/kubelet/pods/e14cca94-de16-4d65-b71a-9ece97424174/volumes"
Oct 30 19:16:07.581047 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:07.581013    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a1d55d-sz5qn" event={"ID":"4d396afe-ec50-4729-bcbb-20256faa39e0","Type":"ContainerStarted","Data":"0a9d59de5d0e82fa09f98c0908f2da13deb71fae97c45ac602ccd704b62bceae"}
Oct 30 19:16:07.582549 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:07.582524    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0458df-qqlcw"
Oct 30 19:16:07.582549 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:07.582532    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0458df-qqlcw" event={"ID":"bb03ccc2-1675-482f-a64b-fa7f9bef180f","Type":"ContainerDied","Data":"b916bd6bffa7cf5fb62c7294de22c0cc6425ca23a67d964034a1ab1ac296bc6b"}
Oct 30 19:16:07.582740 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:07.582578    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b916bd6bffa7cf5fb62c7294de22c0cc6425ca23a67d964034a1ab1ac296bc6b"
Oct 30 19:16:07.849865 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:07.849838    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944"
Oct 30 19:16:07.928872 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:07.928839    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92a5cb37-4cd6-466c-bf3b-740d3573da07-istio-token\" (UniqueName: \"kubernetes.io/projected/92a5cb37-4cd6-466c-bf3b-740d3573da07-istio-token\") pod \"92a5cb37-4cd6-466c-bf3b-740d3573da07\" (UID: \"92a5cb37-4cd6-466c-bf3b-740d3573da07\") "
Oct 30 19:16:07.929055 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:07.928891    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92a5cb37-4cd6-466c-bf3b-740d3573da07-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92a5cb37-4cd6-466c-bf3b-740d3573da07-istio-data\") pod \"92a5cb37-4cd6-466c-bf3b-740d3573da07\" (UID: \"92a5cb37-4cd6-466c-bf3b-740d3573da07\") "
Oct 30 19:16:07.929055 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:07.928945    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92a5cb37-4cd6-466c-bf3b-740d3573da07-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92a5cb37-4cd6-466c-bf3b-740d3573da07-istiod-ca-cert\") pod \"92a5cb37-4cd6-466c-bf3b-740d3573da07\" (UID: \"92a5cb37-4cd6-466c-bf3b-740d3573da07\") "
Oct 30 19:16:07.929055 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:07.929024    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92a5cb37-4cd6-466c-bf3b-740d3573da07-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92a5cb37-4cd6-466c-bf3b-740d3573da07-credential-socket\") pod \"92a5cb37-4cd6-466c-bf3b-740d3573da07\" (UID: \"92a5cb37-4cd6-466c-bf3b-740d3573da07\") "
Oct 30 19:16:07.929239 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:07.929075    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92a5cb37-4cd6-466c-bf3b-740d3573da07-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92a5cb37-4cd6-466c-bf3b-740d3573da07-workload-certs\") pod \"92a5cb37-4cd6-466c-bf3b-740d3573da07\" (UID: \"92a5cb37-4cd6-466c-bf3b-740d3573da07\") "
Oct 30 19:16:07.929316 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:07.929287    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92a5cb37-4cd6-466c-bf3b-740d3573da07-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92a5cb37-4cd6-466c-bf3b-740d3573da07-workload-socket\") pod \"92a5cb37-4cd6-466c-bf3b-740d3573da07\" (UID: \"92a5cb37-4cd6-466c-bf3b-740d3573da07\") "
Oct 30 19:16:07.929381 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:07.929359    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92a5cb37-4cd6-466c-bf3b-740d3573da07-kube-api-access-zhkdd\" (UniqueName: \"kubernetes.io/projected/92a5cb37-4cd6-466c-bf3b-740d3573da07-kube-api-access-zhkdd\") pod \"92a5cb37-4cd6-466c-bf3b-740d3573da07\" (UID: \"92a5cb37-4cd6-466c-bf3b-740d3573da07\") "
Oct 30 19:16:07.929450 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:07.929399    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/92a5cb37-4cd6-466c-bf3b-740d3573da07-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92a5cb37-4cd6-466c-bf3b-740d3573da07-istio-podinfo\") pod \"92a5cb37-4cd6-466c-bf3b-740d3573da07\" (UID: \"92a5cb37-4cd6-466c-bf3b-740d3573da07\") "
Oct 30 19:16:07.930733 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:07.930631    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92a5cb37-4cd6-466c-bf3b-740d3573da07-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92a5cb37-4cd6-466c-bf3b-740d3573da07-istio-envoy\") pod \"92a5cb37-4cd6-466c-bf3b-740d3573da07\" (UID: \"92a5cb37-4cd6-466c-bf3b-740d3573da07\") "
Oct 30 19:16:07.931846 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:07.931822    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92a5cb37-4cd6-466c-bf3b-740d3573da07-istio-data" pod "92a5cb37-4cd6-466c-bf3b-740d3573da07" (UID: "92a5cb37-4cd6-466c-bf3b-740d3573da07"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:07.931991 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:07.931885    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92a5cb37-4cd6-466c-bf3b-740d3573da07-istiod-ca-cert" pod "92a5cb37-4cd6-466c-bf3b-740d3573da07" (UID: "92a5cb37-4cd6-466c-bf3b-740d3573da07"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:16:07.932251 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:07.932207    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92a5cb37-4cd6-466c-bf3b-740d3573da07-workload-socket" pod "92a5cb37-4cd6-466c-bf3b-740d3573da07" (UID: "92a5cb37-4cd6-466c-bf3b-740d3573da07"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:07.932364 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:07.932265    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92a5cb37-4cd6-466c-bf3b-740d3573da07-kube-api-access-zhkdd" pod "92a5cb37-4cd6-466c-bf3b-740d3573da07" (UID: "92a5cb37-4cd6-466c-bf3b-740d3573da07"). InnerVolumeSpecName "kube-api-access-zhkdd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:07.932613 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:07.932580    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92a5cb37-4cd6-466c-bf3b-740d3573da07-workload-certs" pod "92a5cb37-4cd6-466c-bf3b-740d3573da07" (UID: "92a5cb37-4cd6-466c-bf3b-740d3573da07"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:07.933220 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:07.933192    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92a5cb37-4cd6-466c-bf3b-740d3573da07-credential-socket" pod "92a5cb37-4cd6-466c-bf3b-740d3573da07" (UID: "92a5cb37-4cd6-466c-bf3b-740d3573da07"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:07.934781 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:07.934754    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92a5cb37-4cd6-466c-bf3b-740d3573da07-istio-token" pod "92a5cb37-4cd6-466c-bf3b-740d3573da07" (UID: "92a5cb37-4cd6-466c-bf3b-740d3573da07"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:07.935073 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:07.935032    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/92a5cb37-4cd6-466c-bf3b-740d3573da07-istio-podinfo" pod "92a5cb37-4cd6-466c-bf3b-740d3573da07" (UID: "92a5cb37-4cd6-466c-bf3b-740d3573da07"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:16:07.935485 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:07.935462    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92a5cb37-4cd6-466c-bf3b-740d3573da07-istio-envoy" pod "92a5cb37-4cd6-466c-bf3b-740d3573da07" (UID: "92a5cb37-4cd6-466c-bf3b-740d3573da07"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:08.032534 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:08.032456    4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92a5cb37-4cd6-466c-bf3b-740d3573da07-workload-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:08.032534 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:08.032484    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhkdd\" (UniqueName: \"kubernetes.io/projected/92a5cb37-4cd6-466c-bf3b-740d3573da07-kube-api-access-zhkdd\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:08.032534 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:08.032499    4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92a5cb37-4cd6-466c-bf3b-740d3573da07-istio-podinfo\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:08.032534 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:08.032514    4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92a5cb37-4cd6-466c-bf3b-740d3573da07-istio-envoy\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:08.032534 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:08.032522    4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92a5cb37-4cd6-466c-bf3b-740d3573da07-istio-token\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:08.032534 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:08.032530    4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92a5cb37-4cd6-466c-bf3b-740d3573da07-istio-data\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:08.032534 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:08.032538    4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92a5cb37-4cd6-466c-bf3b-740d3573da07-istiod-ca-cert\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:08.032900 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:08.032546    4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92a5cb37-4cd6-466c-bf3b-740d3573da07-credential-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:08.032900 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:08.032554    4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92a5cb37-4cd6-466c-bf3b-740d3573da07-workload-certs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:08.592579 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:08.592548    4525 generic.go:356] "Generic (PLEG): container finished" podID="4d396afe-ec50-4729-bcbb-20256faa39e0" containerID="c3399818326c129136a987629374bc6042c3792ddb42d4c5a876a5b05d4c65ec" exitCode=0
Oct 30 19:16:08.593028 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:08.592639    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a1d55d-sz5qn" event={"ID":"4d396afe-ec50-4729-bcbb-20256faa39e0","Type":"ContainerDied","Data":"c3399818326c129136a987629374bc6042c3792ddb42d4c5a876a5b05d4c65ec"}
Oct 30 19:16:08.594550 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:08.594525    4525 generic.go:356] "Generic (PLEG): container finished" podID="92a5cb37-4cd6-466c-bf3b-740d3573da07" containerID="afda9a783e450c5c8cd9b7ccfde31be7702833962128f3da7b465318d4d5c377" exitCode=0
Oct 30 19:16:08.594728 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:08.594566    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944" event={"ID":"92a5cb37-4cd6-466c-bf3b-740d3573da07","Type":"ContainerDied","Data":"afda9a783e450c5c8cd9b7ccfde31be7702833962128f3da7b465318d4d5c377"}
Oct 30 19:16:08.594728 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:08.594585    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944"
Oct 30 19:16:08.594728 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:08.594601    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944" event={"ID":"92a5cb37-4cd6-466c-bf3b-740d3573da07","Type":"ContainerDied","Data":"f1b4e445fb70c4a339b84efa71bd83fdf020173ece75d4a93046b4907a453994"}
Oct 30 19:16:08.594728 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:08.594625    4525 scope.go:122] "RemoveContainer" containerID="afda9a783e450c5c8cd9b7ccfde31be7702833962128f3da7b465318d4d5c377"
Oct 30 19:16:08.606749 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:08.606729    4525 scope.go:122] "RemoveContainer" containerID="2ff7347f12d88897fab512adef677fc96a7de0cae3bce0381e7b5ac746082424"
Oct 30 19:16:08.618825 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:08.618800    4525 scope.go:122] "RemoveContainer" containerID="afda9a783e450c5c8cd9b7ccfde31be7702833962128f3da7b465318d4d5c377"
Oct 30 19:16:08.619130 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:16:08.619114    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afda9a783e450c5c8cd9b7ccfde31be7702833962128f3da7b465318d4d5c377\": container with ID starting with afda9a783e450c5c8cd9b7ccfde31be7702833962128f3da7b465318d4d5c377 not found: ID does not exist" containerID="afda9a783e450c5c8cd9b7ccfde31be7702833962128f3da7b465318d4d5c377"
Oct 30 19:16:08.619207 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:08.619152    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afda9a783e450c5c8cd9b7ccfde31be7702833962128f3da7b465318d4d5c377"} err="failed to get container status \"afda9a783e450c5c8cd9b7ccfde31be7702833962128f3da7b465318d4d5c377\": rpc error: code = NotFound desc = could not find container \"afda9a783e450c5c8cd9b7ccfde31be7702833962128f3da7b465318d4d5c377\": container with ID starting with afda9a783e450c5c8cd9b7ccfde31be7702833962128f3da7b465318d4d5c377 not found: ID does not exist"
Oct 30 19:16:08.619207 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:08.619167    4525 scope.go:122] "RemoveContainer" containerID="2ff7347f12d88897fab512adef677fc96a7de0cae3bce0381e7b5ac746082424"
Oct 30 19:16:08.619444 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:16:08.619426    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ff7347f12d88897fab512adef677fc96a7de0cae3bce0381e7b5ac746082424\": container with ID starting with 2ff7347f12d88897fab512adef677fc96a7de0cae3bce0381e7b5ac746082424 not found: ID does not exist" containerID="2ff7347f12d88897fab512adef677fc96a7de0cae3bce0381e7b5ac746082424"
Oct 30 19:16:08.619493 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:08.619454    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ff7347f12d88897fab512adef677fc96a7de0cae3bce0381e7b5ac746082424"} err="failed to get container status \"2ff7347f12d88897fab512adef677fc96a7de0cae3bce0381e7b5ac746082424\": rpc error: code = NotFound desc = could not find container \"2ff7347f12d88897fab512adef677fc96a7de0cae3bce0381e7b5ac746082424\": container with ID starting with 2ff7347f12d88897fab512adef677fc96a7de0cae3bce0381e7b5ac746082424 not found: ID does not exist"
Oct 30 19:16:08.626120 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:08.626097    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944"]
Oct 30 19:16:08.630856 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:08.630834    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-ed012e-openshift-default-6ddfc957bc-lc944"]
Oct 30 19:16:09.546622 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:09.546581    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92a5cb37-4cd6-466c-bf3b-740d3573da07" path="/var/lib/kubelet/pods/92a5cb37-4cd6-466c-bf3b-740d3573da07/volumes"
Oct 30 19:16:09.963844 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:09.963817    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9de872-xcgq7"]
Oct 30 19:16:09.974007 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:09.973973    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a1d55d-sz5qn"
Oct 30 19:16:09.981029 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:09.981006    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9de872-xcgq7"]
Oct 30 19:16:10.086823 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:10.086731    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d396afe-ec50-4729-bcbb-20256faa39e0-kube-api-access-8d779\" (UniqueName: \"kubernetes.io/projected/4d396afe-ec50-4729-bcbb-20256faa39e0-kube-api-access-8d779\") pod \"4d396afe-ec50-4729-bcbb-20256faa39e0\" (UID: \"4d396afe-ec50-4729-bcbb-20256faa39e0\") "
Oct 30 19:16:10.086823 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:10.086788    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d396afe-ec50-4729-bcbb-20256faa39e0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d396afe-ec50-4729-bcbb-20256faa39e0-maas-api-xks-ca\") pod \"4d396afe-ec50-4729-bcbb-20256faa39e0\" (UID: \"4d396afe-ec50-4729-bcbb-20256faa39e0\") "
Oct 30 19:16:10.086823 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:10.086821    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d396afe-ec50-4729-bcbb-20256faa39e0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d396afe-ec50-4729-bcbb-20256faa39e0-maas-api-openshift-ca\") pod \"4d396afe-ec50-4729-bcbb-20256faa39e0\" (UID: \"4d396afe-ec50-4729-bcbb-20256faa39e0\") "
Oct 30 19:16:10.087276 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:10.087253    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d396afe-ec50-4729-bcbb-20256faa39e0-maas-api-openshift-ca" pod "4d396afe-ec50-4729-bcbb-20256faa39e0" (UID: "4d396afe-ec50-4729-bcbb-20256faa39e0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:16:10.087468 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:10.087451    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d396afe-ec50-4729-bcbb-20256faa39e0-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:10.088927 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:10.088905    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d396afe-ec50-4729-bcbb-20256faa39e0-kube-api-access-8d779" pod "4d396afe-ec50-4729-bcbb-20256faa39e0" (UID: "4d396afe-ec50-4729-bcbb-20256faa39e0"). InnerVolumeSpecName "kube-api-access-8d779". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:10.089047 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:10.089029    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d396afe-ec50-4729-bcbb-20256faa39e0-maas-api-xks-ca" pod "4d396afe-ec50-4729-bcbb-20256faa39e0" (UID: "4d396afe-ec50-4729-bcbb-20256faa39e0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:16:10.188476 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:10.188441    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8d779\" (UniqueName: \"kubernetes.io/projected/4d396afe-ec50-4729-bcbb-20256faa39e0-kube-api-access-8d779\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:10.188476 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:10.188471    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d396afe-ec50-4729-bcbb-20256faa39e0-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:10.612943 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:10.612909    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a1d55d-sz5qn"
Oct 30 19:16:10.613152 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:10.612985    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a1d55d-sz5qn" event={"ID":"4d396afe-ec50-4729-bcbb-20256faa39e0","Type":"ContainerDied","Data":"0a9d59de5d0e82fa09f98c0908f2da13deb71fae97c45ac602ccd704b62bceae"}
Oct 30 19:16:10.613152 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:10.613029    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a9d59de5d0e82fa09f98c0908f2da13deb71fae97c45ac602ccd704b62bceae"
Oct 30 19:16:11.560704 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:11.560166    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c830b89-50cc-46ea-909d-5f1127534794" path="/var/lib/kubelet/pods/5c830b89-50cc-46ea-909d-5f1127534794/volumes"
Oct 30 19:16:11.636711 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:11.634215    4525 generic.go:356] "Generic (PLEG): container finished" podID="0650ae0c-36c1-4b6e-9212-12264a92c920" containerID="0bc4d925a818f8e98248567d5c9c1915c6cd0b32a9fb2e98b6959553ed535f14" exitCode=0
Oct 30 19:16:11.636711 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:11.634618    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4" event={"ID":"0650ae0c-36c1-4b6e-9212-12264a92c920","Type":"ContainerDied","Data":"0bc4d925a818f8e98248567d5c9c1915c6cd0b32a9fb2e98b6959553ed535f14"}
Oct 30 19:16:11.636711 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:11.634707    4525 scope.go:122] "RemoveContainer" containerID="ddde0bd43d503e1026196638a207f5250f5d53e85a6bd1119d79bc22736da3a3"
Oct 30 19:16:11.787870 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:11.787843    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4"
Oct 30 19:16:11.918166 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:11.918081    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0650ae0c-36c1-4b6e-9212-12264a92c920-kube-api-access-wjnk5\" (UniqueName: \"kubernetes.io/projected/0650ae0c-36c1-4b6e-9212-12264a92c920-kube-api-access-wjnk5\") pod \"0650ae0c-36c1-4b6e-9212-12264a92c920\" (UID: \"0650ae0c-36c1-4b6e-9212-12264a92c920\") "
Oct 30 19:16:11.918166 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:11.918159    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0650ae0c-36c1-4b6e-9212-12264a92c920-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0650ae0c-36c1-4b6e-9212-12264a92c920-istio-envoy\") pod \"0650ae0c-36c1-4b6e-9212-12264a92c920\" (UID: \"0650ae0c-36c1-4b6e-9212-12264a92c920\") "
Oct 30 19:16:11.918372 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:11.918231    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0650ae0c-36c1-4b6e-9212-12264a92c920-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0650ae0c-36c1-4b6e-9212-12264a92c920-istiod-ca-cert\") pod \"0650ae0c-36c1-4b6e-9212-12264a92c920\" (UID: \"0650ae0c-36c1-4b6e-9212-12264a92c920\") "
Oct 30 19:16:11.918372 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:11.918285    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0650ae0c-36c1-4b6e-9212-12264a92c920-istio-token\" (UniqueName: \"kubernetes.io/projected/0650ae0c-36c1-4b6e-9212-12264a92c920-istio-token\") pod \"0650ae0c-36c1-4b6e-9212-12264a92c920\" (UID: \"0650ae0c-36c1-4b6e-9212-12264a92c920\") "
Oct 30 19:16:11.918372 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:11.918317    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0650ae0c-36c1-4b6e-9212-12264a92c920-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0650ae0c-36c1-4b6e-9212-12264a92c920-istio-data\") pod \"0650ae0c-36c1-4b6e-9212-12264a92c920\" (UID: \"0650ae0c-36c1-4b6e-9212-12264a92c920\") "
Oct 30 19:16:11.918528 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:11.918370    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0650ae0c-36c1-4b6e-9212-12264a92c920-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0650ae0c-36c1-4b6e-9212-12264a92c920-workload-socket\") pod \"0650ae0c-36c1-4b6e-9212-12264a92c920\" (UID: \"0650ae0c-36c1-4b6e-9212-12264a92c920\") "
Oct 30 19:16:11.918528 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:11.918392    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0650ae0c-36c1-4b6e-9212-12264a92c920-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0650ae0c-36c1-4b6e-9212-12264a92c920-credential-socket\") pod \"0650ae0c-36c1-4b6e-9212-12264a92c920\" (UID: \"0650ae0c-36c1-4b6e-9212-12264a92c920\") "
Oct 30 19:16:11.918646 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:11.918530    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0650ae0c-36c1-4b6e-9212-12264a92c920-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0650ae0c-36c1-4b6e-9212-12264a92c920-workload-certs\") pod \"0650ae0c-36c1-4b6e-9212-12264a92c920\" (UID: \"0650ae0c-36c1-4b6e-9212-12264a92c920\") "
Oct 30 19:16:11.918646 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:11.918575    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0650ae0c-36c1-4b6e-9212-12264a92c920-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0650ae0c-36c1-4b6e-9212-12264a92c920-istio-podinfo\") pod \"0650ae0c-36c1-4b6e-9212-12264a92c920\" (UID: \"0650ae0c-36c1-4b6e-9212-12264a92c920\") "
Oct 30 19:16:11.918822 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:11.918724    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0650ae0c-36c1-4b6e-9212-12264a92c920-istio-data" pod "0650ae0c-36c1-4b6e-9212-12264a92c920" (UID: "0650ae0c-36c1-4b6e-9212-12264a92c920"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:11.918822 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:11.918743    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0650ae0c-36c1-4b6e-9212-12264a92c920-istiod-ca-cert" pod "0650ae0c-36c1-4b6e-9212-12264a92c920" (UID: "0650ae0c-36c1-4b6e-9212-12264a92c920"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:16:11.918941 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:11.918873    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0650ae0c-36c1-4b6e-9212-12264a92c920-credential-socket" pod "0650ae0c-36c1-4b6e-9212-12264a92c920" (UID: "0650ae0c-36c1-4b6e-9212-12264a92c920"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:11.919009 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:11.918969    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0650ae0c-36c1-4b6e-9212-12264a92c920-workload-socket" pod "0650ae0c-36c1-4b6e-9212-12264a92c920" (UID: "0650ae0c-36c1-4b6e-9212-12264a92c920"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:11.919009 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:11.918985    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0650ae0c-36c1-4b6e-9212-12264a92c920-workload-certs" pod "0650ae0c-36c1-4b6e-9212-12264a92c920" (UID: "0650ae0c-36c1-4b6e-9212-12264a92c920"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:11.919296 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:11.919278    4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0650ae0c-36c1-4b6e-9212-12264a92c920-workload-certs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:11.919367 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:11.919302    4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0650ae0c-36c1-4b6e-9212-12264a92c920-istiod-ca-cert\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:11.919367 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:11.919315    4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0650ae0c-36c1-4b6e-9212-12264a92c920-istio-data\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:11.919367 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:11.919329    4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0650ae0c-36c1-4b6e-9212-12264a92c920-workload-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:11.919367 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:11.919342    4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0650ae0c-36c1-4b6e-9212-12264a92c920-credential-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:11.920568 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:11.920539    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0650ae0c-36c1-4b6e-9212-12264a92c920-kube-api-access-wjnk5" pod "0650ae0c-36c1-4b6e-9212-12264a92c920" (UID: "0650ae0c-36c1-4b6e-9212-12264a92c920"). InnerVolumeSpecName "kube-api-access-wjnk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:11.920984 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:11.920958    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0650ae0c-36c1-4b6e-9212-12264a92c920-istio-podinfo" pod "0650ae0c-36c1-4b6e-9212-12264a92c920" (UID: "0650ae0c-36c1-4b6e-9212-12264a92c920"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:16:11.921159 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:11.921142    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0650ae0c-36c1-4b6e-9212-12264a92c920-istio-envoy" pod "0650ae0c-36c1-4b6e-9212-12264a92c920" (UID: "0650ae0c-36c1-4b6e-9212-12264a92c920"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:11.921212 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:11.921165    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0650ae0c-36c1-4b6e-9212-12264a92c920-istio-token" pod "0650ae0c-36c1-4b6e-9212-12264a92c920" (UID: "0650ae0c-36c1-4b6e-9212-12264a92c920"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:12.021543 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:12.021509    4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0650ae0c-36c1-4b6e-9212-12264a92c920-istio-podinfo\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:12.021543 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:12.021541    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjnk5\" (UniqueName: \"kubernetes.io/projected/0650ae0c-36c1-4b6e-9212-12264a92c920-kube-api-access-wjnk5\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:12.021774 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:12.021552    4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0650ae0c-36c1-4b6e-9212-12264a92c920-istio-envoy\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:12.021774 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:12.021561    4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0650ae0c-36c1-4b6e-9212-12264a92c920-istio-token\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:12.645909 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:12.645877    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4"
Oct 30 19:16:12.645909 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:12.645895    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4" event={"ID":"0650ae0c-36c1-4b6e-9212-12264a92c920","Type":"ContainerDied","Data":"62c6d60a54184c45b9f3bee2bc6561554c860c80a8ce722ce63b658a934f1184"}
Oct 30 19:16:12.646522 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:12.645971    4525 scope.go:122] "RemoveContainer" containerID="0bc4d925a818f8e98248567d5c9c1915c6cd0b32a9fb2e98b6959553ed535f14"
Oct 30 19:16:12.675590 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:12.675561    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4"]
Oct 30 19:16:12.685140 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:12.685111    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-4a477d-openshift-default-66bfdb8649-cj8w4"]
Oct 30 19:16:13.546276 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:13.546242    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0650ae0c-36c1-4b6e-9212-12264a92c920" path="/var/lib/kubelet/pods/0650ae0c-36c1-4b6e-9212-12264a92c920/volumes"
Oct 30 19:16:17.467236 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:17.465750    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-856e5336-gxwh4"]
Oct 30 19:16:17.467236 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:17.466765    4525 cpu_manager.go:414] "containerMap: removing container" podUID="0650ae0c-36c1-4b6e-9212-12264a92c920" containerName="istio-proxy"
Oct 30 19:16:17.467236 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:17.466788    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0650ae0c-36c1-4b6e-9212-12264a92c920" containerName="istio-proxy"
Oct 30 19:16:17.467236 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:17.466810    4525 cpu_manager.go:414] "containerMap: removing container" podUID="4d396afe-ec50-4729-bcbb-20256faa39e0" containerName="revoke-keys"
Oct 30 19:16:17.467236 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:17.466817    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d396afe-ec50-4729-bcbb-20256faa39e0" containerName="revoke-keys"
Oct 30 19:16:17.467236 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:17.466838    4525 cpu_manager.go:414] "containerMap: removing container" podUID="92a5cb37-4cd6-466c-bf3b-740d3573da07" containerName="istio-proxy"
Oct 30 19:16:17.467236 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:17.466846    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92a5cb37-4cd6-466c-bf3b-740d3573da07" containerName="istio-proxy"
Oct 30 19:16:17.467236 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:17.466877    4525 cpu_manager.go:414] "containerMap: removing container" podUID="92a5cb37-4cd6-466c-bf3b-740d3573da07" containerName="istio-proxy"
Oct 30 19:16:17.467236 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:17.466884    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92a5cb37-4cd6-466c-bf3b-740d3573da07" containerName="istio-proxy"
Oct 30 19:16:17.467236 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:17.467033    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="0650ae0c-36c1-4b6e-9212-12264a92c920" containerName="istio-proxy"
Oct 30 19:16:17.467236 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:17.467047    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d396afe-ec50-4729-bcbb-20256faa39e0" containerName="revoke-keys"
Oct 30 19:16:17.467236 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:17.467060    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="92a5cb37-4cd6-466c-bf3b-740d3573da07" containerName="istio-proxy"
Oct 30 19:16:17.467236 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:17.467070    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="0650ae0c-36c1-4b6e-9212-12264a92c920" containerName="istio-proxy"
Oct 30 19:16:17.491702 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:17.480406    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-856e5336-gxwh4"
Oct 30 19:16:17.513755 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:17.499937    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:16:17.513755 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:17.511897    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94hcn\""
Oct 30 19:16:17.513755 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:17.512189    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-743e4446-jtlb9"]
Oct 30 19:16:17.513755 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:17.513376    4525 cpu_manager.go:414] "containerMap: removing container" podUID="0650ae0c-36c1-4b6e-9212-12264a92c920" containerName="istio-proxy"
Oct 30 19:16:17.513755 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:17.513398    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0650ae0c-36c1-4b6e-9212-12264a92c920" containerName="istio-proxy"
Oct 30 19:16:17.513755 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:17.513558    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="92a5cb37-4cd6-466c-bf3b-740d3573da07" containerName="istio-proxy"
Oct 30 19:16:17.588087 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:16:17.585076    4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d49l" containerName="extractor"
Oct 30 19:16:17.646795 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:17.633627    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-743e4446-jtlb9"
Oct 30 19:16:17.714703 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:17.687282    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-856e5336-gxwh4"]
Oct 30 19:16:17.857747 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:17.829044    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e25fa0f4-be7f-4c10-83f8-ce8b479668cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-856e5336-gxwh4\" (UID: \"e25fa0f4-be7f-4c10-83f8-ce8b479668cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-856e5336-gxwh4"
Oct 30 19:16:17.857747 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:17.829116    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f675df4-123a-412a-8aa3-5a61497b06bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-743e4446-jtlb9\" (UID: \"0f675df4-123a-412a-8aa3-5a61497b06bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-743e4446-jtlb9"
Oct 30 19:16:17.857747 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:17.829146    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67lwz\" (UniqueName: \"kubernetes.io/projected/0f675df4-123a-412a-8aa3-5a61497b06bd-kube-api-access-67lwz\") pod \"maas-api-revoke-sub-743e4446-jtlb9\" (UID: \"0f675df4-123a-412a-8aa3-5a61497b06bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-743e4446-jtlb9"
Oct 30 19:16:17.857747 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:17.829225    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e25fa0f4-be7f-4c10-83f8-ce8b479668cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-856e5336-gxwh4\" (UID: \"e25fa0f4-be7f-4c10-83f8-ce8b479668cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-856e5336-gxwh4"
Oct 30 19:16:17.857747 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:17.829387    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2stp\" (UniqueName: \"kubernetes.io/projected/e25fa0f4-be7f-4c10-83f8-ce8b479668cb-kube-api-access-b2stp\") pod \"maas-api-revoke-sub-856e5336-gxwh4\" (UID: \"e25fa0f4-be7f-4c10-83f8-ce8b479668cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-856e5336-gxwh4"
Oct 30 19:16:17.857747 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:17.829443    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f675df4-123a-412a-8aa3-5a61497b06bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-743e4446-jtlb9\" (UID: \"0f675df4-123a-412a-8aa3-5a61497b06bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-743e4446-jtlb9"
Oct 30 19:16:17.866980 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:17.866947    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-743e4446-jtlb9"]
Oct 30 19:16:17.919661 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:17.919624    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-txf2f"]
Oct 30 19:16:17.920258 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:17.920224    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-txf2f" podUID="f94f5889-fce0-46bc-a055-1c5e5e51d6a6" containerName="istio-proxy" containerID="cri-o://bc2cbbfe6b46c07059660f1c6a406daa26774e90ab7a95fb4c51c6ab06012878" gracePeriod=30
Oct 30 19:16:17.924901 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:17.924874    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-zv786"]
Oct 30 19:16:17.925142 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:17.925112    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-zv786" podUID="4f3c674d-b2c1-4a9a-afa1-5bfd33999e99" containerName="istio-proxy" containerID="cri-o://e1d959ee8e2f4a6eba684605cf5f2200ae0605953dff477153b7b806f5df67ae" gracePeriod=30
Oct 30 19:16:17.930730 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:17.930674    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f675df4-123a-412a-8aa3-5a61497b06bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-743e4446-jtlb9\" (UID: \"0f675df4-123a-412a-8aa3-5a61497b06bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-743e4446-jtlb9"
Oct 30 19:16:17.930851 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:17.930736    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67lwz\" (UniqueName: \"kubernetes.io/projected/0f675df4-123a-412a-8aa3-5a61497b06bd-kube-api-access-67lwz\") pod \"maas-api-revoke-sub-743e4446-jtlb9\" (UID: \"0f675df4-123a-412a-8aa3-5a61497b06bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-743e4446-jtlb9"
Oct 30 19:16:17.930851 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:17.930806    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e25fa0f4-be7f-4c10-83f8-ce8b479668cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-856e5336-gxwh4\" (UID: \"e25fa0f4-be7f-4c10-83f8-ce8b479668cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-856e5336-gxwh4"
Oct 30 19:16:17.930963 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:17.930926    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2stp\" (UniqueName: \"kubernetes.io/projected/e25fa0f4-be7f-4c10-83f8-ce8b479668cb-kube-api-access-b2stp\") pod \"maas-api-revoke-sub-856e5336-gxwh4\" (UID: \"e25fa0f4-be7f-4c10-83f8-ce8b479668cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-856e5336-gxwh4"
Oct 30 19:16:17.931019 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:17.930973    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f675df4-123a-412a-8aa3-5a61497b06bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-743e4446-jtlb9\" (UID: \"0f675df4-123a-412a-8aa3-5a61497b06bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-743e4446-jtlb9"
Oct 30 19:16:17.931131 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:17.931114    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e25fa0f4-be7f-4c10-83f8-ce8b479668cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-856e5336-gxwh4\" (UID: \"e25fa0f4-be7f-4c10-83f8-ce8b479668cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-856e5336-gxwh4"
Oct 30 19:16:17.931673 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:17.931651    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f675df4-123a-412a-8aa3-5a61497b06bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-743e4446-jtlb9\" (UID: \"0f675df4-123a-412a-8aa3-5a61497b06bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-743e4446-jtlb9"
Oct 30 19:16:17.931673 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:17.931664    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e25fa0f4-be7f-4c10-83f8-ce8b479668cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-856e5336-gxwh4\" (UID: \"e25fa0f4-be7f-4c10-83f8-ce8b479668cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-856e5336-gxwh4"
Oct 30 19:16:17.933278 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:17.933260    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:16:17.943405 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:17.943375    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e25fa0f4-be7f-4c10-83f8-ce8b479668cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-856e5336-gxwh4\" (UID: \"e25fa0f4-be7f-4c10-83f8-ce8b479668cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-856e5336-gxwh4"
Oct 30 19:16:17.943769 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:17.943665    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f675df4-123a-412a-8aa3-5a61497b06bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-743e4446-jtlb9\" (UID: \"0f675df4-123a-412a-8aa3-5a61497b06bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-743e4446-jtlb9"
Oct 30 19:16:17.944932 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:17.944913    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67lwz\" (UniqueName: \"kubernetes.io/projected/0f675df4-123a-412a-8aa3-5a61497b06bd-kube-api-access-67lwz\") pod \"maas-api-revoke-sub-743e4446-jtlb9\" (UID: \"0f675df4-123a-412a-8aa3-5a61497b06bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-743e4446-jtlb9"
Oct 30 19:16:17.946563 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:17.946544    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2stp\" (UniqueName: \"kubernetes.io/projected/e25fa0f4-be7f-4c10-83f8-ce8b479668cb-kube-api-access-b2stp\") pod \"maas-api-revoke-sub-856e5336-gxwh4\" (UID: \"e25fa0f4-be7f-4c10-83f8-ce8b479668cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-856e5336-gxwh4"
Oct 30 19:16:18.131015 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:18.130976    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94hcn\""
Oct 30 19:16:18.139330 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:18.139299    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-743e4446-jtlb9"
Oct 30 19:16:18.167964 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:18.167933    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-856e5336-gxwh4"
Oct 30 19:16:18.432912 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:16:18.432873    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f675df4_123a_412a_8aa3_5a61497b06bd.slice/crio-1b89030b0ea40e765ecbf3fdf4585596313bb2e9aef1c64c5132c2df7524f1e9 WatchSource:0}: Error finding container 1b89030b0ea40e765ecbf3fdf4585596313bb2e9aef1c64c5132c2df7524f1e9: Status 404 returned error can't find the container with id 1b89030b0ea40e765ecbf3fdf4585596313bb2e9aef1c64c5132c2df7524f1e9
Oct 30 19:16:18.435258 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:18.435231    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-743e4446-jtlb9"]
Oct 30 19:16:18.484354 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:18.484326    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-856e5336-gxwh4"]
Oct 30 19:16:18.580104 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:16:18.580068    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode25fa0f4_be7f_4c10_83f8_ce8b479668cb.slice/crio-17cd5fbb7d72ddbe3aa28168ef870fa184e4b6949d5c93d19c3eb68b7218199a WatchSource:0}: Error finding container 17cd5fbb7d72ddbe3aa28168ef870fa184e4b6949d5c93d19c3eb68b7218199a: Status 404 returned error can't find the container with id 17cd5fbb7d72ddbe3aa28168ef870fa184e4b6949d5c93d19c3eb68b7218199a
Oct 30 19:16:18.879722 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:18.879665    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-743e4446-jtlb9" event={"ID":"0f675df4-123a-412a-8aa3-5a61497b06bd","Type":"ContainerStarted","Data":"87e6b8bf22f5dc305509fa83bbe73f98f4ef86ef1e3ca55813274e5e3b16253e"}
Oct 30 19:16:18.880129 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:18.879747    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-743e4446-jtlb9" event={"ID":"0f675df4-123a-412a-8aa3-5a61497b06bd","Type":"ContainerStarted","Data":"1b89030b0ea40e765ecbf3fdf4585596313bb2e9aef1c64c5132c2df7524f1e9"}
Oct 30 19:16:18.881261 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:18.881238    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-856e5336-gxwh4" event={"ID":"e25fa0f4-be7f-4c10-83f8-ce8b479668cb","Type":"ContainerStarted","Data":"17cd5fbb7d72ddbe3aa28168ef870fa184e4b6949d5c93d19c3eb68b7218199a"}
Oct 30 19:16:18.900377 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:18.900324    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-743e4446-jtlb9" podStartSLOduration=1.900309503 podStartE2EDuration="1.900309503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:16:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:18.896806584 +0000 UTC m=+2401.522476562" watchObservedRunningTime="2026-10-30 19:16:18.900309503 +0000 UTC m=+2401.525979518"
Oct 30 19:16:19.836076 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:19.835985    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2d1c71-2mg9x"]
Oct 30 19:16:19.840382 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:19.840351    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2d1c71-2mg9x"]
Oct 30 19:16:19.890898 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:19.890865    4525 generic.go:356] "Generic (PLEG): container finished" podID="e25fa0f4-be7f-4c10-83f8-ce8b479668cb" containerID="186c10b9d1f7005cdcdbaea25146f0583398737658135265d50bb55fcd8d3943" exitCode=0
Oct 30 19:16:19.891097 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:19.890952    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-856e5336-gxwh4" event={"ID":"e25fa0f4-be7f-4c10-83f8-ce8b479668cb","Type":"ContainerDied","Data":"186c10b9d1f7005cdcdbaea25146f0583398737658135265d50bb55fcd8d3943"}
Oct 30 19:16:19.892621 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:19.892600    4525 generic.go:356] "Generic (PLEG): container finished" podID="0f675df4-123a-412a-8aa3-5a61497b06bd" containerID="87e6b8bf22f5dc305509fa83bbe73f98f4ef86ef1e3ca55813274e5e3b16253e" exitCode=0
Oct 30 19:16:19.892770 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:19.892634    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-743e4446-jtlb9" event={"ID":"0f675df4-123a-412a-8aa3-5a61497b06bd","Type":"ContainerDied","Data":"87e6b8bf22f5dc305509fa83bbe73f98f4ef86ef1e3ca55813274e5e3b16253e"}
Oct 30 19:16:20.516604 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:20.515722    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ca2cd5-6nfd4"]
Oct 30 19:16:20.523411 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:20.523388    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ca2cd5-6nfd4"
Oct 30 19:16:20.529655 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:20.529624    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ca2cd5-6nfd4"]
Oct 30 19:16:20.641229 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:20.641195    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cdebbe2-6dd6-4516-9ada-71b5083bd266-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-ca2cd5-6nfd4\" (UID: \"0cdebbe2-6dd6-4516-9ada-71b5083bd266\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ca2cd5-6nfd4"
Oct 30 19:16:20.641421 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:20.641236    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cdebbe2-6dd6-4516-9ada-71b5083bd266-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-ca2cd5-6nfd4\" (UID: \"0cdebbe2-6dd6-4516-9ada-71b5083bd266\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ca2cd5-6nfd4"
Oct 30 19:16:20.641485 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:20.641446    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnbgz\" (UniqueName: \"kubernetes.io/projected/0cdebbe2-6dd6-4516-9ada-71b5083bd266-kube-api-access-tnbgz\") pod \"maas-api-revoke-keys-e2e-extmod-a-ca2cd5-6nfd4\" (UID: \"0cdebbe2-6dd6-4516-9ada-71b5083bd266\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ca2cd5-6nfd4"
Oct 30 19:16:20.742870 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:20.742835    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cdebbe2-6dd6-4516-9ada-71b5083bd266-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-ca2cd5-6nfd4\" (UID: \"0cdebbe2-6dd6-4516-9ada-71b5083bd266\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ca2cd5-6nfd4"
Oct 30 19:16:20.743056 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:20.742879    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cdebbe2-6dd6-4516-9ada-71b5083bd266-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-ca2cd5-6nfd4\" (UID: \"0cdebbe2-6dd6-4516-9ada-71b5083bd266\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ca2cd5-6nfd4"
Oct 30 19:16:20.743056 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:20.743001    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnbgz\" (UniqueName: \"kubernetes.io/projected/0cdebbe2-6dd6-4516-9ada-71b5083bd266-kube-api-access-tnbgz\") pod \"maas-api-revoke-keys-e2e-extmod-a-ca2cd5-6nfd4\" (UID: \"0cdebbe2-6dd6-4516-9ada-71b5083bd266\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ca2cd5-6nfd4"
Oct 30 19:16:20.744211 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:20.744181    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cdebbe2-6dd6-4516-9ada-71b5083bd266-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-ca2cd5-6nfd4\" (UID: \"0cdebbe2-6dd6-4516-9ada-71b5083bd266\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ca2cd5-6nfd4"
Oct 30 19:16:20.752541 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:20.752492    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cdebbe2-6dd6-4516-9ada-71b5083bd266-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-ca2cd5-6nfd4\" (UID: \"0cdebbe2-6dd6-4516-9ada-71b5083bd266\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ca2cd5-6nfd4"
Oct 30 19:16:20.755842 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:20.755762    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnbgz\" (UniqueName: \"kubernetes.io/projected/0cdebbe2-6dd6-4516-9ada-71b5083bd266-kube-api-access-tnbgz\") pod \"maas-api-revoke-keys-e2e-extmod-a-ca2cd5-6nfd4\" (UID: \"0cdebbe2-6dd6-4516-9ada-71b5083bd266\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ca2cd5-6nfd4"
Oct 30 19:16:20.881484 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:20.881458    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ca2cd5-6nfd4"
Oct 30 19:16:21.235985 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:21.235950    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a1d55d-sz5qn"]
Oct 30 19:16:21.244184 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:21.244148    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a1d55d-sz5qn"]
Oct 30 19:16:21.290441 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:21.290415    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ca2cd5-6nfd4"]
Oct 30 19:16:21.330872 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:21.330852    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-856e5336-gxwh4"
Oct 30 19:16:21.333858 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:21.333826    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-743e4446-jtlb9"
Oct 30 19:16:21.468831 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:21.468796    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e25fa0f4-be7f-4c10-83f8-ce8b479668cb-kube-api-access-b2stp\" (UniqueName: \"kubernetes.io/projected/e25fa0f4-be7f-4c10-83f8-ce8b479668cb-kube-api-access-b2stp\") pod \"e25fa0f4-be7f-4c10-83f8-ce8b479668cb\" (UID: \"e25fa0f4-be7f-4c10-83f8-ce8b479668cb\") "
Oct 30 19:16:21.469013 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:21.468919    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e25fa0f4-be7f-4c10-83f8-ce8b479668cb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e25fa0f4-be7f-4c10-83f8-ce8b479668cb-maas-api-xks-ca\") pod \"e25fa0f4-be7f-4c10-83f8-ce8b479668cb\" (UID: \"e25fa0f4-be7f-4c10-83f8-ce8b479668cb\") "
Oct 30 19:16:21.469013 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:21.468995    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e25fa0f4-be7f-4c10-83f8-ce8b479668cb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e25fa0f4-be7f-4c10-83f8-ce8b479668cb-maas-api-openshift-ca\") pod \"e25fa0f4-be7f-4c10-83f8-ce8b479668cb\" (UID: \"e25fa0f4-be7f-4c10-83f8-ce8b479668cb\") "
Oct 30 19:16:21.469138 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:21.469060    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f675df4-123a-412a-8aa3-5a61497b06bd-kube-api-access-67lwz\" (UniqueName: \"kubernetes.io/projected/0f675df4-123a-412a-8aa3-5a61497b06bd-kube-api-access-67lwz\") pod \"0f675df4-123a-412a-8aa3-5a61497b06bd\" (UID: \"0f675df4-123a-412a-8aa3-5a61497b06bd\") "
Oct 30 19:16:21.469219 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:21.469200    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f675df4-123a-412a-8aa3-5a61497b06bd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f675df4-123a-412a-8aa3-5a61497b06bd-maas-api-xks-ca\") pod \"0f675df4-123a-412a-8aa3-5a61497b06bd\" (UID: \"0f675df4-123a-412a-8aa3-5a61497b06bd\") "
Oct 30 19:16:21.469291 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:21.469276    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f675df4-123a-412a-8aa3-5a61497b06bd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f675df4-123a-412a-8aa3-5a61497b06bd-maas-api-openshift-ca\") pod \"0f675df4-123a-412a-8aa3-5a61497b06bd\" (UID: \"0f675df4-123a-412a-8aa3-5a61497b06bd\") "
Oct 30 19:16:21.469409 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:21.469356    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e25fa0f4-be7f-4c10-83f8-ce8b479668cb-maas-api-openshift-ca" pod "e25fa0f4-be7f-4c10-83f8-ce8b479668cb" (UID: "e25fa0f4-be7f-4c10-83f8-ce8b479668cb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:16:21.469887 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:21.469793    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f675df4-123a-412a-8aa3-5a61497b06bd-maas-api-openshift-ca" pod "0f675df4-123a-412a-8aa3-5a61497b06bd" (UID: "0f675df4-123a-412a-8aa3-5a61497b06bd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:16:21.469973 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:21.469908    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e25fa0f4-be7f-4c10-83f8-ce8b479668cb-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:21.471428 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:21.471396    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e25fa0f4-be7f-4c10-83f8-ce8b479668cb-kube-api-access-b2stp" pod "e25fa0f4-be7f-4c10-83f8-ce8b479668cb" (UID: "e25fa0f4-be7f-4c10-83f8-ce8b479668cb"). InnerVolumeSpecName "kube-api-access-b2stp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:21.471904 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:21.471852    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e25fa0f4-be7f-4c10-83f8-ce8b479668cb-maas-api-xks-ca" pod "e25fa0f4-be7f-4c10-83f8-ce8b479668cb" (UID: "e25fa0f4-be7f-4c10-83f8-ce8b479668cb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:16:21.472001 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:21.471959    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f675df4-123a-412a-8aa3-5a61497b06bd-kube-api-access-67lwz" pod "0f675df4-123a-412a-8aa3-5a61497b06bd" (UID: "0f675df4-123a-412a-8aa3-5a61497b06bd"). InnerVolumeSpecName "kube-api-access-67lwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:21.472059 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:21.472037    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f675df4-123a-412a-8aa3-5a61497b06bd-maas-api-xks-ca" pod "0f675df4-123a-412a-8aa3-5a61497b06bd" (UID: "0f675df4-123a-412a-8aa3-5a61497b06bd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:16:21.546015 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:21.545976    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d396afe-ec50-4729-bcbb-20256faa39e0" path="/var/lib/kubelet/pods/4d396afe-ec50-4729-bcbb-20256faa39e0/volumes"
Oct 30 19:16:21.546580 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:21.546557    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80950513-d45b-4a63-a6b0-91d6a6ef112e" path="/var/lib/kubelet/pods/80950513-d45b-4a63-a6b0-91d6a6ef112e/volumes"
Oct 30 19:16:21.571178 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:21.571155    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f675df4-123a-412a-8aa3-5a61497b06bd-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:21.571178 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:21.571179    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f675df4-123a-412a-8aa3-5a61497b06bd-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:21.571309 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:21.571189    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2stp\" (UniqueName: \"kubernetes.io/projected/e25fa0f4-be7f-4c10-83f8-ce8b479668cb-kube-api-access-b2stp\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:21.571309 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:21.571200    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e25fa0f4-be7f-4c10-83f8-ce8b479668cb-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:21.571309 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:21.571209    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67lwz\" (UniqueName: \"kubernetes.io/projected/0f675df4-123a-412a-8aa3-5a61497b06bd-kube-api-access-67lwz\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:21.915087 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:21.915054    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-743e4446-jtlb9"
Oct 30 19:16:21.915087 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:21.915075    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-743e4446-jtlb9" event={"ID":"0f675df4-123a-412a-8aa3-5a61497b06bd","Type":"ContainerDied","Data":"1b89030b0ea40e765ecbf3fdf4585596313bb2e9aef1c64c5132c2df7524f1e9"}
Oct 30 19:16:21.915588 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:21.915131    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b89030b0ea40e765ecbf3fdf4585596313bb2e9aef1c64c5132c2df7524f1e9"
Oct 30 19:16:21.916692 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:21.916656    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-856e5336-gxwh4"
Oct 30 19:16:21.916692 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:21.916655    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-856e5336-gxwh4" event={"ID":"e25fa0f4-be7f-4c10-83f8-ce8b479668cb","Type":"ContainerDied","Data":"17cd5fbb7d72ddbe3aa28168ef870fa184e4b6949d5c93d19c3eb68b7218199a"}
Oct 30 19:16:21.916889 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:21.916733    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17cd5fbb7d72ddbe3aa28168ef870fa184e4b6949d5c93d19c3eb68b7218199a"
Oct 30 19:16:21.918737 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:21.918711    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ca2cd5-6nfd4" event={"ID":"0cdebbe2-6dd6-4516-9ada-71b5083bd266","Type":"ContainerStarted","Data":"79dee914536e253993659426223988f33948650bbc884664e7d2a0bef3b84684"}
Oct 30 19:16:21.918863 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:21.918760    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ca2cd5-6nfd4" event={"ID":"0cdebbe2-6dd6-4516-9ada-71b5083bd266","Type":"ContainerStarted","Data":"fb5da2698540f9ee458a58eb6fb602c846b8d82bee331f7a99f7b58d9f917d9d"}
Oct 30 19:16:21.970200 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:21.970153    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ca2cd5-6nfd4" podStartSLOduration=1.970141737 podStartE2EDuration="1.970141737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:16:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:21.966651276 +0000 UTC m=+2404.592321245" watchObservedRunningTime="2026-10-30 19:16:21.970141737 +0000 UTC m=+2404.595811746"
Oct 30 19:16:22.053733 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:22.053649    4525 patch_prober.go:29] interesting pod/e2e-worker-w0-9de872-openshift-default-77cf8df555-zv786 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:16:22.053857 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:22.053733    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-zv786" podUID="4f3c674d-b2c1-4a9a-afa1-5bfd33999e99" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:16:22.931891 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:22.931856    4525 generic.go:356] "Generic (PLEG): container finished" podID="0cdebbe2-6dd6-4516-9ada-71b5083bd266" containerID="79dee914536e253993659426223988f33948650bbc884664e7d2a0bef3b84684" exitCode=0
Oct 30 19:16:22.932362 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:22.931946    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ca2cd5-6nfd4" event={"ID":"0cdebbe2-6dd6-4516-9ada-71b5083bd266","Type":"ContainerDied","Data":"79dee914536e253993659426223988f33948650bbc884664e7d2a0bef3b84684"}
Oct 30 19:16:23.213877 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:23.213851    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-txf2f"
Oct 30 19:16:23.308270 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:23.300070    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-zv786"
Oct 30 19:16:23.308476 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:23.308311    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f94f5889-fce0-46bc-a055-1c5e5e51d6a6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f94f5889-fce0-46bc-a055-1c5e5e51d6a6-istiod-ca-cert\") pod \"f94f5889-fce0-46bc-a055-1c5e5e51d6a6\" (UID: \"f94f5889-fce0-46bc-a055-1c5e5e51d6a6\") "
Oct 30 19:16:23.308476 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:23.308358    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f94f5889-fce0-46bc-a055-1c5e5e51d6a6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f94f5889-fce0-46bc-a055-1c5e5e51d6a6-credential-socket\") pod \"f94f5889-fce0-46bc-a055-1c5e5e51d6a6\" (UID: \"f94f5889-fce0-46bc-a055-1c5e5e51d6a6\") "
Oct 30 19:16:23.308599 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:23.308480    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f94f5889-fce0-46bc-a055-1c5e5e51d6a6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f94f5889-fce0-46bc-a055-1c5e5e51d6a6-workload-certs\") pod \"f94f5889-fce0-46bc-a055-1c5e5e51d6a6\" (UID: \"f94f5889-fce0-46bc-a055-1c5e5e51d6a6\") "
Oct 30 19:16:23.308599 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:23.308538    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f94f5889-fce0-46bc-a055-1c5e5e51d6a6-kube-api-access-jfptz\" (UniqueName: \"kubernetes.io/projected/f94f5889-fce0-46bc-a055-1c5e5e51d6a6-kube-api-access-jfptz\") pod \"f94f5889-fce0-46bc-a055-1c5e5e51d6a6\" (UID: \"f94f5889-fce0-46bc-a055-1c5e5e51d6a6\") "
Oct 30 19:16:23.308737 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:23.308602    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f94f5889-fce0-46bc-a055-1c5e5e51d6a6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f94f5889-fce0-46bc-a055-1c5e5e51d6a6-istio-podinfo\") pod \"f94f5889-fce0-46bc-a055-1c5e5e51d6a6\" (UID: \"f94f5889-fce0-46bc-a055-1c5e5e51d6a6\") "
Oct 30 19:16:23.308798 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:23.308668    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f94f5889-fce0-46bc-a055-1c5e5e51d6a6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f94f5889-fce0-46bc-a055-1c5e5e51d6a6-workload-socket\") pod \"f94f5889-fce0-46bc-a055-1c5e5e51d6a6\" (UID: \"f94f5889-fce0-46bc-a055-1c5e5e51d6a6\") "
Oct 30 19:16:23.308871 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:23.308817    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f94f5889-fce0-46bc-a055-1c5e5e51d6a6-istio-token\" (UniqueName: \"kubernetes.io/projected/f94f5889-fce0-46bc-a055-1c5e5e51d6a6-istio-token\") pod \"f94f5889-fce0-46bc-a055-1c5e5e51d6a6\" (UID: \"f94f5889-fce0-46bc-a055-1c5e5e51d6a6\") "
Oct 30 19:16:23.308871 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:23.308852    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f94f5889-fce0-46bc-a055-1c5e5e51d6a6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f94f5889-fce0-46bc-a055-1c5e5e51d6a6-istio-data\") pod \"f94f5889-fce0-46bc-a055-1c5e5e51d6a6\" (UID: \"f94f5889-fce0-46bc-a055-1c5e5e51d6a6\") "
Oct 30 19:16:23.308989 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:23.308965    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f94f5889-fce0-46bc-a055-1c5e5e51d6a6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f94f5889-fce0-46bc-a055-1c5e5e51d6a6-istio-envoy\") pod \"f94f5889-fce0-46bc-a055-1c5e5e51d6a6\" (UID: \"f94f5889-fce0-46bc-a055-1c5e5e51d6a6\") "
Oct 30 19:16:23.310237 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:23.310149    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f94f5889-fce0-46bc-a055-1c5e5e51d6a6-workload-socket" pod "f94f5889-fce0-46bc-a055-1c5e5e51d6a6" (UID: "f94f5889-fce0-46bc-a055-1c5e5e51d6a6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:23.311106 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:23.311070    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f94f5889-fce0-46bc-a055-1c5e5e51d6a6-istiod-ca-cert" pod "f94f5889-fce0-46bc-a055-1c5e5e51d6a6" (UID: "f94f5889-fce0-46bc-a055-1c5e5e51d6a6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:16:23.311445 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:23.311353    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f94f5889-fce0-46bc-a055-1c5e5e51d6a6-istio-data" pod "f94f5889-fce0-46bc-a055-1c5e5e51d6a6" (UID: "f94f5889-fce0-46bc-a055-1c5e5e51d6a6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:23.311445 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:23.311420    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f94f5889-fce0-46bc-a055-1c5e5e51d6a6-workload-certs" pod "f94f5889-fce0-46bc-a055-1c5e5e51d6a6" (UID: "f94f5889-fce0-46bc-a055-1c5e5e51d6a6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:23.311646 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:23.311611    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f94f5889-fce0-46bc-a055-1c5e5e51d6a6-credential-socket" pod "f94f5889-fce0-46bc-a055-1c5e5e51d6a6" (UID: "f94f5889-fce0-46bc-a055-1c5e5e51d6a6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:23.315120 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:23.315077    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f94f5889-fce0-46bc-a055-1c5e5e51d6a6-istio-token" pod "f94f5889-fce0-46bc-a055-1c5e5e51d6a6" (UID: "f94f5889-fce0-46bc-a055-1c5e5e51d6a6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:23.315555 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:23.315270    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f94f5889-fce0-46bc-a055-1c5e5e51d6a6-istio-podinfo" pod "f94f5889-fce0-46bc-a055-1c5e5e51d6a6" (UID: "f94f5889-fce0-46bc-a055-1c5e5e51d6a6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:16:23.316118 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:23.316098    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f94f5889-fce0-46bc-a055-1c5e5e51d6a6-istio-envoy" pod "f94f5889-fce0-46bc-a055-1c5e5e51d6a6" (UID: "f94f5889-fce0-46bc-a055-1c5e5e51d6a6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:23.327296 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:23.327253    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f94f5889-fce0-46bc-a055-1c5e5e51d6a6-kube-api-access-jfptz" pod "f94f5889-fce0-46bc-a055-1c5e5e51d6a6" (UID: "f94f5889-fce0-46bc-a055-1c5e5e51d6a6"). InnerVolumeSpecName "kube-api-access-jfptz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:23.411324 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:23.411295    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f3c674d-b2c1-4a9a-afa1-5bfd33999e99-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f3c674d-b2c1-4a9a-afa1-5bfd33999e99-istio-envoy\") pod \"4f3c674d-b2c1-4a9a-afa1-5bfd33999e99\" (UID: \"4f3c674d-b2c1-4a9a-afa1-5bfd33999e99\") "
Oct 30 19:16:23.411492 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:23.411334    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f3c674d-b2c1-4a9a-afa1-5bfd33999e99-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f3c674d-b2c1-4a9a-afa1-5bfd33999e99-workload-certs\") pod \"4f3c674d-b2c1-4a9a-afa1-5bfd33999e99\" (UID: \"4f3c674d-b2c1-4a9a-afa1-5bfd33999e99\") "
Oct 30 19:16:23.411713 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:23.411664    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f3c674d-b2c1-4a9a-afa1-5bfd33999e99-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f3c674d-b2c1-4a9a-afa1-5bfd33999e99-istio-data\") pod \"4f3c674d-b2c1-4a9a-afa1-5bfd33999e99\" (UID: \"4f3c674d-b2c1-4a9a-afa1-5bfd33999e99\") "
Oct 30 19:16:23.411797 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:23.411746    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f3c674d-b2c1-4a9a-afa1-5bfd33999e99-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f3c674d-b2c1-4a9a-afa1-5bfd33999e99-workload-socket\") pod \"4f3c674d-b2c1-4a9a-afa1-5bfd33999e99\" (UID: \"4f3c674d-b2c1-4a9a-afa1-5bfd33999e99\") "
Oct 30 19:16:23.411797 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:23.411745    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f3c674d-b2c1-4a9a-afa1-5bfd33999e99-workload-certs" pod "4f3c674d-b2c1-4a9a-afa1-5bfd33999e99" (UID: "4f3c674d-b2c1-4a9a-afa1-5bfd33999e99"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:23.411893 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:23.411797    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f3c674d-b2c1-4a9a-afa1-5bfd33999e99-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f3c674d-b2c1-4a9a-afa1-5bfd33999e99-istiod-ca-cert\") pod \"4f3c674d-b2c1-4a9a-afa1-5bfd33999e99\" (UID: \"4f3c674d-b2c1-4a9a-afa1-5bfd33999e99\") "
Oct 30 19:16:23.411893 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:23.411874    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f3c674d-b2c1-4a9a-afa1-5bfd33999e99-istio-token\" (UniqueName: \"kubernetes.io/projected/4f3c674d-b2c1-4a9a-afa1-5bfd33999e99-istio-token\") pod \"4f3c674d-b2c1-4a9a-afa1-5bfd33999e99\" (UID: \"4f3c674d-b2c1-4a9a-afa1-5bfd33999e99\") "
Oct 30 19:16:23.411971 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:23.411945    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f3c674d-b2c1-4a9a-afa1-5bfd33999e99-istio-data" pod "4f3c674d-b2c1-4a9a-afa1-5bfd33999e99" (UID: "4f3c674d-b2c1-4a9a-afa1-5bfd33999e99"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:23.411971 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:23.411962    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f3c674d-b2c1-4a9a-afa1-5bfd33999e99-kube-api-access-q28jp\" (UniqueName: \"kubernetes.io/projected/4f3c674d-b2c1-4a9a-afa1-5bfd33999e99-kube-api-access-q28jp\") pod \"4f3c674d-b2c1-4a9a-afa1-5bfd33999e99\" (UID: \"4f3c674d-b2c1-4a9a-afa1-5bfd33999e99\") "
Oct 30 19:16:23.412450 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:23.412029    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f3c674d-b2c1-4a9a-afa1-5bfd33999e99-workload-socket" pod "4f3c674d-b2c1-4a9a-afa1-5bfd33999e99" (UID: "4f3c674d-b2c1-4a9a-afa1-5bfd33999e99"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:23.412450 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:23.412095    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f3c674d-b2c1-4a9a-afa1-5bfd33999e99-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f3c674d-b2c1-4a9a-afa1-5bfd33999e99-credential-socket\") pod \"4f3c674d-b2c1-4a9a-afa1-5bfd33999e99\" (UID: \"4f3c674d-b2c1-4a9a-afa1-5bfd33999e99\") "
Oct 30 19:16:23.412450 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:23.412154    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4f3c674d-b2c1-4a9a-afa1-5bfd33999e99-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f3c674d-b2c1-4a9a-afa1-5bfd33999e99-istio-podinfo\") pod \"4f3c674d-b2c1-4a9a-afa1-5bfd33999e99\" (UID: \"4f3c674d-b2c1-4a9a-afa1-5bfd33999e99\") "
Oct 30 19:16:23.412450 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:23.412380    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f3c674d-b2c1-4a9a-afa1-5bfd33999e99-credential-socket" pod "4f3c674d-b2c1-4a9a-afa1-5bfd33999e99" (UID: "4f3c674d-b2c1-4a9a-afa1-5bfd33999e99"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:23.412450 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:23.412419    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f3c674d-b2c1-4a9a-afa1-5bfd33999e99-istiod-ca-cert" pod "4f3c674d-b2c1-4a9a-afa1-5bfd33999e99" (UID: "4f3c674d-b2c1-4a9a-afa1-5bfd33999e99"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:16:23.413017 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:23.412992    4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f94f5889-fce0-46bc-a055-1c5e5e51d6a6-istio-envoy\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:23.413091 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:23.413023    4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f3c674d-b2c1-4a9a-afa1-5bfd33999e99-credential-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:23.413091 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:23.413039    4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f94f5889-fce0-46bc-a055-1c5e5e51d6a6-istiod-ca-cert\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:23.413091 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:23.413055    4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f94f5889-fce0-46bc-a055-1c5e5e51d6a6-credential-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:23.413091 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:23.413069    4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f3c674d-b2c1-4a9a-afa1-5bfd33999e99-workload-certs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:23.413091 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:23.413084    4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f94f5889-fce0-46bc-a055-1c5e5e51d6a6-workload-certs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:23.413271 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:23.413098    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfptz\" (UniqueName: \"kubernetes.io/projected/f94f5889-fce0-46bc-a055-1c5e5e51d6a6-kube-api-access-jfptz\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:23.413271 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:23.413115    4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f3c674d-b2c1-4a9a-afa1-5bfd33999e99-istio-data\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:23.413271 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:23.413129    4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f3c674d-b2c1-4a9a-afa1-5bfd33999e99-workload-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:23.413271 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:23.413142    4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f94f5889-fce0-46bc-a055-1c5e5e51d6a6-istio-podinfo\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:23.413271 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:23.413156    4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f3c674d-b2c1-4a9a-afa1-5bfd33999e99-istiod-ca-cert\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:23.413271 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:23.413169    4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f94f5889-fce0-46bc-a055-1c5e5e51d6a6-workload-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:23.413271 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:23.413181    4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f94f5889-fce0-46bc-a055-1c5e5e51d6a6-istio-token\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:23.413271 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:23.413194    4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f94f5889-fce0-46bc-a055-1c5e5e51d6a6-istio-data\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:23.413938 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:23.413918    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f3c674d-b2c1-4a9a-afa1-5bfd33999e99-istio-envoy" pod "4f3c674d-b2c1-4a9a-afa1-5bfd33999e99" (UID: "4f3c674d-b2c1-4a9a-afa1-5bfd33999e99"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:23.414329 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:23.414311    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4f3c674d-b2c1-4a9a-afa1-5bfd33999e99-istio-podinfo" pod "4f3c674d-b2c1-4a9a-afa1-5bfd33999e99" (UID: "4f3c674d-b2c1-4a9a-afa1-5bfd33999e99"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:16:23.414716 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:23.414662    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f3c674d-b2c1-4a9a-afa1-5bfd33999e99-kube-api-access-q28jp" pod "4f3c674d-b2c1-4a9a-afa1-5bfd33999e99" (UID: "4f3c674d-b2c1-4a9a-afa1-5bfd33999e99"). InnerVolumeSpecName "kube-api-access-q28jp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:23.414716 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:23.414705    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f3c674d-b2c1-4a9a-afa1-5bfd33999e99-istio-token" pod "4f3c674d-b2c1-4a9a-afa1-5bfd33999e99" (UID: "4f3c674d-b2c1-4a9a-afa1-5bfd33999e99"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:23.514659 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:23.514574    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q28jp\" (UniqueName: \"kubernetes.io/projected/4f3c674d-b2c1-4a9a-afa1-5bfd33999e99-kube-api-access-q28jp\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:23.514659 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:23.514604    4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f3c674d-b2c1-4a9a-afa1-5bfd33999e99-istio-podinfo\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:23.514659 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:23.514614    4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f3c674d-b2c1-4a9a-afa1-5bfd33999e99-istio-envoy\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:23.514659 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:23.514623    4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f3c674d-b2c1-4a9a-afa1-5bfd33999e99-istio-token\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:23.941801 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:23.941765    4525 generic.go:356] "Generic (PLEG): container finished" podID="f94f5889-fce0-46bc-a055-1c5e5e51d6a6" containerID="bc2cbbfe6b46c07059660f1c6a406daa26774e90ab7a95fb4c51c6ab06012878" exitCode=0
Oct 30 19:16:23.942353 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:23.941853    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-txf2f"
Oct 30 19:16:23.942353 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:23.941868    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-txf2f" event={"ID":"f94f5889-fce0-46bc-a055-1c5e5e51d6a6","Type":"ContainerDied","Data":"bc2cbbfe6b46c07059660f1c6a406daa26774e90ab7a95fb4c51c6ab06012878"}
Oct 30 19:16:23.942353 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:23.941943    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-txf2f" event={"ID":"f94f5889-fce0-46bc-a055-1c5e5e51d6a6","Type":"ContainerDied","Data":"05db23b54e8cf57cda893d1ba6028c2273f24b744e3ea47f2e907e2553054563"}
Oct 30 19:16:23.942353 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:23.941978    4525 scope.go:122] "RemoveContainer" containerID="bc2cbbfe6b46c07059660f1c6a406daa26774e90ab7a95fb4c51c6ab06012878"
Oct 30 19:16:23.944088 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:23.944061    4525 generic.go:356] "Generic (PLEG): container finished" podID="4f3c674d-b2c1-4a9a-afa1-5bfd33999e99" containerID="e1d959ee8e2f4a6eba684605cf5f2200ae0605953dff477153b7b806f5df67ae" exitCode=0
Oct 30 19:16:23.944220 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:23.944134    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-zv786"
Oct 30 19:16:23.944335 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:23.944313    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-zv786" event={"ID":"4f3c674d-b2c1-4a9a-afa1-5bfd33999e99","Type":"ContainerDied","Data":"e1d959ee8e2f4a6eba684605cf5f2200ae0605953dff477153b7b806f5df67ae"}
Oct 30 19:16:23.944425 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:23.944402    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-zv786" event={"ID":"4f3c674d-b2c1-4a9a-afa1-5bfd33999e99","Type":"ContainerDied","Data":"0f100e3b2dbefb2628a82c66e0d9456ba0a6698df3c5be9abc9fd4f1013ecb3a"}
Oct 30 19:16:23.954568 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:23.954551    4525 scope.go:122] "RemoveContainer" containerID="bc2cbbfe6b46c07059660f1c6a406daa26774e90ab7a95fb4c51c6ab06012878"
Oct 30 19:16:23.954890 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:16:23.954867    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc2cbbfe6b46c07059660f1c6a406daa26774e90ab7a95fb4c51c6ab06012878\": container with ID starting with bc2cbbfe6b46c07059660f1c6a406daa26774e90ab7a95fb4c51c6ab06012878 not found: ID does not exist" containerID="bc2cbbfe6b46c07059660f1c6a406daa26774e90ab7a95fb4c51c6ab06012878"
Oct 30 19:16:23.954999 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:23.954895    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc2cbbfe6b46c07059660f1c6a406daa26774e90ab7a95fb4c51c6ab06012878"} err="failed to get container status \"bc2cbbfe6b46c07059660f1c6a406daa26774e90ab7a95fb4c51c6ab06012878\": rpc error: code = NotFound desc = could not find container \"bc2cbbfe6b46c07059660f1c6a406daa26774e90ab7a95fb4c51c6ab06012878\": container with ID starting with bc2cbbfe6b46c07059660f1c6a406daa26774e90ab7a95fb4c51c6ab06012878 not found: ID does not exist"
Oct 30 19:16:23.954999 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:23.954911    4525 scope.go:122] "RemoveContainer" containerID="e1d959ee8e2f4a6eba684605cf5f2200ae0605953dff477153b7b806f5df67ae"
Oct 30 19:16:23.971221 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:23.971193    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-zv786"]
Oct 30 19:16:23.972390 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:23.972373    4525 scope.go:122] "RemoveContainer" containerID="e1d959ee8e2f4a6eba684605cf5f2200ae0605953dff477153b7b806f5df67ae"
Oct 30 19:16:23.973075 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:16:23.972821    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1d959ee8e2f4a6eba684605cf5f2200ae0605953dff477153b7b806f5df67ae\": container with ID starting with e1d959ee8e2f4a6eba684605cf5f2200ae0605953dff477153b7b806f5df67ae not found: ID does not exist" containerID="e1d959ee8e2f4a6eba684605cf5f2200ae0605953dff477153b7b806f5df67ae"
Oct 30 19:16:23.973075 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:23.972849    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1d959ee8e2f4a6eba684605cf5f2200ae0605953dff477153b7b806f5df67ae"} err="failed to get container status \"e1d959ee8e2f4a6eba684605cf5f2200ae0605953dff477153b7b806f5df67ae\": rpc error: code = NotFound desc = could not find container \"e1d959ee8e2f4a6eba684605cf5f2200ae0605953dff477153b7b806f5df67ae\": container with ID starting with e1d959ee8e2f4a6eba684605cf5f2200ae0605953dff477153b7b806f5df67ae not found: ID does not exist"
Oct 30 19:16:23.976820 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:23.976796    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-zv786"]
Oct 30 19:16:23.982960 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:23.982937    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-txf2f"]
Oct 30 19:16:23.989555 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:23.989536    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-9de872-openshift-default-77cf8df555-txf2f"]
Oct 30 19:16:24.181456 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:24.181433    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ca2cd5-6nfd4"
Oct 30 19:16:24.223709 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:24.223602    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cdebbe2-6dd6-4516-9ada-71b5083bd266-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cdebbe2-6dd6-4516-9ada-71b5083bd266-maas-api-openshift-ca\") pod \"0cdebbe2-6dd6-4516-9ada-71b5083bd266\" (UID: \"0cdebbe2-6dd6-4516-9ada-71b5083bd266\") "
Oct 30 19:16:24.223709 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:24.223650    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0cdebbe2-6dd6-4516-9ada-71b5083bd266-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cdebbe2-6dd6-4516-9ada-71b5083bd266-maas-api-xks-ca\") pod \"0cdebbe2-6dd6-4516-9ada-71b5083bd266\" (UID: \"0cdebbe2-6dd6-4516-9ada-71b5083bd266\") "
Oct 30 19:16:24.223949 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:24.223764    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cdebbe2-6dd6-4516-9ada-71b5083bd266-kube-api-access-tnbgz\" (UniqueName: \"kubernetes.io/projected/0cdebbe2-6dd6-4516-9ada-71b5083bd266-kube-api-access-tnbgz\") pod \"0cdebbe2-6dd6-4516-9ada-71b5083bd266\" (UID: \"0cdebbe2-6dd6-4516-9ada-71b5083bd266\") "
Oct 30 19:16:24.224053 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:24.224024    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cdebbe2-6dd6-4516-9ada-71b5083bd266-maas-api-openshift-ca" pod "0cdebbe2-6dd6-4516-9ada-71b5083bd266" (UID: "0cdebbe2-6dd6-4516-9ada-71b5083bd266"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:16:24.224396 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:24.224359    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cdebbe2-6dd6-4516-9ada-71b5083bd266-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:24.226082 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:24.226055    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cdebbe2-6dd6-4516-9ada-71b5083bd266-kube-api-access-tnbgz" pod "0cdebbe2-6dd6-4516-9ada-71b5083bd266" (UID: "0cdebbe2-6dd6-4516-9ada-71b5083bd266"). InnerVolumeSpecName "kube-api-access-tnbgz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:24.226410 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:24.226379    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cdebbe2-6dd6-4516-9ada-71b5083bd266-maas-api-xks-ca" pod "0cdebbe2-6dd6-4516-9ada-71b5083bd266" (UID: "0cdebbe2-6dd6-4516-9ada-71b5083bd266"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:16:24.325271 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:24.325233    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cdebbe2-6dd6-4516-9ada-71b5083bd266-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:24.325271 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:24.325266    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnbgz\" (UniqueName: \"kubernetes.io/projected/0cdebbe2-6dd6-4516-9ada-71b5083bd266-kube-api-access-tnbgz\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:24.953904 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:24.953866    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ca2cd5-6nfd4"
Oct 30 19:16:24.953904 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:24.953897    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ca2cd5-6nfd4" event={"ID":"0cdebbe2-6dd6-4516-9ada-71b5083bd266","Type":"ContainerDied","Data":"fb5da2698540f9ee458a58eb6fb602c846b8d82bee331f7a99f7b58d9f917d9d"}
Oct 30 19:16:24.954428 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:24.953947    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb5da2698540f9ee458a58eb6fb602c846b8d82bee331f7a99f7b58d9f917d9d"
Oct 30 19:16:25.593657 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:25.593617    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f3c674d-b2c1-4a9a-afa1-5bfd33999e99" path="/var/lib/kubelet/pods/4f3c674d-b2c1-4a9a-afa1-5bfd33999e99/volumes"
Oct 30 19:16:25.597021 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:25.594481    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f94f5889-fce0-46bc-a055-1c5e5e51d6a6" path="/var/lib/kubelet/pods/f94f5889-fce0-46bc-a055-1c5e5e51d6a6/volumes"
Oct 30 19:16:27.572738 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:27.572703    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh"]
Oct 30 19:16:27.573239 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:27.572914    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh" podUID="8b37419e-12f0-4e1e-8b2a-48c39c4c9e99" containerName="istio-proxy" containerID="cri-o://1eac631cc83233ff18fd8b0a52d9b7d6d2f6c362b9577146208fd2b742196dac" gracePeriod=30
Oct 30 19:16:28.300185 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:28.300151    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2889dd-2fxrn"]
Oct 30 19:16:28.300735 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:28.300720    4525 cpu_manager.go:414] "containerMap: removing container" podUID="f94f5889-fce0-46bc-a055-1c5e5e51d6a6" containerName="istio-proxy"
Oct 30 19:16:28.300780 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:28.300737    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f94f5889-fce0-46bc-a055-1c5e5e51d6a6" containerName="istio-proxy"
Oct 30 19:16:28.300780 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:28.300751    4525 cpu_manager.go:414] "containerMap: removing container" podUID="0f675df4-123a-412a-8aa3-5a61497b06bd" containerName="revoke-keys"
Oct 30 19:16:28.300780 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:28.300756    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f675df4-123a-412a-8aa3-5a61497b06bd" containerName="revoke-keys"
Oct 30 19:16:28.300780 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:28.300767    4525 cpu_manager.go:414] "containerMap: removing container" podUID="4f3c674d-b2c1-4a9a-afa1-5bfd33999e99" containerName="istio-proxy"
Oct 30 19:16:28.300780 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:28.300771    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f3c674d-b2c1-4a9a-afa1-5bfd33999e99" containerName="istio-proxy"
Oct 30 19:16:28.300939 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:28.300785    4525 cpu_manager.go:414] "containerMap: removing container" podUID="0cdebbe2-6dd6-4516-9ada-71b5083bd266" containerName="revoke-keys"
Oct 30 19:16:28.300939 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:28.300790    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cdebbe2-6dd6-4516-9ada-71b5083bd266" containerName="revoke-keys"
Oct 30 19:16:28.300939 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:28.300805    4525 cpu_manager.go:414] "containerMap: removing container" podUID="e25fa0f4-be7f-4c10-83f8-ce8b479668cb" containerName="revoke-keys"
Oct 30 19:16:28.300939 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:28.300810    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e25fa0f4-be7f-4c10-83f8-ce8b479668cb" containerName="revoke-keys"
Oct 30 19:16:28.300939 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:28.300895    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cdebbe2-6dd6-4516-9ada-71b5083bd266" containerName="revoke-keys"
Oct 30 19:16:28.300939 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:28.300907    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="e25fa0f4-be7f-4c10-83f8-ce8b479668cb" containerName="revoke-keys"
Oct 30 19:16:28.300939 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:28.300916    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="f94f5889-fce0-46bc-a055-1c5e5e51d6a6" containerName="istio-proxy"
Oct 30 19:16:28.300939 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:28.300922    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f675df4-123a-412a-8aa3-5a61497b06bd" containerName="revoke-keys"
Oct 30 19:16:28.300939 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:28.300929    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f3c674d-b2c1-4a9a-afa1-5bfd33999e99" containerName="istio-proxy"
Oct 30 19:16:28.306293 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:28.306266    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2889dd-2fxrn"
Oct 30 19:16:28.314529 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:28.314497    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2889dd-2fxrn"]
Oct 30 19:16:28.337172 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:28.337139    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:16:28.337693 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:28.337654    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94hcn\""
Oct 30 19:16:28.416178 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:28.416142    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0128f9e4-fbac-46b9-a7ec-2c19cac0d30a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-2889dd-2fxrn\" (UID: \"0128f9e4-fbac-46b9-a7ec-2c19cac0d30a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2889dd-2fxrn"
Oct 30 19:16:28.416355 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:28.416335    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lq69c\" (UniqueName: \"kubernetes.io/projected/0128f9e4-fbac-46b9-a7ec-2c19cac0d30a-kube-api-access-lq69c\") pod \"maas-api-revoke-keys-e2e-worker-w3-2889dd-2fxrn\" (UID: \"0128f9e4-fbac-46b9-a7ec-2c19cac0d30a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2889dd-2fxrn"
Oct 30 19:16:28.416406 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:28.416393    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0128f9e4-fbac-46b9-a7ec-2c19cac0d30a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-2889dd-2fxrn\" (UID: \"0128f9e4-fbac-46b9-a7ec-2c19cac0d30a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2889dd-2fxrn"
Oct 30 19:16:28.517955 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:28.517921    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lq69c\" (UniqueName: \"kubernetes.io/projected/0128f9e4-fbac-46b9-a7ec-2c19cac0d30a-kube-api-access-lq69c\") pod \"maas-api-revoke-keys-e2e-worker-w3-2889dd-2fxrn\" (UID: \"0128f9e4-fbac-46b9-a7ec-2c19cac0d30a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2889dd-2fxrn"
Oct 30 19:16:28.518135 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:28.517990    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0128f9e4-fbac-46b9-a7ec-2c19cac0d30a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-2889dd-2fxrn\" (UID: \"0128f9e4-fbac-46b9-a7ec-2c19cac0d30a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2889dd-2fxrn"
Oct 30 19:16:28.518135 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:28.518073    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0128f9e4-fbac-46b9-a7ec-2c19cac0d30a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-2889dd-2fxrn\" (UID: \"0128f9e4-fbac-46b9-a7ec-2c19cac0d30a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2889dd-2fxrn"
Oct 30 19:16:28.518710 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:28.518643    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0128f9e4-fbac-46b9-a7ec-2c19cac0d30a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-2889dd-2fxrn\" (UID: \"0128f9e4-fbac-46b9-a7ec-2c19cac0d30a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2889dd-2fxrn"
Oct 30 19:16:28.520554 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:28.520528    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0128f9e4-fbac-46b9-a7ec-2c19cac0d30a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-2889dd-2fxrn\" (UID: \"0128f9e4-fbac-46b9-a7ec-2c19cac0d30a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2889dd-2fxrn"
Oct 30 19:16:28.527148 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:28.527111    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lq69c\" (UniqueName: \"kubernetes.io/projected/0128f9e4-fbac-46b9-a7ec-2c19cac0d30a-kube-api-access-lq69c\") pod \"maas-api-revoke-keys-e2e-worker-w3-2889dd-2fxrn\" (UID: \"0128f9e4-fbac-46b9-a7ec-2c19cac0d30a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2889dd-2fxrn"
Oct 30 19:16:28.625505 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:28.625467    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2889dd-2fxrn"
Oct 30 19:16:28.913519 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:28.913488    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2889dd-2fxrn"]
Oct 30 19:16:28.913792 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:16:28.913762    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0128f9e4_fbac_46b9_a7ec_2c19cac0d30a.slice/crio-e3008e28b79658d284d2d09498b6b6e101738f1ccf99f531b95ec9803dea4ce7 WatchSource:0}: Error finding container e3008e28b79658d284d2d09498b6b6e101738f1ccf99f531b95ec9803dea4ce7: Status 404 returned error can't find the container with id e3008e28b79658d284d2d09498b6b6e101738f1ccf99f531b95ec9803dea4ce7
Oct 30 19:16:28.990615 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:28.990581    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2889dd-2fxrn" event={"ID":"0128f9e4-fbac-46b9-a7ec-2c19cac0d30a","Type":"ContainerStarted","Data":"e3008e28b79658d284d2d09498b6b6e101738f1ccf99f531b95ec9803dea4ce7"}
Oct 30 19:16:29.519866 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:29.519787    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a1d55d-openshift-default-6646f694ff-c6hjq"]
Oct 30 19:16:29.520018 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:29.519995    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-a1d55d-openshift-default-6646f694ff-c6hjq" podUID="d05765ac-da9e-4fad-b2e8-e26b8630849e" containerName="istio-proxy" containerID="cri-o://d6768cbe6b7d423925def8357d00c26ab069b4c0fc1c95b03abdd16f7875e590" gracePeriod=30
Oct 30 19:16:29.999757 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:29.999722    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ef6be7-46xv8"]
Oct 30 19:16:30.001578 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:30.001551    4525 generic.go:356] "Generic (PLEG): container finished" podID="0128f9e4-fbac-46b9-a7ec-2c19cac0d30a" containerID="3966ecbf6d02fb6803c359865c0a9c427ed12e09d8aa833466d72b17f1ec05e5" exitCode=0
Oct 30 19:16:30.004210 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:30.004190    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2889dd-2fxrn" event={"ID":"0128f9e4-fbac-46b9-a7ec-2c19cac0d30a","Type":"ContainerDied","Data":"3966ecbf6d02fb6803c359865c0a9c427ed12e09d8aa833466d72b17f1ec05e5"}
Oct 30 19:16:30.004279 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:30.004269    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ef6be7-46xv8"
Oct 30 19:16:30.028269 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:30.028240    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ef6be7-46xv8"]
Oct 30 19:16:30.158826 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:30.158789    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25e308fe-6246-452a-9c60-f1304509a027-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-ef6be7-46xv8\" (UID: \"25e308fe-6246-452a-9c60-f1304509a027\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ef6be7-46xv8"
Oct 30 19:16:30.159001 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:30.158859    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nplr8\" (UniqueName: \"kubernetes.io/projected/25e308fe-6246-452a-9c60-f1304509a027-kube-api-access-nplr8\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-ef6be7-46xv8\" (UID: \"25e308fe-6246-452a-9c60-f1304509a027\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ef6be7-46xv8"
Oct 30 19:16:30.159094 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:30.158999    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25e308fe-6246-452a-9c60-f1304509a027-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-ef6be7-46xv8\" (UID: \"25e308fe-6246-452a-9c60-f1304509a027\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ef6be7-46xv8"
Oct 30 19:16:30.260849 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:30.260809    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25e308fe-6246-452a-9c60-f1304509a027-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-ef6be7-46xv8\" (UID: \"25e308fe-6246-452a-9c60-f1304509a027\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ef6be7-46xv8"
Oct 30 19:16:30.260995 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:30.260906    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nplr8\" (UniqueName: \"kubernetes.io/projected/25e308fe-6246-452a-9c60-f1304509a027-kube-api-access-nplr8\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-ef6be7-46xv8\" (UID: \"25e308fe-6246-452a-9c60-f1304509a027\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ef6be7-46xv8"
Oct 30 19:16:30.260995 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:30.260958    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25e308fe-6246-452a-9c60-f1304509a027-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-ef6be7-46xv8\" (UID: \"25e308fe-6246-452a-9c60-f1304509a027\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ef6be7-46xv8"
Oct 30 19:16:30.261529 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:30.261495    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25e308fe-6246-452a-9c60-f1304509a027-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-ef6be7-46xv8\" (UID: \"25e308fe-6246-452a-9c60-f1304509a027\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ef6be7-46xv8"
Oct 30 19:16:30.263620 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:30.263601    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25e308fe-6246-452a-9c60-f1304509a027-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-ef6be7-46xv8\" (UID: \"25e308fe-6246-452a-9c60-f1304509a027\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ef6be7-46xv8"
Oct 30 19:16:30.269609 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:30.269578    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nplr8\" (UniqueName: \"kubernetes.io/projected/25e308fe-6246-452a-9c60-f1304509a027-kube-api-access-nplr8\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-ef6be7-46xv8\" (UID: \"25e308fe-6246-452a-9c60-f1304509a027\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ef6be7-46xv8"
Oct 30 19:16:30.335741 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:30.335709    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ef6be7-46xv8"
Oct 30 19:16:30.644013 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:30.643976    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ef6be7-46xv8"]
Oct 30 19:16:30.652540 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:16:30.652506    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25e308fe_6246_452a_9c60_f1304509a027.slice/crio-ab478e10d68e6abd535e4b4c6a5d3753c0d947824c628d1e4b9b28815dc81766 WatchSource:0}: Error finding container ab478e10d68e6abd535e4b4c6a5d3753c0d947824c628d1e4b9b28815dc81766: Status 404 returned error can't find the container with id ab478e10d68e6abd535e4b4c6a5d3753c0d947824c628d1e4b9b28815dc81766
Oct 30 19:16:30.848372 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:30.848334    4525 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a1d55d-openshift-default-6646f694ff-c6hjq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:16:30.848528 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:30.848403    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-a1d55d-openshift-default-6646f694ff-c6hjq" podUID="d05765ac-da9e-4fad-b2e8-e26b8630849e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:16:31.011044 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:31.011010    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ef6be7-46xv8" event={"ID":"25e308fe-6246-452a-9c60-f1304509a027","Type":"ContainerStarted","Data":"cd102df63b4d5d3b390045193e00982ad585fb9b9ecd807eeca67b6f0e2235bc"}
Oct 30 19:16:31.011401 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:31.011065    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ef6be7-46xv8" event={"ID":"25e308fe-6246-452a-9c60-f1304509a027","Type":"ContainerStarted","Data":"ab478e10d68e6abd535e4b4c6a5d3753c0d947824c628d1e4b9b28815dc81766"}
Oct 30 19:16:31.037919 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:31.037862    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ef6be7-46xv8" podStartSLOduration=2.03784608 podStartE2EDuration="2.03784608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:16:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:31.035605813 +0000 UTC m=+2413.661275776" watchObservedRunningTime="2026-10-30 19:16:31.03784608 +0000 UTC m=+2413.663516044"
Oct 30 19:16:31.289758 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:31.289729    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2889dd-2fxrn"
Oct 30 19:16:31.377655 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:31.377620    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0128f9e4-fbac-46b9-a7ec-2c19cac0d30a-kube-api-access-lq69c\" (UniqueName: \"kubernetes.io/projected/0128f9e4-fbac-46b9-a7ec-2c19cac0d30a-kube-api-access-lq69c\") pod \"0128f9e4-fbac-46b9-a7ec-2c19cac0d30a\" (UID: \"0128f9e4-fbac-46b9-a7ec-2c19cac0d30a\") "
Oct 30 19:16:31.377881 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:31.377668    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0128f9e4-fbac-46b9-a7ec-2c19cac0d30a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0128f9e4-fbac-46b9-a7ec-2c19cac0d30a-maas-api-openshift-ca\") pod \"0128f9e4-fbac-46b9-a7ec-2c19cac0d30a\" (UID: \"0128f9e4-fbac-46b9-a7ec-2c19cac0d30a\") "
Oct 30 19:16:31.377881 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:31.377746    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0128f9e4-fbac-46b9-a7ec-2c19cac0d30a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0128f9e4-fbac-46b9-a7ec-2c19cac0d30a-maas-api-xks-ca\") pod \"0128f9e4-fbac-46b9-a7ec-2c19cac0d30a\" (UID: \"0128f9e4-fbac-46b9-a7ec-2c19cac0d30a\") "
Oct 30 19:16:31.378115 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:31.378087    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0128f9e4-fbac-46b9-a7ec-2c19cac0d30a-maas-api-openshift-ca" pod "0128f9e4-fbac-46b9-a7ec-2c19cac0d30a" (UID: "0128f9e4-fbac-46b9-a7ec-2c19cac0d30a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:16:31.378406 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:31.378377    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0128f9e4-fbac-46b9-a7ec-2c19cac0d30a-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:31.380122 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:31.380098    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0128f9e4-fbac-46b9-a7ec-2c19cac0d30a-maas-api-xks-ca" pod "0128f9e4-fbac-46b9-a7ec-2c19cac0d30a" (UID: "0128f9e4-fbac-46b9-a7ec-2c19cac0d30a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:16:31.380122 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:31.380118    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0128f9e4-fbac-46b9-a7ec-2c19cac0d30a-kube-api-access-lq69c" pod "0128f9e4-fbac-46b9-a7ec-2c19cac0d30a" (UID: "0128f9e4-fbac-46b9-a7ec-2c19cac0d30a"). InnerVolumeSpecName "kube-api-access-lq69c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:31.479300 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:31.479274    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lq69c\" (UniqueName: \"kubernetes.io/projected/0128f9e4-fbac-46b9-a7ec-2c19cac0d30a-kube-api-access-lq69c\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:31.479300 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:31.479300    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0128f9e4-fbac-46b9-a7ec-2c19cac0d30a-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:31.941004 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:31.940010    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ca2cd5-6nfd4"]
Oct 30 19:16:31.955741 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:31.954422    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ca2cd5-6nfd4"]
Oct 30 19:16:32.022576 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:32.022535    4525 generic.go:356] "Generic (PLEG): container finished" podID="25e308fe-6246-452a-9c60-f1304509a027" containerID="cd102df63b4d5d3b390045193e00982ad585fb9b9ecd807eeca67b6f0e2235bc" exitCode=0
Oct 30 19:16:32.023128 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:32.022638    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ef6be7-46xv8" event={"ID":"25e308fe-6246-452a-9c60-f1304509a027","Type":"ContainerDied","Data":"cd102df63b4d5d3b390045193e00982ad585fb9b9ecd807eeca67b6f0e2235bc"}
Oct 30 19:16:32.024449 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:32.024424    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2889dd-2fxrn"
Oct 30 19:16:32.024560 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:32.024447    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2889dd-2fxrn" event={"ID":"0128f9e4-fbac-46b9-a7ec-2c19cac0d30a","Type":"ContainerDied","Data":"e3008e28b79658d284d2d09498b6b6e101738f1ccf99f531b95ec9803dea4ce7"}
Oct 30 19:16:32.024560 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:32.024490    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e3008e28b79658d284d2d09498b6b6e101738f1ccf99f531b95ec9803dea4ce7"
Oct 30 19:16:32.906447 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:32.906413    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh"
Oct 30 19:16:32.952569 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:32.952536    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b37419e-12f0-4e1e-8b2a-48c39c4c9e99-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b37419e-12f0-4e1e-8b2a-48c39c4c9e99-credential-socket\") pod \"8b37419e-12f0-4e1e-8b2a-48c39c4c9e99\" (UID: \"8b37419e-12f0-4e1e-8b2a-48c39c4c9e99\") "
Oct 30 19:16:32.952800 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:32.952582    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b37419e-12f0-4e1e-8b2a-48c39c4c9e99-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b37419e-12f0-4e1e-8b2a-48c39c4c9e99-istiod-ca-cert\") pod \"8b37419e-12f0-4e1e-8b2a-48c39c4c9e99\" (UID: \"8b37419e-12f0-4e1e-8b2a-48c39c4c9e99\") "
Oct 30 19:16:32.952800 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:32.952644    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b37419e-12f0-4e1e-8b2a-48c39c4c9e99-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b37419e-12f0-4e1e-8b2a-48c39c4c9e99-istio-envoy\") pod \"8b37419e-12f0-4e1e-8b2a-48c39c4c9e99\" (UID: \"8b37419e-12f0-4e1e-8b2a-48c39c4c9e99\") "
Oct 30 19:16:32.952800 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:32.952745    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b37419e-12f0-4e1e-8b2a-48c39c4c9e99-istio-token\" (UniqueName: \"kubernetes.io/projected/8b37419e-12f0-4e1e-8b2a-48c39c4c9e99-istio-token\") pod \"8b37419e-12f0-4e1e-8b2a-48c39c4c9e99\" (UID: \"8b37419e-12f0-4e1e-8b2a-48c39c4c9e99\") "
Oct 30 19:16:32.952800 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:32.952774    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b37419e-12f0-4e1e-8b2a-48c39c4c9e99-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b37419e-12f0-4e1e-8b2a-48c39c4c9e99-istio-data\") pod \"8b37419e-12f0-4e1e-8b2a-48c39c4c9e99\" (UID: \"8b37419e-12f0-4e1e-8b2a-48c39c4c9e99\") "
Oct 30 19:16:32.953054 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:32.952835    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b37419e-12f0-4e1e-8b2a-48c39c4c9e99-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b37419e-12f0-4e1e-8b2a-48c39c4c9e99-workload-socket\") pod \"8b37419e-12f0-4e1e-8b2a-48c39c4c9e99\" (UID: \"8b37419e-12f0-4e1e-8b2a-48c39c4c9e99\") "
Oct 30 19:16:32.953054 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:32.952885    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b37419e-12f0-4e1e-8b2a-48c39c4c9e99-credential-socket" pod "8b37419e-12f0-4e1e-8b2a-48c39c4c9e99" (UID: "8b37419e-12f0-4e1e-8b2a-48c39c4c9e99"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:32.953054 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:32.952989    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8b37419e-12f0-4e1e-8b2a-48c39c4c9e99-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b37419e-12f0-4e1e-8b2a-48c39c4c9e99-istio-podinfo\") pod \"8b37419e-12f0-4e1e-8b2a-48c39c4c9e99\" (UID: \"8b37419e-12f0-4e1e-8b2a-48c39c4c9e99\") "
Oct 30 19:16:32.953054 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:32.953019    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b37419e-12f0-4e1e-8b2a-48c39c4c9e99-kube-api-access-zjxbz\" (UniqueName: \"kubernetes.io/projected/8b37419e-12f0-4e1e-8b2a-48c39c4c9e99-kube-api-access-zjxbz\") pod \"8b37419e-12f0-4e1e-8b2a-48c39c4c9e99\" (UID: \"8b37419e-12f0-4e1e-8b2a-48c39c4c9e99\") "
Oct 30 19:16:32.953281 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:32.953090    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b37419e-12f0-4e1e-8b2a-48c39c4c9e99-istiod-ca-cert" pod "8b37419e-12f0-4e1e-8b2a-48c39c4c9e99" (UID: "8b37419e-12f0-4e1e-8b2a-48c39c4c9e99"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:16:32.953281 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:32.953150    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b37419e-12f0-4e1e-8b2a-48c39c4c9e99-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b37419e-12f0-4e1e-8b2a-48c39c4c9e99-workload-certs\") pod \"8b37419e-12f0-4e1e-8b2a-48c39c4c9e99\" (UID: \"8b37419e-12f0-4e1e-8b2a-48c39c4c9e99\") "
Oct 30 19:16:32.953402 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:32.953356    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b37419e-12f0-4e1e-8b2a-48c39c4c9e99-istio-data" pod "8b37419e-12f0-4e1e-8b2a-48c39c4c9e99" (UID: "8b37419e-12f0-4e1e-8b2a-48c39c4c9e99"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:32.953580 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:32.953554    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b37419e-12f0-4e1e-8b2a-48c39c4c9e99-workload-certs" pod "8b37419e-12f0-4e1e-8b2a-48c39c4c9e99" (UID: "8b37419e-12f0-4e1e-8b2a-48c39c4c9e99"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:32.953824 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:32.953790    4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b37419e-12f0-4e1e-8b2a-48c39c4c9e99-workload-certs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:32.953979 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:32.953898    4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b37419e-12f0-4e1e-8b2a-48c39c4c9e99-credential-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:32.954089 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:32.954004    4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b37419e-12f0-4e1e-8b2a-48c39c4c9e99-istiod-ca-cert\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:32.954089 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:32.954019    4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b37419e-12f0-4e1e-8b2a-48c39c4c9e99-istio-data\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:32.954089 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:32.953941    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b37419e-12f0-4e1e-8b2a-48c39c4c9e99-workload-socket" pod "8b37419e-12f0-4e1e-8b2a-48c39c4c9e99" (UID: "8b37419e-12f0-4e1e-8b2a-48c39c4c9e99"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:32.956422 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:32.955999    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b37419e-12f0-4e1e-8b2a-48c39c4c9e99-istio-token" pod "8b37419e-12f0-4e1e-8b2a-48c39c4c9e99" (UID: "8b37419e-12f0-4e1e-8b2a-48c39c4c9e99"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:32.956422 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:32.956089    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b37419e-12f0-4e1e-8b2a-48c39c4c9e99-kube-api-access-zjxbz" pod "8b37419e-12f0-4e1e-8b2a-48c39c4c9e99" (UID: "8b37419e-12f0-4e1e-8b2a-48c39c4c9e99"). InnerVolumeSpecName "kube-api-access-zjxbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:32.956422 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:32.956217    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b37419e-12f0-4e1e-8b2a-48c39c4c9e99-istio-podinfo" pod "8b37419e-12f0-4e1e-8b2a-48c39c4c9e99" (UID: "8b37419e-12f0-4e1e-8b2a-48c39c4c9e99"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:16:32.956836 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:32.956808    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b37419e-12f0-4e1e-8b2a-48c39c4c9e99-istio-envoy" pod "8b37419e-12f0-4e1e-8b2a-48c39c4c9e99" (UID: "8b37419e-12f0-4e1e-8b2a-48c39c4c9e99"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:33.031845 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:33.031811    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj"]
Oct 30 19:16:33.036814 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:33.036786    4525 generic.go:356] "Generic (PLEG): container finished" podID="8b37419e-12f0-4e1e-8b2a-48c39c4c9e99" containerID="1eac631cc83233ff18fd8b0a52d9b7d6d2f6c362b9577146208fd2b742196dac" exitCode=0
Oct 30 19:16:33.036962 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:33.036840    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh"
Oct 30 19:16:33.036962 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:33.036867    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh" event={"ID":"8b37419e-12f0-4e1e-8b2a-48c39c4c9e99","Type":"ContainerDied","Data":"1eac631cc83233ff18fd8b0a52d9b7d6d2f6c362b9577146208fd2b742196dac"}
Oct 30 19:16:33.036962 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:33.036919    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh" event={"ID":"8b37419e-12f0-4e1e-8b2a-48c39c4c9e99","Type":"ContainerDied","Data":"d593c3d545e83f9a933c45411de311309704637838fd777abbae289788524f4a"}
Oct 30 19:16:33.036962 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:33.036941    4525 scope.go:122] "RemoveContainer" containerID="1eac631cc83233ff18fd8b0a52d9b7d6d2f6c362b9577146208fd2b742196dac"
Oct 30 19:16:33.050941 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:33.050912    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv"]
Oct 30 19:16:33.051887 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:33.051787    4525 scope.go:122] "RemoveContainer" containerID="61529cb0c57fb81b6f1ff254514dbb0a5ae3aa1ce6910eacfc8b06288ce15574"
Oct 30 19:16:33.055955 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:33.055930    4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b37419e-12f0-4e1e-8b2a-48c39c4c9e99-istio-envoy\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:33.056085 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:33.055956    4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b37419e-12f0-4e1e-8b2a-48c39c4c9e99-istio-token\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:33.056085 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:33.055971    4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b37419e-12f0-4e1e-8b2a-48c39c4c9e99-workload-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:33.056085 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:33.055982    4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b37419e-12f0-4e1e-8b2a-48c39c4c9e99-istio-podinfo\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:33.056085 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:33.055991    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjxbz\" (UniqueName: \"kubernetes.io/projected/8b37419e-12f0-4e1e-8b2a-48c39c4c9e99-kube-api-access-zjxbz\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:33.065999 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:33.065968    4525 scope.go:122] "RemoveContainer" containerID="1eac631cc83233ff18fd8b0a52d9b7d6d2f6c362b9577146208fd2b742196dac"
Oct 30 19:16:33.066415 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:16:33.066385    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1eac631cc83233ff18fd8b0a52d9b7d6d2f6c362b9577146208fd2b742196dac\": container with ID starting with 1eac631cc83233ff18fd8b0a52d9b7d6d2f6c362b9577146208fd2b742196dac not found: ID does not exist" containerID="1eac631cc83233ff18fd8b0a52d9b7d6d2f6c362b9577146208fd2b742196dac"
Oct 30 19:16:33.066506 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:33.066427    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1eac631cc83233ff18fd8b0a52d9b7d6d2f6c362b9577146208fd2b742196dac"} err="failed to get container status \"1eac631cc83233ff18fd8b0a52d9b7d6d2f6c362b9577146208fd2b742196dac\": rpc error: code = NotFound desc = could not find container \"1eac631cc83233ff18fd8b0a52d9b7d6d2f6c362b9577146208fd2b742196dac\": container with ID starting with 1eac631cc83233ff18fd8b0a52d9b7d6d2f6c362b9577146208fd2b742196dac not found: ID does not exist"
Oct 30 19:16:33.066506 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:33.066450    4525 scope.go:122] "RemoveContainer" containerID="61529cb0c57fb81b6f1ff254514dbb0a5ae3aa1ce6910eacfc8b06288ce15574"
Oct 30 19:16:33.066846 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:16:33.066777    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61529cb0c57fb81b6f1ff254514dbb0a5ae3aa1ce6910eacfc8b06288ce15574\": container with ID starting with 61529cb0c57fb81b6f1ff254514dbb0a5ae3aa1ce6910eacfc8b06288ce15574 not found: ID does not exist" containerID="61529cb0c57fb81b6f1ff254514dbb0a5ae3aa1ce6910eacfc8b06288ce15574"
Oct 30 19:16:33.066846 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:33.066810    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61529cb0c57fb81b6f1ff254514dbb0a5ae3aa1ce6910eacfc8b06288ce15574"} err="failed to get container status \"61529cb0c57fb81b6f1ff254514dbb0a5ae3aa1ce6910eacfc8b06288ce15574\": rpc error: code = NotFound desc = could not find container \"61529cb0c57fb81b6f1ff254514dbb0a5ae3aa1ce6910eacfc8b06288ce15574\": container with ID starting with 61529cb0c57fb81b6f1ff254514dbb0a5ae3aa1ce6910eacfc8b06288ce15574 not found: ID does not exist"
Oct 30 19:16:33.077413 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:33.077375    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t"]
Oct 30 19:16:33.128524 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:33.128435    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh"]
Oct 30 19:16:33.135786 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:33.135754    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-2d1c71-openshift-default-5dc796b975-l5bkh"]
Oct 30 19:16:33.207250 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:33.207219    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ef6be7-46xv8"
Oct 30 19:16:33.258542 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:33.258507    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25e308fe-6246-452a-9c60-f1304509a027-kube-api-access-nplr8\" (UniqueName: \"kubernetes.io/projected/25e308fe-6246-452a-9c60-f1304509a027-kube-api-access-nplr8\") pod \"25e308fe-6246-452a-9c60-f1304509a027\" (UID: \"25e308fe-6246-452a-9c60-f1304509a027\") "
Oct 30 19:16:33.258748 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:33.258610    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25e308fe-6246-452a-9c60-f1304509a027-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25e308fe-6246-452a-9c60-f1304509a027-maas-api-xks-ca\") pod \"25e308fe-6246-452a-9c60-f1304509a027\" (UID: \"25e308fe-6246-452a-9c60-f1304509a027\") "
Oct 30 19:16:33.258863 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:33.258843    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25e308fe-6246-452a-9c60-f1304509a027-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25e308fe-6246-452a-9c60-f1304509a027-maas-api-openshift-ca\") pod \"25e308fe-6246-452a-9c60-f1304509a027\" (UID: \"25e308fe-6246-452a-9c60-f1304509a027\") "
Oct 30 19:16:33.259457 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:33.259410    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25e308fe-6246-452a-9c60-f1304509a027-maas-api-openshift-ca" pod "25e308fe-6246-452a-9c60-f1304509a027" (UID: "25e308fe-6246-452a-9c60-f1304509a027"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:16:33.261405 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:33.261376    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25e308fe-6246-452a-9c60-f1304509a027-kube-api-access-nplr8" pod "25e308fe-6246-452a-9c60-f1304509a027" (UID: "25e308fe-6246-452a-9c60-f1304509a027"). InnerVolumeSpecName "kube-api-access-nplr8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:33.261850 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:33.261806    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25e308fe-6246-452a-9c60-f1304509a027-maas-api-xks-ca" pod "25e308fe-6246-452a-9c60-f1304509a027" (UID: "25e308fe-6246-452a-9c60-f1304509a027"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:16:33.361580 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:33.361540    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nplr8\" (UniqueName: \"kubernetes.io/projected/25e308fe-6246-452a-9c60-f1304509a027-kube-api-access-nplr8\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:33.361580 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:33.361584    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25e308fe-6246-452a-9c60-f1304509a027-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:33.361827 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:33.361600    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25e308fe-6246-452a-9c60-f1304509a027-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:33.621713 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:33.619640    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cdebbe2-6dd6-4516-9ada-71b5083bd266" path="/var/lib/kubelet/pods/0cdebbe2-6dd6-4516-9ada-71b5083bd266/volumes"
Oct 30 19:16:33.621713 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:33.620472    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b37419e-12f0-4e1e-8b2a-48c39c4c9e99" path="/var/lib/kubelet/pods/8b37419e-12f0-4e1e-8b2a-48c39c4c9e99/volumes"
Oct 30 19:16:33.734200 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:16:33.734171    4525 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25e308fe_6246_452a_9c60_f1304509a027.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 19:16:33.753241 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:33.753211    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0458df-qqlcw"]
Oct 30 19:16:33.760056 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:33.760032    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-0458df-qqlcw"]
Oct 30 19:16:34.047453 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:34.047372    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ef6be7-46xv8"
Oct 30 19:16:34.047453 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:34.047401    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ef6be7-46xv8" event={"ID":"25e308fe-6246-452a-9c60-f1304509a027","Type":"ContainerDied","Data":"ab478e10d68e6abd535e4b4c6a5d3753c0d947824c628d1e4b9b28815dc81766"}
Oct 30 19:16:34.047453 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:34.047451    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab478e10d68e6abd535e4b4c6a5d3753c0d947824c628d1e4b9b28815dc81766"
Oct 30 19:16:34.820540 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:34.820426    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a1d55d-openshift-default-6646f694ff-c6hjq"
Oct 30 19:16:34.898617 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:34.898588    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d05765ac-da9e-4fad-b2e8-e26b8630849e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d05765ac-da9e-4fad-b2e8-e26b8630849e-workload-certs\") pod \"d05765ac-da9e-4fad-b2e8-e26b8630849e\" (UID: \"d05765ac-da9e-4fad-b2e8-e26b8630849e\") "
Oct 30 19:16:34.898995 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:34.898711    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d05765ac-da9e-4fad-b2e8-e26b8630849e-kube-api-access-4bhgr\" (UniqueName: \"kubernetes.io/projected/d05765ac-da9e-4fad-b2e8-e26b8630849e-kube-api-access-4bhgr\") pod \"d05765ac-da9e-4fad-b2e8-e26b8630849e\" (UID: \"d05765ac-da9e-4fad-b2e8-e26b8630849e\") "
Oct 30 19:16:34.898995 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:34.898746    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d05765ac-da9e-4fad-b2e8-e26b8630849e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d05765ac-da9e-4fad-b2e8-e26b8630849e-istio-envoy\") pod \"d05765ac-da9e-4fad-b2e8-e26b8630849e\" (UID: \"d05765ac-da9e-4fad-b2e8-e26b8630849e\") "
Oct 30 19:16:34.898995 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:34.898767    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d05765ac-da9e-4fad-b2e8-e26b8630849e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d05765ac-da9e-4fad-b2e8-e26b8630849e-istiod-ca-cert\") pod \"d05765ac-da9e-4fad-b2e8-e26b8630849e\" (UID: \"d05765ac-da9e-4fad-b2e8-e26b8630849e\") "
Oct 30 19:16:34.898995 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:34.898868    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d05765ac-da9e-4fad-b2e8-e26b8630849e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d05765ac-da9e-4fad-b2e8-e26b8630849e-istio-podinfo\") pod \"d05765ac-da9e-4fad-b2e8-e26b8630849e\" (UID: \"d05765ac-da9e-4fad-b2e8-e26b8630849e\") "
Oct 30 19:16:34.898995 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:34.898908    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d05765ac-da9e-4fad-b2e8-e26b8630849e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d05765ac-da9e-4fad-b2e8-e26b8630849e-istio-data\") pod \"d05765ac-da9e-4fad-b2e8-e26b8630849e\" (UID: \"d05765ac-da9e-4fad-b2e8-e26b8630849e\") "
Oct 30 19:16:34.898995 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:34.898918    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d05765ac-da9e-4fad-b2e8-e26b8630849e-workload-certs" pod "d05765ac-da9e-4fad-b2e8-e26b8630849e" (UID: "d05765ac-da9e-4fad-b2e8-e26b8630849e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:34.898995 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:34.898936    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d05765ac-da9e-4fad-b2e8-e26b8630849e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d05765ac-da9e-4fad-b2e8-e26b8630849e-credential-socket\") pod \"d05765ac-da9e-4fad-b2e8-e26b8630849e\" (UID: \"d05765ac-da9e-4fad-b2e8-e26b8630849e\") "
Oct 30 19:16:34.899413 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:34.899251    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d05765ac-da9e-4fad-b2e8-e26b8630849e-credential-socket" pod "d05765ac-da9e-4fad-b2e8-e26b8630849e" (UID: "d05765ac-da9e-4fad-b2e8-e26b8630849e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:34.899413 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:34.899330    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d05765ac-da9e-4fad-b2e8-e26b8630849e-istio-token\" (UniqueName: \"kubernetes.io/projected/d05765ac-da9e-4fad-b2e8-e26b8630849e-istio-token\") pod \"d05765ac-da9e-4fad-b2e8-e26b8630849e\" (UID: \"d05765ac-da9e-4fad-b2e8-e26b8630849e\") "
Oct 30 19:16:34.899525 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:34.899437    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d05765ac-da9e-4fad-b2e8-e26b8630849e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d05765ac-da9e-4fad-b2e8-e26b8630849e-workload-socket\") pod \"d05765ac-da9e-4fad-b2e8-e26b8630849e\" (UID: \"d05765ac-da9e-4fad-b2e8-e26b8630849e\") "
Oct 30 19:16:34.899641 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:34.899613    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d05765ac-da9e-4fad-b2e8-e26b8630849e-istiod-ca-cert" pod "d05765ac-da9e-4fad-b2e8-e26b8630849e" (UID: "d05765ac-da9e-4fad-b2e8-e26b8630849e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:16:34.899741 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:34.899722    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d05765ac-da9e-4fad-b2e8-e26b8630849e-istio-data" pod "d05765ac-da9e-4fad-b2e8-e26b8630849e" (UID: "d05765ac-da9e-4fad-b2e8-e26b8630849e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:34.899908 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:34.899883    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d05765ac-da9e-4fad-b2e8-e26b8630849e-workload-socket" pod "d05765ac-da9e-4fad-b2e8-e26b8630849e" (UID: "d05765ac-da9e-4fad-b2e8-e26b8630849e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:34.900281 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:34.900250    4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d05765ac-da9e-4fad-b2e8-e26b8630849e-workload-certs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:34.900281 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:34.900282    4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d05765ac-da9e-4fad-b2e8-e26b8630849e-istiod-ca-cert\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:34.900472 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:34.900296    4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d05765ac-da9e-4fad-b2e8-e26b8630849e-istio-data\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:34.900472 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:34.900310    4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d05765ac-da9e-4fad-b2e8-e26b8630849e-credential-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:34.900472 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:34.900323    4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d05765ac-da9e-4fad-b2e8-e26b8630849e-workload-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:34.901518 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:34.901490    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d05765ac-da9e-4fad-b2e8-e26b8630849e-istio-envoy" pod "d05765ac-da9e-4fad-b2e8-e26b8630849e" (UID: "d05765ac-da9e-4fad-b2e8-e26b8630849e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:34.901618 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:34.901562    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d05765ac-da9e-4fad-b2e8-e26b8630849e-istio-podinfo" pod "d05765ac-da9e-4fad-b2e8-e26b8630849e" (UID: "d05765ac-da9e-4fad-b2e8-e26b8630849e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:16:34.902084 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:34.902060    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d05765ac-da9e-4fad-b2e8-e26b8630849e-kube-api-access-4bhgr" pod "d05765ac-da9e-4fad-b2e8-e26b8630849e" (UID: "d05765ac-da9e-4fad-b2e8-e26b8630849e"). InnerVolumeSpecName "kube-api-access-4bhgr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:34.902171 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:34.902089    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d05765ac-da9e-4fad-b2e8-e26b8630849e-istio-token" pod "d05765ac-da9e-4fad-b2e8-e26b8630849e" (UID: "d05765ac-da9e-4fad-b2e8-e26b8630849e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:35.001771 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:35.001725    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bhgr\" (UniqueName: \"kubernetes.io/projected/d05765ac-da9e-4fad-b2e8-e26b8630849e-kube-api-access-4bhgr\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:35.001771 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:35.001767    4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d05765ac-da9e-4fad-b2e8-e26b8630849e-istio-envoy\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:35.001771 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:35.001777    4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d05765ac-da9e-4fad-b2e8-e26b8630849e-istio-podinfo\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:35.001771 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:35.001785    4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d05765ac-da9e-4fad-b2e8-e26b8630849e-istio-token\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:35.056537 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:35.056497    4525 generic.go:356] "Generic (PLEG): container finished" podID="d05765ac-da9e-4fad-b2e8-e26b8630849e" containerID="d6768cbe6b7d423925def8357d00c26ab069b4c0fc1c95b03abdd16f7875e590" exitCode=0
Oct 30 19:16:35.057039 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:35.056550    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a1d55d-openshift-default-6646f694ff-c6hjq"
Oct 30 19:16:35.057039 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:35.056589    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a1d55d-openshift-default-6646f694ff-c6hjq" event={"ID":"d05765ac-da9e-4fad-b2e8-e26b8630849e","Type":"ContainerDied","Data":"d6768cbe6b7d423925def8357d00c26ab069b4c0fc1c95b03abdd16f7875e590"}
Oct 30 19:16:35.057039 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:35.056644    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a1d55d-openshift-default-6646f694ff-c6hjq" event={"ID":"d05765ac-da9e-4fad-b2e8-e26b8630849e","Type":"ContainerDied","Data":"d4c854c63e9fb74b6a4670da13938b913997d12619be5ca2e72372667ad4b0e8"}
Oct 30 19:16:35.057039 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:35.056670    4525 scope.go:122] "RemoveContainer" containerID="d6768cbe6b7d423925def8357d00c26ab069b4c0fc1c95b03abdd16f7875e590"
Oct 30 19:16:35.068481 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:35.068459    4525 scope.go:122] "RemoveContainer" containerID="d6768cbe6b7d423925def8357d00c26ab069b4c0fc1c95b03abdd16f7875e590"
Oct 30 19:16:35.068784 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:16:35.068762    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6768cbe6b7d423925def8357d00c26ab069b4c0fc1c95b03abdd16f7875e590\": container with ID starting with d6768cbe6b7d423925def8357d00c26ab069b4c0fc1c95b03abdd16f7875e590 not found: ID does not exist" containerID="d6768cbe6b7d423925def8357d00c26ab069b4c0fc1c95b03abdd16f7875e590"
Oct 30 19:16:35.068839 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:35.068792    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6768cbe6b7d423925def8357d00c26ab069b4c0fc1c95b03abdd16f7875e590"} err="failed to get container status \"d6768cbe6b7d423925def8357d00c26ab069b4c0fc1c95b03abdd16f7875e590\": rpc error: code = NotFound desc = could not find container \"d6768cbe6b7d423925def8357d00c26ab069b4c0fc1c95b03abdd16f7875e590\": container with ID starting with d6768cbe6b7d423925def8357d00c26ab069b4c0fc1c95b03abdd16f7875e590 not found: ID does not exist"
Oct 30 19:16:35.087942 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:35.087868    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a1d55d-openshift-default-6646f694ff-c6hjq"]
Oct 30 19:16:35.093394 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:35.093371    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a1d55d-openshift-default-6646f694ff-c6hjq"]
Oct 30 19:16:35.546100 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:35.546065    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb03ccc2-1675-482f-a64b-fa7f9bef180f" path="/var/lib/kubelet/pods/bb03ccc2-1675-482f-a64b-fa7f9bef180f/volumes"
Oct 30 19:16:35.546725 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:35.546708    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d05765ac-da9e-4fad-b2e8-e26b8630849e" path="/var/lib/kubelet/pods/d05765ac-da9e-4fad-b2e8-e26b8630849e/volumes"
Oct 30 19:16:39.552202 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:39.552171    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-ca2cd5-openshift-default-59664bf6df-xqn5s"]
Oct 30 19:16:39.552846 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:39.552349    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-ca2cd5-openshift-default-59664bf6df-xqn5s" podUID="84c44eb7-6483-4f83-ab59-447f04ba5d1d" containerName="istio-proxy" containerID="cri-o://6842a25642aa14d8c33d61180051c1fc518646a16433f9ea1fc5b7ac15e5364a" gracePeriod=30
Oct 30 19:16:40.152153 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:40.152114    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f3d6a5-dsrdc"]
Oct 30 19:16:40.152691 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:40.152661    4525 cpu_manager.go:414] "containerMap: removing container" podUID="25e308fe-6246-452a-9c60-f1304509a027" containerName="revoke-keys"
Oct 30 19:16:40.152741 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:40.152690    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25e308fe-6246-452a-9c60-f1304509a027" containerName="revoke-keys"
Oct 30 19:16:40.152741 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:40.152707    4525 cpu_manager.go:414] "containerMap: removing container" podUID="8b37419e-12f0-4e1e-8b2a-48c39c4c9e99" containerName="istio-proxy"
Oct 30 19:16:40.152741 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:40.152712    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b37419e-12f0-4e1e-8b2a-48c39c4c9e99" containerName="istio-proxy"
Oct 30 19:16:40.152741 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:40.152723    4525 cpu_manager.go:414] "containerMap: removing container" podUID="d05765ac-da9e-4fad-b2e8-e26b8630849e" containerName="istio-proxy"
Oct 30 19:16:40.152741 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:40.152728    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d05765ac-da9e-4fad-b2e8-e26b8630849e" containerName="istio-proxy"
Oct 30 19:16:40.152903 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:40.152744    4525 cpu_manager.go:414] "containerMap: removing container" podUID="0128f9e4-fbac-46b9-a7ec-2c19cac0d30a" containerName="revoke-keys"
Oct 30 19:16:40.152903 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:40.152749    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0128f9e4-fbac-46b9-a7ec-2c19cac0d30a" containerName="revoke-keys"
Oct 30 19:16:40.152903 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:40.152757    4525 cpu_manager.go:414] "containerMap: removing container" podUID="8b37419e-12f0-4e1e-8b2a-48c39c4c9e99" containerName="istio-proxy"
Oct 30 19:16:40.152903 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:40.152762    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b37419e-12f0-4e1e-8b2a-48c39c4c9e99" containerName="istio-proxy"
Oct 30 19:16:40.152903 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:40.152851    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b37419e-12f0-4e1e-8b2a-48c39c4c9e99" containerName="istio-proxy"
Oct 30 19:16:40.152903 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:40.152860    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="d05765ac-da9e-4fad-b2e8-e26b8630849e" containerName="istio-proxy"
Oct 30 19:16:40.152903 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:40.152868    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="0128f9e4-fbac-46b9-a7ec-2c19cac0d30a" containerName="revoke-keys"
Oct 30 19:16:40.152903 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:40.152878    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="25e308fe-6246-452a-9c60-f1304509a027" containerName="revoke-keys"
Oct 30 19:16:40.157928 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:40.157909    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f3d6a5-dsrdc"
Oct 30 19:16:40.171327 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:40.171303    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f3d6a5-dsrdc"]
Oct 30 19:16:40.214774 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:40.214733    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6a3bc4f-a918-46c9-9e3f-cf58fe1b499e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-f3d6a5-dsrdc\" (UID: \"c6a3bc4f-a918-46c9-9e3f-cf58fe1b499e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f3d6a5-dsrdc"
Oct 30 19:16:40.214983 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:40.214791    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6a3bc4f-a918-46c9-9e3f-cf58fe1b499e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-f3d6a5-dsrdc\" (UID: \"c6a3bc4f-a918-46c9-9e3f-cf58fe1b499e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f3d6a5-dsrdc"
Oct 30 19:16:40.215070 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:40.214984    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkq2h\" (UniqueName: \"kubernetes.io/projected/c6a3bc4f-a918-46c9-9e3f-cf58fe1b499e-kube-api-access-nkq2h\") pod \"maas-api-revoke-keys-e2e-extmod-b-f3d6a5-dsrdc\" (UID: \"c6a3bc4f-a918-46c9-9e3f-cf58fe1b499e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f3d6a5-dsrdc"
Oct 30 19:16:40.247532 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:40.247498    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:16:40.247532 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:40.247521    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94hcn\""
Oct 30 19:16:40.320241 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:40.320198    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6a3bc4f-a918-46c9-9e3f-cf58fe1b499e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-f3d6a5-dsrdc\" (UID: \"c6a3bc4f-a918-46c9-9e3f-cf58fe1b499e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f3d6a5-dsrdc"
Oct 30 19:16:40.320241 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:40.320247    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6a3bc4f-a918-46c9-9e3f-cf58fe1b499e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-f3d6a5-dsrdc\" (UID: \"c6a3bc4f-a918-46c9-9e3f-cf58fe1b499e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f3d6a5-dsrdc"
Oct 30 19:16:40.320513 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:40.320380    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkq2h\" (UniqueName: \"kubernetes.io/projected/c6a3bc4f-a918-46c9-9e3f-cf58fe1b499e-kube-api-access-nkq2h\") pod \"maas-api-revoke-keys-e2e-extmod-b-f3d6a5-dsrdc\" (UID: \"c6a3bc4f-a918-46c9-9e3f-cf58fe1b499e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f3d6a5-dsrdc"
Oct 30 19:16:40.321082 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:40.321052    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6a3bc4f-a918-46c9-9e3f-cf58fe1b499e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-f3d6a5-dsrdc\" (UID: \"c6a3bc4f-a918-46c9-9e3f-cf58fe1b499e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f3d6a5-dsrdc"
Oct 30 19:16:40.323043 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:40.323013    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6a3bc4f-a918-46c9-9e3f-cf58fe1b499e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-f3d6a5-dsrdc\" (UID: \"c6a3bc4f-a918-46c9-9e3f-cf58fe1b499e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f3d6a5-dsrdc"
Oct 30 19:16:40.333001 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:40.332977    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkq2h\" (UniqueName: \"kubernetes.io/projected/c6a3bc4f-a918-46c9-9e3f-cf58fe1b499e-kube-api-access-nkq2h\") pod \"maas-api-revoke-keys-e2e-extmod-b-f3d6a5-dsrdc\" (UID: \"c6a3bc4f-a918-46c9-9e3f-cf58fe1b499e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f3d6a5-dsrdc"
Oct 30 19:16:40.523349 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:40.523270    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f3d6a5-dsrdc"
Oct 30 19:16:40.837737 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:40.837671    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f3d6a5-dsrdc"]
Oct 30 19:16:41.118363 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:41.118322    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f3d6a5-dsrdc" event={"ID":"c6a3bc4f-a918-46c9-9e3f-cf58fe1b499e","Type":"ContainerStarted","Data":"c79eb140a8cdeddc3407f427be2bd2f34c9bdfc4236379359407aa1e02423ed3"}
Oct 30 19:16:42.238190 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:42.238119    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f3d6a5-dsrdc" event={"ID":"c6a3bc4f-a918-46c9-9e3f-cf58fe1b499e","Type":"ContainerStarted","Data":"3e384821123281bd6fd778ac90ef259efa7272103b3556402f27e5e824c3019c"}
Oct 30 19:16:43.258654 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:43.258618    4525 generic.go:356] "Generic (PLEG): container finished" podID="c6a3bc4f-a918-46c9-9e3f-cf58fe1b499e" containerID="3e384821123281bd6fd778ac90ef259efa7272103b3556402f27e5e824c3019c" exitCode=0
Oct 30 19:16:43.259226 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:43.258714    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f3d6a5-dsrdc" event={"ID":"c6a3bc4f-a918-46c9-9e3f-cf58fe1b499e","Type":"ContainerDied","Data":"3e384821123281bd6fd778ac90ef259efa7272103b3556402f27e5e824c3019c"}
Oct 30 19:16:43.421932 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:43.421907    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f3d6a5-dsrdc"
Oct 30 19:16:43.517006 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:43.516925    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6a3bc4f-a918-46c9-9e3f-cf58fe1b499e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6a3bc4f-a918-46c9-9e3f-cf58fe1b499e-maas-api-xks-ca\") pod \"c6a3bc4f-a918-46c9-9e3f-cf58fe1b499e\" (UID: \"c6a3bc4f-a918-46c9-9e3f-cf58fe1b499e\") "
Oct 30 19:16:43.517006 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:43.516961    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6a3bc4f-a918-46c9-9e3f-cf58fe1b499e-kube-api-access-nkq2h\" (UniqueName: \"kubernetes.io/projected/c6a3bc4f-a918-46c9-9e3f-cf58fe1b499e-kube-api-access-nkq2h\") pod \"c6a3bc4f-a918-46c9-9e3f-cf58fe1b499e\" (UID: \"c6a3bc4f-a918-46c9-9e3f-cf58fe1b499e\") "
Oct 30 19:16:43.517901 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:43.517017    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6a3bc4f-a918-46c9-9e3f-cf58fe1b499e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6a3bc4f-a918-46c9-9e3f-cf58fe1b499e-maas-api-openshift-ca\") pod \"c6a3bc4f-a918-46c9-9e3f-cf58fe1b499e\" (UID: \"c6a3bc4f-a918-46c9-9e3f-cf58fe1b499e\") "
Oct 30 19:16:43.517901 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:43.517402    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6a3bc4f-a918-46c9-9e3f-cf58fe1b499e-maas-api-openshift-ca" pod "c6a3bc4f-a918-46c9-9e3f-cf58fe1b499e" (UID: "c6a3bc4f-a918-46c9-9e3f-cf58fe1b499e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:16:43.519317 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:43.519286    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6a3bc4f-a918-46c9-9e3f-cf58fe1b499e-maas-api-xks-ca" pod "c6a3bc4f-a918-46c9-9e3f-cf58fe1b499e" (UID: "c6a3bc4f-a918-46c9-9e3f-cf58fe1b499e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:16:43.519317 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:43.519311    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6a3bc4f-a918-46c9-9e3f-cf58fe1b499e-kube-api-access-nkq2h" pod "c6a3bc4f-a918-46c9-9e3f-cf58fe1b499e" (UID: "c6a3bc4f-a918-46c9-9e3f-cf58fe1b499e"). InnerVolumeSpecName "kube-api-access-nkq2h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:43.619149 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:43.619108    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6a3bc4f-a918-46c9-9e3f-cf58fe1b499e-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:43.619149 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:43.619144    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkq2h\" (UniqueName: \"kubernetes.io/projected/c6a3bc4f-a918-46c9-9e3f-cf58fe1b499e-kube-api-access-nkq2h\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:43.619149 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:43.619154    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6a3bc4f-a918-46c9-9e3f-cf58fe1b499e-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:43.776419 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:43.776343    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-868e180c-xqlh2"]
Oct 30 19:16:43.776915 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:43.776900    4525 cpu_manager.go:414] "containerMap: removing container" podUID="c6a3bc4f-a918-46c9-9e3f-cf58fe1b499e" containerName="revoke-keys"
Oct 30 19:16:43.776972 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:43.776918    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6a3bc4f-a918-46c9-9e3f-cf58fe1b499e" containerName="revoke-keys"
Oct 30 19:16:43.777010 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:43.777002    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b37419e-12f0-4e1e-8b2a-48c39c4c9e99" containerName="istio-proxy"
Oct 30 19:16:43.777049 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:43.777012    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6a3bc4f-a918-46c9-9e3f-cf58fe1b499e" containerName="revoke-keys"
Oct 30 19:16:43.780528 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:43.780509    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-868e180c-xqlh2"
Oct 30 19:16:43.790580 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:43.790556    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2641515c-vf5vk"]
Oct 30 19:16:43.796563 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:43.796541    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-868e180c-xqlh2"]
Oct 30 19:16:43.796833 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:43.796810    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2641515c-vf5vk"
Oct 30 19:16:43.806905 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:43.803543    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2641515c-vf5vk"]
Oct 30 19:16:43.922979 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:43.922941    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2ae470b-e56c-49f2-ae97-f0a4ec3a98fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-868e180c-xqlh2\" (UID: \"f2ae470b-e56c-49f2-ae97-f0a4ec3a98fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-868e180c-xqlh2"
Oct 30 19:16:43.923156 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:43.922999    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7cx9\" (UniqueName: \"kubernetes.io/projected/f2ae470b-e56c-49f2-ae97-f0a4ec3a98fa-kube-api-access-n7cx9\") pod \"maas-api-revoke-sub-868e180c-xqlh2\" (UID: \"f2ae470b-e56c-49f2-ae97-f0a4ec3a98fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-868e180c-xqlh2"
Oct 30 19:16:43.923156 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:43.923084    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2ae470b-e56c-49f2-ae97-f0a4ec3a98fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-868e180c-xqlh2\" (UID: \"f2ae470b-e56c-49f2-ae97-f0a4ec3a98fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-868e180c-xqlh2"
Oct 30 19:16:43.923156 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:43.923146    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/399c5bbf-b0b3-49a1-ae99-6bc544052d99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2641515c-vf5vk\" (UID: \"399c5bbf-b0b3-49a1-ae99-6bc544052d99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2641515c-vf5vk"
Oct 30 19:16:43.923638 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:43.923185    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/399c5bbf-b0b3-49a1-ae99-6bc544052d99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2641515c-vf5vk\" (UID: \"399c5bbf-b0b3-49a1-ae99-6bc544052d99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2641515c-vf5vk"
Oct 30 19:16:43.923638 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:43.923201    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fh5sk\" (UniqueName: \"kubernetes.io/projected/399c5bbf-b0b3-49a1-ae99-6bc544052d99-kube-api-access-fh5sk\") pod \"maas-api-revoke-sub-2641515c-vf5vk\" (UID: \"399c5bbf-b0b3-49a1-ae99-6bc544052d99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2641515c-vf5vk"
Oct 30 19:16:44.024939 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:44.024908    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2ae470b-e56c-49f2-ae97-f0a4ec3a98fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-868e180c-xqlh2\" (UID: \"f2ae470b-e56c-49f2-ae97-f0a4ec3a98fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-868e180c-xqlh2"
Oct 30 19:16:44.025092 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:44.024953    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7cx9\" (UniqueName: \"kubernetes.io/projected/f2ae470b-e56c-49f2-ae97-f0a4ec3a98fa-kube-api-access-n7cx9\") pod \"maas-api-revoke-sub-868e180c-xqlh2\" (UID: \"f2ae470b-e56c-49f2-ae97-f0a4ec3a98fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-868e180c-xqlh2"
Oct 30 19:16:44.025092 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:44.024988    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2ae470b-e56c-49f2-ae97-f0a4ec3a98fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-868e180c-xqlh2\" (UID: \"f2ae470b-e56c-49f2-ae97-f0a4ec3a98fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-868e180c-xqlh2"
Oct 30 19:16:44.025092 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:44.025041    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/399c5bbf-b0b3-49a1-ae99-6bc544052d99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2641515c-vf5vk\" (UID: \"399c5bbf-b0b3-49a1-ae99-6bc544052d99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2641515c-vf5vk"
Oct 30 19:16:44.025092 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:44.025089    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/399c5bbf-b0b3-49a1-ae99-6bc544052d99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2641515c-vf5vk\" (UID: \"399c5bbf-b0b3-49a1-ae99-6bc544052d99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2641515c-vf5vk"
Oct 30 19:16:44.025317 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:44.025111    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh5sk\" (UniqueName: \"kubernetes.io/projected/399c5bbf-b0b3-49a1-ae99-6bc544052d99-kube-api-access-fh5sk\") pod \"maas-api-revoke-sub-2641515c-vf5vk\" (UID: \"399c5bbf-b0b3-49a1-ae99-6bc544052d99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2641515c-vf5vk"
Oct 30 19:16:44.025603 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:44.025577    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2ae470b-e56c-49f2-ae97-f0a4ec3a98fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-868e180c-xqlh2\" (UID: \"f2ae470b-e56c-49f2-ae97-f0a4ec3a98fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-868e180c-xqlh2"
Oct 30 19:16:44.025771 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:44.025751    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/399c5bbf-b0b3-49a1-ae99-6bc544052d99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2641515c-vf5vk\" (UID: \"399c5bbf-b0b3-49a1-ae99-6bc544052d99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2641515c-vf5vk"
Oct 30 19:16:44.027662 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:44.027619    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2ae470b-e56c-49f2-ae97-f0a4ec3a98fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-868e180c-xqlh2\" (UID: \"f2ae470b-e56c-49f2-ae97-f0a4ec3a98fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-868e180c-xqlh2"
Oct 30 19:16:44.027662 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:44.027623    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/399c5bbf-b0b3-49a1-ae99-6bc544052d99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2641515c-vf5vk\" (UID: \"399c5bbf-b0b3-49a1-ae99-6bc544052d99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2641515c-vf5vk"
Oct 30 19:16:44.034384 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:44.034362    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7cx9\" (UniqueName: \"kubernetes.io/projected/f2ae470b-e56c-49f2-ae97-f0a4ec3a98fa-kube-api-access-n7cx9\") pod \"maas-api-revoke-sub-868e180c-xqlh2\" (UID: \"f2ae470b-e56c-49f2-ae97-f0a4ec3a98fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-868e180c-xqlh2"
Oct 30 19:16:44.034494 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:44.034420    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh5sk\" (UniqueName: \"kubernetes.io/projected/399c5bbf-b0b3-49a1-ae99-6bc544052d99-kube-api-access-fh5sk\") pod \"maas-api-revoke-sub-2641515c-vf5vk\" (UID: \"399c5bbf-b0b3-49a1-ae99-6bc544052d99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2641515c-vf5vk"
Oct 30 19:16:44.105865 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:44.105840    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-868e180c-xqlh2"
Oct 30 19:16:44.119362 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:44.119285    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2641515c-vf5vk"
Oct 30 19:16:44.280855 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:44.278247    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f3d6a5-dsrdc" event={"ID":"c6a3bc4f-a918-46c9-9e3f-cf58fe1b499e","Type":"ContainerDied","Data":"c79eb140a8cdeddc3407f427be2bd2f34c9bdfc4236379359407aa1e02423ed3"}
Oct 30 19:16:44.280855 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:44.278301    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c79eb140a8cdeddc3407f427be2bd2f34c9bdfc4236379359407aa1e02423ed3"
Oct 30 19:16:44.280855 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:44.278358    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f3d6a5-dsrdc"
Oct 30 19:16:44.339890 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:44.339757    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp"]
Oct 30 19:16:44.340094 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:44.340009    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp" podUID="99aac9ad-b8dc-46cf-ac3f-69a814014bd2" containerName="istio-proxy" containerID="cri-o://491e4e8e384983e5e1bc92aa47ce905b5bd8c20fe88b7dcb3668f37382e7100c" gracePeriod=30
Oct 30 19:16:44.345248 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:44.345221    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-h4qrf"]
Oct 30 19:16:44.345949 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:44.345463    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-h4qrf" podUID="cd500131-13db-47b6-a1a7-b377e950a539" containerName="istio-proxy" containerID="cri-o://43ebfce1803a281dc319f4fb9c1ed5af0ec0a5c21c5a2daf6f78e1adc22e8a74" gracePeriod=30
Oct 30 19:16:44.590560 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:44.590532    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2641515c-vf5vk"]
Oct 30 19:16:44.597823 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:44.597800    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-868e180c-xqlh2"]
Oct 30 19:16:44.895882 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:44.895855    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ca2cd5-openshift-default-59664bf6df-xqn5s"
Oct 30 19:16:44.963102 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:44.963068    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84c44eb7-6483-4f83-ab59-447f04ba5d1d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84c44eb7-6483-4f83-ab59-447f04ba5d1d-credential-socket\") pod \"84c44eb7-6483-4f83-ab59-447f04ba5d1d\" (UID: \"84c44eb7-6483-4f83-ab59-447f04ba5d1d\") "
Oct 30 19:16:44.963378 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:44.963148    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84c44eb7-6483-4f83-ab59-447f04ba5d1d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84c44eb7-6483-4f83-ab59-447f04ba5d1d-workload-socket\") pod \"84c44eb7-6483-4f83-ab59-447f04ba5d1d\" (UID: \"84c44eb7-6483-4f83-ab59-447f04ba5d1d\") "
Oct 30 19:16:44.963378 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:44.963194    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84c44eb7-6483-4f83-ab59-447f04ba5d1d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84c44eb7-6483-4f83-ab59-447f04ba5d1d-istiod-ca-cert\") pod \"84c44eb7-6483-4f83-ab59-447f04ba5d1d\" (UID: \"84c44eb7-6483-4f83-ab59-447f04ba5d1d\") "
Oct 30 19:16:44.963378 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:44.963297    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84c44eb7-6483-4f83-ab59-447f04ba5d1d-kube-api-access-52499\" (UniqueName: \"kubernetes.io/projected/84c44eb7-6483-4f83-ab59-447f04ba5d1d-kube-api-access-52499\") pod \"84c44eb7-6483-4f83-ab59-447f04ba5d1d\" (UID: \"84c44eb7-6483-4f83-ab59-447f04ba5d1d\") "
Oct 30 19:16:44.963568 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:44.963394    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84c44eb7-6483-4f83-ab59-447f04ba5d1d-credential-socket" pod "84c44eb7-6483-4f83-ab59-447f04ba5d1d" (UID: "84c44eb7-6483-4f83-ab59-447f04ba5d1d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:44.963568 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:44.963442    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84c44eb7-6483-4f83-ab59-447f04ba5d1d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84c44eb7-6483-4f83-ab59-447f04ba5d1d-istio-data\") pod \"84c44eb7-6483-4f83-ab59-447f04ba5d1d\" (UID: \"84c44eb7-6483-4f83-ab59-447f04ba5d1d\") "
Oct 30 19:16:44.963568 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:44.963500    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84c44eb7-6483-4f83-ab59-447f04ba5d1d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84c44eb7-6483-4f83-ab59-447f04ba5d1d-workload-certs\") pod \"84c44eb7-6483-4f83-ab59-447f04ba5d1d\" (UID: \"84c44eb7-6483-4f83-ab59-447f04ba5d1d\") "
Oct 30 19:16:44.963568 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:44.963533    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84c44eb7-6483-4f83-ab59-447f04ba5d1d-workload-socket" pod "84c44eb7-6483-4f83-ab59-447f04ba5d1d" (UID: "84c44eb7-6483-4f83-ab59-447f04ba5d1d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:44.963568 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:44.963542    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/84c44eb7-6483-4f83-ab59-447f04ba5d1d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84c44eb7-6483-4f83-ab59-447f04ba5d1d-istio-podinfo\") pod \"84c44eb7-6483-4f83-ab59-447f04ba5d1d\" (UID: \"84c44eb7-6483-4f83-ab59-447f04ba5d1d\") "
Oct 30 19:16:44.963846 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:44.963603    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84c44eb7-6483-4f83-ab59-447f04ba5d1d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84c44eb7-6483-4f83-ab59-447f04ba5d1d-istio-envoy\") pod \"84c44eb7-6483-4f83-ab59-447f04ba5d1d\" (UID: \"84c44eb7-6483-4f83-ab59-447f04ba5d1d\") "
Oct 30 19:16:44.963846 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:44.963635    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84c44eb7-6483-4f83-ab59-447f04ba5d1d-istio-token\" (UniqueName: \"kubernetes.io/projected/84c44eb7-6483-4f83-ab59-447f04ba5d1d-istio-token\") pod \"84c44eb7-6483-4f83-ab59-447f04ba5d1d\" (UID: \"84c44eb7-6483-4f83-ab59-447f04ba5d1d\") "
Oct 30 19:16:44.963846 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:44.963700    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84c44eb7-6483-4f83-ab59-447f04ba5d1d-istiod-ca-cert" pod "84c44eb7-6483-4f83-ab59-447f04ba5d1d" (UID: "84c44eb7-6483-4f83-ab59-447f04ba5d1d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:16:44.963846 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:44.963778    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84c44eb7-6483-4f83-ab59-447f04ba5d1d-istio-data" pod "84c44eb7-6483-4f83-ab59-447f04ba5d1d" (UID: "84c44eb7-6483-4f83-ab59-447f04ba5d1d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:44.964270 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:44.964244    4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84c44eb7-6483-4f83-ab59-447f04ba5d1d-istio-data\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:44.964369 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:44.964276    4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84c44eb7-6483-4f83-ab59-447f04ba5d1d-credential-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:44.964369 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:44.964292    4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84c44eb7-6483-4f83-ab59-447f04ba5d1d-workload-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:44.964369 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:44.964306    4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84c44eb7-6483-4f83-ab59-447f04ba5d1d-istiod-ca-cert\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:44.964369 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:44.964311    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84c44eb7-6483-4f83-ab59-447f04ba5d1d-workload-certs" pod "84c44eb7-6483-4f83-ab59-447f04ba5d1d" (UID: "84c44eb7-6483-4f83-ab59-447f04ba5d1d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:44.966225 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:44.966200    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84c44eb7-6483-4f83-ab59-447f04ba5d1d-kube-api-access-52499" pod "84c44eb7-6483-4f83-ab59-447f04ba5d1d" (UID: "84c44eb7-6483-4f83-ab59-447f04ba5d1d"). InnerVolumeSpecName "kube-api-access-52499". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:44.966393 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:44.966378    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/84c44eb7-6483-4f83-ab59-447f04ba5d1d-istio-podinfo" pod "84c44eb7-6483-4f83-ab59-447f04ba5d1d" (UID: "84c44eb7-6483-4f83-ab59-447f04ba5d1d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:16:44.966735 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:44.966714    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84c44eb7-6483-4f83-ab59-447f04ba5d1d-istio-envoy" pod "84c44eb7-6483-4f83-ab59-447f04ba5d1d" (UID: "84c44eb7-6483-4f83-ab59-447f04ba5d1d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:44.966808 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:44.966795    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84c44eb7-6483-4f83-ab59-447f04ba5d1d-istio-token" pod "84c44eb7-6483-4f83-ab59-447f04ba5d1d" (UID: "84c44eb7-6483-4f83-ab59-447f04ba5d1d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:45.065541 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:45.065505    4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84c44eb7-6483-4f83-ab59-447f04ba5d1d-workload-certs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:45.065541 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:45.065540    4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84c44eb7-6483-4f83-ab59-447f04ba5d1d-istio-podinfo\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:45.065541 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:45.065550    4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84c44eb7-6483-4f83-ab59-447f04ba5d1d-istio-envoy\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:45.065541 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:45.065559    4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84c44eb7-6483-4f83-ab59-447f04ba5d1d-istio-token\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:45.065899 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:45.065573    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52499\" (UniqueName: \"kubernetes.io/projected/84c44eb7-6483-4f83-ab59-447f04ba5d1d-kube-api-access-52499\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:45.210342 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:45.209728    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ef6be7-46xv8"]
Oct 30 19:16:45.217493 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:45.217459    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ef6be7-46xv8"]
Oct 30 19:16:45.300087 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:45.300051    4525 generic.go:356] "Generic (PLEG): container finished" podID="84c44eb7-6483-4f83-ab59-447f04ba5d1d" containerID="6842a25642aa14d8c33d61180051c1fc518646a16433f9ea1fc5b7ac15e5364a" exitCode=0
Oct 30 19:16:45.300758 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:45.300729    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ca2cd5-openshift-default-59664bf6df-xqn5s" event={"ID":"84c44eb7-6483-4f83-ab59-447f04ba5d1d","Type":"ContainerDied","Data":"6842a25642aa14d8c33d61180051c1fc518646a16433f9ea1fc5b7ac15e5364a"}
Oct 30 19:16:45.300878 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:45.300796    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ca2cd5-openshift-default-59664bf6df-xqn5s" event={"ID":"84c44eb7-6483-4f83-ab59-447f04ba5d1d","Type":"ContainerDied","Data":"802ece6cc8f0d1270248adfcfb0cf0a553ce181a1f79354b873f579a5960ead0"}
Oct 30 19:16:45.300878 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:45.300829    4525 scope.go:122] "RemoveContainer" containerID="6842a25642aa14d8c33d61180051c1fc518646a16433f9ea1fc5b7ac15e5364a"
Oct 30 19:16:45.303794 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:45.303772    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ca2cd5-openshift-default-59664bf6df-xqn5s"
Oct 30 19:16:45.308988 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:45.308949    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2641515c-vf5vk" event={"ID":"399c5bbf-b0b3-49a1-ae99-6bc544052d99","Type":"ContainerStarted","Data":"93a14a9ce74f013033ab4a87faaf66cfa46aa45dc949f330eacf857fd04b3fe1"}
Oct 30 19:16:45.309189 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:45.309021    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2641515c-vf5vk" event={"ID":"399c5bbf-b0b3-49a1-ae99-6bc544052d99","Type":"ContainerStarted","Data":"fae1f4a47599800e83efa7e604e64c5bc31add9b47757b7c1c4424c074821060"}
Oct 30 19:16:45.319371 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:45.319342    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-868e180c-xqlh2" event={"ID":"f2ae470b-e56c-49f2-ae97-f0a4ec3a98fa","Type":"ContainerStarted","Data":"9a0a2c16ff62e5ebc7c07d0b9ee38f0c58c1a013233c7d676b321db2b188e5f9"}
Oct 30 19:16:45.324280 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:45.324245    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-868e180c-xqlh2" event={"ID":"f2ae470b-e56c-49f2-ae97-f0a4ec3a98fa","Type":"ContainerStarted","Data":"f78d1e600c468cfca01dd4e73b307cbb79b3610086d97fd6691443de4c0fff38"}
Oct 30 19:16:45.352317 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:45.352192    4525 scope.go:122] "RemoveContainer" containerID="aa921ad64f5ffd5ff960ec19542da03046337d91f1e9db386923174b3632c874"
Oct 30 19:16:45.367202 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:45.367181    4525 scope.go:122] "RemoveContainer" containerID="6842a25642aa14d8c33d61180051c1fc518646a16433f9ea1fc5b7ac15e5364a"
Oct 30 19:16:45.367476 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:16:45.367446    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6842a25642aa14d8c33d61180051c1fc518646a16433f9ea1fc5b7ac15e5364a\": container with ID starting with 6842a25642aa14d8c33d61180051c1fc518646a16433f9ea1fc5b7ac15e5364a not found: ID does not exist" containerID="6842a25642aa14d8c33d61180051c1fc518646a16433f9ea1fc5b7ac15e5364a"
Oct 30 19:16:45.367524 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:45.367490    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6842a25642aa14d8c33d61180051c1fc518646a16433f9ea1fc5b7ac15e5364a"} err="failed to get container status \"6842a25642aa14d8c33d61180051c1fc518646a16433f9ea1fc5b7ac15e5364a\": rpc error: code = NotFound desc = could not find container \"6842a25642aa14d8c33d61180051c1fc518646a16433f9ea1fc5b7ac15e5364a\": container with ID starting with 6842a25642aa14d8c33d61180051c1fc518646a16433f9ea1fc5b7ac15e5364a not found: ID does not exist"
Oct 30 19:16:45.367524 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:45.367513    4525 scope.go:122] "RemoveContainer" containerID="aa921ad64f5ffd5ff960ec19542da03046337d91f1e9db386923174b3632c874"
Oct 30 19:16:45.367864 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:16:45.367841    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa921ad64f5ffd5ff960ec19542da03046337d91f1e9db386923174b3632c874\": container with ID starting with aa921ad64f5ffd5ff960ec19542da03046337d91f1e9db386923174b3632c874 not found: ID does not exist" containerID="aa921ad64f5ffd5ff960ec19542da03046337d91f1e9db386923174b3632c874"
Oct 30 19:16:45.367955 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:45.367872    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa921ad64f5ffd5ff960ec19542da03046337d91f1e9db386923174b3632c874"} err="failed to get container status \"aa921ad64f5ffd5ff960ec19542da03046337d91f1e9db386923174b3632c874\": rpc error: code = NotFound desc = could not find container \"aa921ad64f5ffd5ff960ec19542da03046337d91f1e9db386923174b3632c874\": container with ID starting with aa921ad64f5ffd5ff960ec19542da03046337d91f1e9db386923174b3632c874 not found: ID does not exist"
Oct 30 19:16:45.373508 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:45.373464    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-868e180c-xqlh2" podStartSLOduration=2.373452414 podStartE2EDuration="2.373452414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:16:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:45.37136533 +0000 UTC m=+2427.997035343" watchObservedRunningTime="2026-10-30 19:16:45.373452414 +0000 UTC m=+2427.999122379"
Oct 30 19:16:45.374859 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:45.374826    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2641515c-vf5vk" podStartSLOduration=2.374814251 podStartE2EDuration="2.374814251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:16:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:45.350553381 +0000 UTC m=+2427.976223364" watchObservedRunningTime="2026-10-30 19:16:45.374814251 +0000 UTC m=+2428.000484215"
Oct 30 19:16:45.390446 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:45.390423    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-ca2cd5-openshift-default-59664bf6df-xqn5s"]
Oct 30 19:16:45.394767 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:45.394743    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-ca2cd5-openshift-default-59664bf6df-xqn5s"]
Oct 30 19:16:45.545373 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:45.545278    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25e308fe-6246-452a-9c60-f1304509a027" path="/var/lib/kubelet/pods/25e308fe-6246-452a-9c60-f1304509a027/volumes"
Oct 30 19:16:45.545928 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:45.545904    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84c44eb7-6483-4f83-ab59-447f04ba5d1d" path="/var/lib/kubelet/pods/84c44eb7-6483-4f83-ab59-447f04ba5d1d/volumes"
Oct 30 19:16:46.329748 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:46.329712    4525 generic.go:356] "Generic (PLEG): container finished" podID="f2ae470b-e56c-49f2-ae97-f0a4ec3a98fa" containerID="9a0a2c16ff62e5ebc7c07d0b9ee38f0c58c1a013233c7d676b321db2b188e5f9" exitCode=0
Oct 30 19:16:46.330202 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:46.329719    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-868e180c-xqlh2" event={"ID":"f2ae470b-e56c-49f2-ae97-f0a4ec3a98fa","Type":"ContainerDied","Data":"9a0a2c16ff62e5ebc7c07d0b9ee38f0c58c1a013233c7d676b321db2b188e5f9"}
Oct 30 19:16:46.332694 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:46.332658    4525 generic.go:356] "Generic (PLEG): container finished" podID="399c5bbf-b0b3-49a1-ae99-6bc544052d99" containerID="93a14a9ce74f013033ab4a87faaf66cfa46aa45dc949f330eacf857fd04b3fe1" exitCode=0
Oct 30 19:16:46.332841 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:46.332756    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2641515c-vf5vk" event={"ID":"399c5bbf-b0b3-49a1-ae99-6bc544052d99","Type":"ContainerDied","Data":"93a14a9ce74f013033ab4a87faaf66cfa46aa45dc949f330eacf857fd04b3fe1"}
Oct 30 19:16:47.414449 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:16:47.414412    4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d49l" containerName="extractor"
Oct 30 19:16:47.514909 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:47.514876    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2641515c-vf5vk"
Oct 30 19:16:47.561166 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:47.561140    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-868e180c-xqlh2"
Oct 30 19:16:47.597404 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:47.597328    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/399c5bbf-b0b3-49a1-ae99-6bc544052d99-kube-api-access-fh5sk\" (UniqueName: \"kubernetes.io/projected/399c5bbf-b0b3-49a1-ae99-6bc544052d99-kube-api-access-fh5sk\") pod \"399c5bbf-b0b3-49a1-ae99-6bc544052d99\" (UID: \"399c5bbf-b0b3-49a1-ae99-6bc544052d99\") "
Oct 30 19:16:47.597404 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:47.597387    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/399c5bbf-b0b3-49a1-ae99-6bc544052d99-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/399c5bbf-b0b3-49a1-ae99-6bc544052d99-maas-api-xks-ca\") pod \"399c5bbf-b0b3-49a1-ae99-6bc544052d99\" (UID: \"399c5bbf-b0b3-49a1-ae99-6bc544052d99\") "
Oct 30 19:16:47.597585 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:47.597546    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/399c5bbf-b0b3-49a1-ae99-6bc544052d99-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/399c5bbf-b0b3-49a1-ae99-6bc544052d99-maas-api-openshift-ca\") pod \"399c5bbf-b0b3-49a1-ae99-6bc544052d99\" (UID: \"399c5bbf-b0b3-49a1-ae99-6bc544052d99\") "
Oct 30 19:16:47.598020 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:47.597982    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/399c5bbf-b0b3-49a1-ae99-6bc544052d99-maas-api-openshift-ca" pod "399c5bbf-b0b3-49a1-ae99-6bc544052d99" (UID: "399c5bbf-b0b3-49a1-ae99-6bc544052d99"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:16:47.598263 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:47.598238    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/399c5bbf-b0b3-49a1-ae99-6bc544052d99-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:47.599987 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:47.599958    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/399c5bbf-b0b3-49a1-ae99-6bc544052d99-kube-api-access-fh5sk" pod "399c5bbf-b0b3-49a1-ae99-6bc544052d99" (UID: "399c5bbf-b0b3-49a1-ae99-6bc544052d99"). InnerVolumeSpecName "kube-api-access-fh5sk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:47.600302 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:47.600280    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/399c5bbf-b0b3-49a1-ae99-6bc544052d99-maas-api-xks-ca" pod "399c5bbf-b0b3-49a1-ae99-6bc544052d99" (UID: "399c5bbf-b0b3-49a1-ae99-6bc544052d99"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:16:47.699209 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:47.699177    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2ae470b-e56c-49f2-ae97-f0a4ec3a98fa-kube-api-access-n7cx9\" (UniqueName: \"kubernetes.io/projected/f2ae470b-e56c-49f2-ae97-f0a4ec3a98fa-kube-api-access-n7cx9\") pod \"f2ae470b-e56c-49f2-ae97-f0a4ec3a98fa\" (UID: \"f2ae470b-e56c-49f2-ae97-f0a4ec3a98fa\") "
Oct 30 19:16:47.699421 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:47.699237    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2ae470b-e56c-49f2-ae97-f0a4ec3a98fa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2ae470b-e56c-49f2-ae97-f0a4ec3a98fa-maas-api-xks-ca\") pod \"f2ae470b-e56c-49f2-ae97-f0a4ec3a98fa\" (UID: \"f2ae470b-e56c-49f2-ae97-f0a4ec3a98fa\") "
Oct 30 19:16:47.699421 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:47.699333    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2ae470b-e56c-49f2-ae97-f0a4ec3a98fa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2ae470b-e56c-49f2-ae97-f0a4ec3a98fa-maas-api-openshift-ca\") pod \"f2ae470b-e56c-49f2-ae97-f0a4ec3a98fa\" (UID: \"f2ae470b-e56c-49f2-ae97-f0a4ec3a98fa\") "
Oct 30 19:16:47.699654 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:47.699627    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2ae470b-e56c-49f2-ae97-f0a4ec3a98fa-maas-api-openshift-ca" pod "f2ae470b-e56c-49f2-ae97-f0a4ec3a98fa" (UID: "f2ae470b-e56c-49f2-ae97-f0a4ec3a98fa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:16:47.699818 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:47.699673    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fh5sk\" (UniqueName: \"kubernetes.io/projected/399c5bbf-b0b3-49a1-ae99-6bc544052d99-kube-api-access-fh5sk\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:47.699818 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:47.699702    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/399c5bbf-b0b3-49a1-ae99-6bc544052d99-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:47.701513 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:47.701494    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2ae470b-e56c-49f2-ae97-f0a4ec3a98fa-kube-api-access-n7cx9" pod "f2ae470b-e56c-49f2-ae97-f0a4ec3a98fa" (UID: "f2ae470b-e56c-49f2-ae97-f0a4ec3a98fa"). InnerVolumeSpecName "kube-api-access-n7cx9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:47.701572 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:47.701512    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2ae470b-e56c-49f2-ae97-f0a4ec3a98fa-maas-api-xks-ca" pod "f2ae470b-e56c-49f2-ae97-f0a4ec3a98fa" (UID: "f2ae470b-e56c-49f2-ae97-f0a4ec3a98fa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:16:47.800569 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:47.800534    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7cx9\" (UniqueName: \"kubernetes.io/projected/f2ae470b-e56c-49f2-ae97-f0a4ec3a98fa-kube-api-access-n7cx9\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:47.800569 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:47.800565    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2ae470b-e56c-49f2-ae97-f0a4ec3a98fa-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:47.800569 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:47.800576    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2ae470b-e56c-49f2-ae97-f0a4ec3a98fa-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:48.074170 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:48.074129    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj" podUID="84c99c49-1d60-4176-bda4-cad29c7d801c" containerName="main" containerID="cri-o://4e5700a3ef9944f99c7b66b54ee07727051d81b2a5be20c6653b3dd34b9c3a9a" gracePeriod=45
Oct 30 19:16:48.074380 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:48.074128    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv" podUID="595230e3-55f3-43eb-9a7b-6d89250af9b1" containerName="main" containerID="cri-o://1f987361694e0e8596a7c31276cf9bbd2b27d8df7ae6928a9cce99c8c1dd3846" gracePeriod=45
Oct 30 19:16:48.110064 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:48.110022    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t" podUID="424fca14-6d8a-4663-86ec-0c91165f34ee" containerName="main" containerID="cri-o://3bf2d5d3b7b75971e6e70c2687e5ce91006ba56f8cf2121445ce947b7dfb8d08" gracePeriod=45
Oct 30 19:16:48.381924 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:48.380321    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2641515c-vf5vk" event={"ID":"399c5bbf-b0b3-49a1-ae99-6bc544052d99","Type":"ContainerDied","Data":"fae1f4a47599800e83efa7e604e64c5bc31add9b47757b7c1c4424c074821060"}
Oct 30 19:16:48.381924 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:48.380384    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fae1f4a47599800e83efa7e604e64c5bc31add9b47757b7c1c4424c074821060"
Oct 30 19:16:48.381924 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:48.380397    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2641515c-vf5vk"
Oct 30 19:16:48.387584 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:48.387546    4525 generic.go:356] "Generic (PLEG): container finished" podID="424fca14-6d8a-4663-86ec-0c91165f34ee" containerID="3bf2d5d3b7b75971e6e70c2687e5ce91006ba56f8cf2121445ce947b7dfb8d08" exitCode=0
Oct 30 19:16:48.387748 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:48.387725    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t" event={"ID":"424fca14-6d8a-4663-86ec-0c91165f34ee","Type":"ContainerDied","Data":"3bf2d5d3b7b75971e6e70c2687e5ce91006ba56f8cf2121445ce947b7dfb8d08"}
Oct 30 19:16:48.391021 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:48.390350    4525 generic.go:356] "Generic (PLEG): container finished" podID="84c99c49-1d60-4176-bda4-cad29c7d801c" containerID="4e5700a3ef9944f99c7b66b54ee07727051d81b2a5be20c6653b3dd34b9c3a9a" exitCode=0
Oct 30 19:16:48.391021 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:48.390441    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj" event={"ID":"84c99c49-1d60-4176-bda4-cad29c7d801c","Type":"ContainerDied","Data":"4e5700a3ef9944f99c7b66b54ee07727051d81b2a5be20c6653b3dd34b9c3a9a"}
Oct 30 19:16:48.395973 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:48.392799    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-868e180c-xqlh2"
Oct 30 19:16:48.396126 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:48.396050    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-868e180c-xqlh2" event={"ID":"f2ae470b-e56c-49f2-ae97-f0a4ec3a98fa","Type":"ContainerDied","Data":"f78d1e600c468cfca01dd4e73b307cbb79b3610086d97fd6691443de4c0fff38"}
Oct 30 19:16:48.396126 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:48.396102    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f78d1e600c468cfca01dd4e73b307cbb79b3610086d97fd6691443de4c0fff38"
Oct 30 19:16:48.402126 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:48.402097    4525 generic.go:356] "Generic (PLEG): container finished" podID="595230e3-55f3-43eb-9a7b-6d89250af9b1" containerID="1f987361694e0e8596a7c31276cf9bbd2b27d8df7ae6928a9cce99c8c1dd3846" exitCode=0
Oct 30 19:16:48.402238 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:48.402160    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv" event={"ID":"595230e3-55f3-43eb-9a7b-6d89250af9b1","Type":"ContainerDied","Data":"1f987361694e0e8596a7c31276cf9bbd2b27d8df7ae6928a9cce99c8c1dd3846"}
Oct 30 19:16:48.423387 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:48.423360    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj"
Oct 30 19:16:48.479471 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:48.479449    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv"
Oct 30 19:16:48.483667 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:48.483641    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t"
Oct 30 19:16:48.508907 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:48.508874    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84c99c49-1d60-4176-bda4-cad29c7d801c-kube-api-access-xctzt\" (UniqueName: \"kubernetes.io/projected/84c99c49-1d60-4176-bda4-cad29c7d801c-kube-api-access-xctzt\") pod \"84c99c49-1d60-4176-bda4-cad29c7d801c\" (UID: \"84c99c49-1d60-4176-bda4-cad29c7d801c\") "
Oct 30 19:16:48.509075 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:48.508976    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84c99c49-1d60-4176-bda4-cad29c7d801c-tls-certs\" (UniqueName: \"kubernetes.io/secret/84c99c49-1d60-4176-bda4-cad29c7d801c-tls-certs\") pod \"84c99c49-1d60-4176-bda4-cad29c7d801c\" (UID: \"84c99c49-1d60-4176-bda4-cad29c7d801c\") "
Oct 30 19:16:48.509075 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:48.509023    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84c99c49-1d60-4176-bda4-cad29c7d801c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84c99c49-1d60-4176-bda4-cad29c7d801c-model-cache\") pod \"84c99c49-1d60-4176-bda4-cad29c7d801c\" (UID: \"84c99c49-1d60-4176-bda4-cad29c7d801c\") "
Oct 30 19:16:48.509583 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:48.509558    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84c99c49-1d60-4176-bda4-cad29c7d801c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/84c99c49-1d60-4176-bda4-cad29c7d801c-dshm\") pod \"84c99c49-1d60-4176-bda4-cad29c7d801c\" (UID: \"84c99c49-1d60-4176-bda4-cad29c7d801c\") "
Oct 30 19:16:48.509783 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:48.509753    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84c99c49-1d60-4176-bda4-cad29c7d801c-home\" (UniqueName: \"kubernetes.io/empty-dir/84c99c49-1d60-4176-bda4-cad29c7d801c-home\") pod \"84c99c49-1d60-4176-bda4-cad29c7d801c\" (UID: \"84c99c49-1d60-4176-bda4-cad29c7d801c\") "
Oct 30 19:16:48.510069 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:48.510037    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84c99c49-1d60-4176-bda4-cad29c7d801c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84c99c49-1d60-4176-bda4-cad29c7d801c-tmp-dir\") pod \"84c99c49-1d60-4176-bda4-cad29c7d801c\" (UID: \"84c99c49-1d60-4176-bda4-cad29c7d801c\") "
Oct 30 19:16:48.510413 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:48.510389    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84c99c49-1d60-4176-bda4-cad29c7d801c-home" pod "84c99c49-1d60-4176-bda4-cad29c7d801c" (UID: "84c99c49-1d60-4176-bda4-cad29c7d801c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:48.510586 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:48.510555    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84c99c49-1d60-4176-bda4-cad29c7d801c-model-cache" pod "84c99c49-1d60-4176-bda4-cad29c7d801c" (UID: "84c99c49-1d60-4176-bda4-cad29c7d801c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:48.510765 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:48.510747    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84c99c49-1d60-4176-bda4-cad29c7d801c-tmp-dir" pod "84c99c49-1d60-4176-bda4-cad29c7d801c" (UID: "84c99c49-1d60-4176-bda4-cad29c7d801c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:48.511159 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:48.511125    4525 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84c99c49-1d60-4176-bda4-cad29c7d801c-tmp-dir\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:48.511298 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:48.511278    4525 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84c99c49-1d60-4176-bda4-cad29c7d801c-model-cache\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:48.511427 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:48.511300    4525 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84c99c49-1d60-4176-bda4-cad29c7d801c-home\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:48.519640 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:48.519516    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84c99c49-1d60-4176-bda4-cad29c7d801c-tls-certs" pod "84c99c49-1d60-4176-bda4-cad29c7d801c" (UID: "84c99c49-1d60-4176-bda4-cad29c7d801c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:16:48.519640 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:48.519596    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84c99c49-1d60-4176-bda4-cad29c7d801c-dshm" pod "84c99c49-1d60-4176-bda4-cad29c7d801c" (UID: "84c99c49-1d60-4176-bda4-cad29c7d801c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:48.524983 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:48.524039    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84c99c49-1d60-4176-bda4-cad29c7d801c-kube-api-access-xctzt" pod "84c99c49-1d60-4176-bda4-cad29c7d801c" (UID: "84c99c49-1d60-4176-bda4-cad29c7d801c"). InnerVolumeSpecName "kube-api-access-xctzt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:48.612746 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:48.612715    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/424fca14-6d8a-4663-86ec-0c91165f34ee-home\" (UniqueName: \"kubernetes.io/empty-dir/424fca14-6d8a-4663-86ec-0c91165f34ee-home\") pod \"424fca14-6d8a-4663-86ec-0c91165f34ee\" (UID: \"424fca14-6d8a-4663-86ec-0c91165f34ee\") "
Oct 30 19:16:48.612923 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:48.612778    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/424fca14-6d8a-4663-86ec-0c91165f34ee-tls-certs\" (UniqueName: \"kubernetes.io/secret/424fca14-6d8a-4663-86ec-0c91165f34ee-tls-certs\") pod \"424fca14-6d8a-4663-86ec-0c91165f34ee\" (UID: \"424fca14-6d8a-4663-86ec-0c91165f34ee\") "
Oct 30 19:16:48.612923 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:48.612825    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/424fca14-6d8a-4663-86ec-0c91165f34ee-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/424fca14-6d8a-4663-86ec-0c91165f34ee-tmp-dir\") pod \"424fca14-6d8a-4663-86ec-0c91165f34ee\" (UID: \"424fca14-6d8a-4663-86ec-0c91165f34ee\") "
Oct 30 19:16:48.612923 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:48.612847    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/595230e3-55f3-43eb-9a7b-6d89250af9b1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/595230e3-55f3-43eb-9a7b-6d89250af9b1-model-cache\") pod \"595230e3-55f3-43eb-9a7b-6d89250af9b1\" (UID: \"595230e3-55f3-43eb-9a7b-6d89250af9b1\") "
Oct 30 19:16:48.612923 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:48.612867    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/595230e3-55f3-43eb-9a7b-6d89250af9b1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/595230e3-55f3-43eb-9a7b-6d89250af9b1-tmp-dir\") pod \"595230e3-55f3-43eb-9a7b-6d89250af9b1\" (UID: \"595230e3-55f3-43eb-9a7b-6d89250af9b1\") "
Oct 30 19:16:48.613145 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:48.612971    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/424fca14-6d8a-4663-86ec-0c91165f34ee-dshm\" (UniqueName: \"kubernetes.io/empty-dir/424fca14-6d8a-4663-86ec-0c91165f34ee-dshm\") pod \"424fca14-6d8a-4663-86ec-0c91165f34ee\" (UID: \"424fca14-6d8a-4663-86ec-0c91165f34ee\") "
Oct 30 19:16:48.613145 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:48.613015    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/424fca14-6d8a-4663-86ec-0c91165f34ee-kube-api-access-d8b28\" (UniqueName: \"kubernetes.io/projected/424fca14-6d8a-4663-86ec-0c91165f34ee-kube-api-access-d8b28\") pod \"424fca14-6d8a-4663-86ec-0c91165f34ee\" (UID: \"424fca14-6d8a-4663-86ec-0c91165f34ee\") "
Oct 30 19:16:48.613145 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:48.613075    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/595230e3-55f3-43eb-9a7b-6d89250af9b1-tls-certs\" (UniqueName: \"kubernetes.io/secret/595230e3-55f3-43eb-9a7b-6d89250af9b1-tls-certs\") pod \"595230e3-55f3-43eb-9a7b-6d89250af9b1\" (UID: \"595230e3-55f3-43eb-9a7b-6d89250af9b1\") "
Oct 30 19:16:48.613145 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:48.613102    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/595230e3-55f3-43eb-9a7b-6d89250af9b1-home\" (UniqueName: \"kubernetes.io/empty-dir/595230e3-55f3-43eb-9a7b-6d89250af9b1-home\") pod \"595230e3-55f3-43eb-9a7b-6d89250af9b1\" (UID: \"595230e3-55f3-43eb-9a7b-6d89250af9b1\") "
Oct 30 19:16:48.613145 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:48.613103    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/424fca14-6d8a-4663-86ec-0c91165f34ee-home" pod "424fca14-6d8a-4663-86ec-0c91165f34ee" (UID: "424fca14-6d8a-4663-86ec-0c91165f34ee"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:48.613432 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:48.613142    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/424fca14-6d8a-4663-86ec-0c91165f34ee-tmp-dir" pod "424fca14-6d8a-4663-86ec-0c91165f34ee" (UID: "424fca14-6d8a-4663-86ec-0c91165f34ee"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:48.613432 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:48.613158    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/595230e3-55f3-43eb-9a7b-6d89250af9b1-model-cache" pod "595230e3-55f3-43eb-9a7b-6d89250af9b1" (UID: "595230e3-55f3-43eb-9a7b-6d89250af9b1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:48.613432 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:48.613160    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/595230e3-55f3-43eb-9a7b-6d89250af9b1-kube-api-access-c5nbk\" (UniqueName: \"kubernetes.io/projected/595230e3-55f3-43eb-9a7b-6d89250af9b1-kube-api-access-c5nbk\") pod \"595230e3-55f3-43eb-9a7b-6d89250af9b1\" (UID: \"595230e3-55f3-43eb-9a7b-6d89250af9b1\") "
Oct 30 19:16:48.613432 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:48.613214    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/424fca14-6d8a-4663-86ec-0c91165f34ee-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/424fca14-6d8a-4663-86ec-0c91165f34ee-model-cache\") pod \"424fca14-6d8a-4663-86ec-0c91165f34ee\" (UID: \"424fca14-6d8a-4663-86ec-0c91165f34ee\") "
Oct 30 19:16:48.613432 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:48.613251    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/595230e3-55f3-43eb-9a7b-6d89250af9b1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/595230e3-55f3-43eb-9a7b-6d89250af9b1-dshm\") pod \"595230e3-55f3-43eb-9a7b-6d89250af9b1\" (UID: \"595230e3-55f3-43eb-9a7b-6d89250af9b1\") "
Oct 30 19:16:48.613432 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:48.613390    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/595230e3-55f3-43eb-9a7b-6d89250af9b1-home" pod "595230e3-55f3-43eb-9a7b-6d89250af9b1" (UID: "595230e3-55f3-43eb-9a7b-6d89250af9b1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:48.613953 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:48.613743    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/424fca14-6d8a-4663-86ec-0c91165f34ee-model-cache" pod "424fca14-6d8a-4663-86ec-0c91165f34ee" (UID: "424fca14-6d8a-4663-86ec-0c91165f34ee"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:48.614117 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:48.614096    4525 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/424fca14-6d8a-4663-86ec-0c91165f34ee-home\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:48.614212 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:48.614122    4525 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/424fca14-6d8a-4663-86ec-0c91165f34ee-tmp-dir\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:48.614212 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:48.614137    4525 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/595230e3-55f3-43eb-9a7b-6d89250af9b1-model-cache\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:48.614212 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:48.614152    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xctzt\" (UniqueName: \"kubernetes.io/projected/84c99c49-1d60-4176-bda4-cad29c7d801c-kube-api-access-xctzt\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:48.614212 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:48.614169    4525 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84c99c49-1d60-4176-bda4-cad29c7d801c-tls-certs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:48.614212 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:48.614183    4525 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/595230e3-55f3-43eb-9a7b-6d89250af9b1-home\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:48.614212 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:48.614198    4525 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84c99c49-1d60-4176-bda4-cad29c7d801c-dshm\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:48.614212 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:48.614210    4525 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/424fca14-6d8a-4663-86ec-0c91165f34ee-model-cache\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:48.614594 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:48.614449    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/595230e3-55f3-43eb-9a7b-6d89250af9b1-tmp-dir" pod "595230e3-55f3-43eb-9a7b-6d89250af9b1" (UID: "595230e3-55f3-43eb-9a7b-6d89250af9b1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:48.615590 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:48.615563    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/424fca14-6d8a-4663-86ec-0c91165f34ee-dshm" pod "424fca14-6d8a-4663-86ec-0c91165f34ee" (UID: "424fca14-6d8a-4663-86ec-0c91165f34ee"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:48.615772 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:48.615752    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/595230e3-55f3-43eb-9a7b-6d89250af9b1-kube-api-access-c5nbk" pod "595230e3-55f3-43eb-9a7b-6d89250af9b1" (UID: "595230e3-55f3-43eb-9a7b-6d89250af9b1"). InnerVolumeSpecName "kube-api-access-c5nbk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:48.615945 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:48.615921    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/424fca14-6d8a-4663-86ec-0c91165f34ee-tls-certs" pod "424fca14-6d8a-4663-86ec-0c91165f34ee" (UID: "424fca14-6d8a-4663-86ec-0c91165f34ee"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:16:48.615999 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:48.615986    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/595230e3-55f3-43eb-9a7b-6d89250af9b1-tls-certs" pod "595230e3-55f3-43eb-9a7b-6d89250af9b1" (UID: "595230e3-55f3-43eb-9a7b-6d89250af9b1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:16:48.616083 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:48.616060    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/424fca14-6d8a-4663-86ec-0c91165f34ee-kube-api-access-d8b28" pod "424fca14-6d8a-4663-86ec-0c91165f34ee" (UID: "424fca14-6d8a-4663-86ec-0c91165f34ee"). InnerVolumeSpecName "kube-api-access-d8b28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:48.616182 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:48.616153    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/595230e3-55f3-43eb-9a7b-6d89250af9b1-dshm" pod "595230e3-55f3-43eb-9a7b-6d89250af9b1" (UID: "595230e3-55f3-43eb-9a7b-6d89250af9b1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:48.715478 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:48.715438    4525 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/424fca14-6d8a-4663-86ec-0c91165f34ee-tls-certs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:48.715478 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:48.715469    4525 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/595230e3-55f3-43eb-9a7b-6d89250af9b1-tmp-dir\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:48.715478 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:48.715479    4525 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/424fca14-6d8a-4663-86ec-0c91165f34ee-dshm\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:48.715756 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:48.715488    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8b28\" (UniqueName: \"kubernetes.io/projected/424fca14-6d8a-4663-86ec-0c91165f34ee-kube-api-access-d8b28\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:48.715756 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:48.715498    4525 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/595230e3-55f3-43eb-9a7b-6d89250af9b1-tls-certs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:48.715756 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:48.715507    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5nbk\" (UniqueName: \"kubernetes.io/projected/595230e3-55f3-43eb-9a7b-6d89250af9b1-kube-api-access-c5nbk\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:48.715756 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:48.715515    4525 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/595230e3-55f3-43eb-9a7b-6d89250af9b1-dshm\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:49.392518 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:49.392471    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t" podUID="424fca14-6d8a-4663-86ec-0c91165f34ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:16:49.412755 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:49.412665    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj"
Oct 30 19:16:49.412929 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:49.412706    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj" event={"ID":"84c99c49-1d60-4176-bda4-cad29c7d801c","Type":"ContainerDied","Data":"f152ac13cb8664c977bbf2508f57a16be644adc8b8935ee2eae71cc30724ca2d"}
Oct 30 19:16:49.412929 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:49.412832    4525 scope.go:122] "RemoveContainer" containerID="4e5700a3ef9944f99c7b66b54ee07727051d81b2a5be20c6653b3dd34b9c3a9a"
Oct 30 19:16:49.414493 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:49.414476    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv"
Oct 30 19:16:49.414598 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:49.414516    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv" event={"ID":"595230e3-55f3-43eb-9a7b-6d89250af9b1","Type":"ContainerDied","Data":"764ea897293e1a4e9835fdd389457a033a0af866b02c30786620a9466241365e"}
Oct 30 19:16:49.416280 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:49.416258    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t" event={"ID":"424fca14-6d8a-4663-86ec-0c91165f34ee","Type":"ContainerDied","Data":"145ce1a5461a2f5fdc34a2adf2210cbca7271df2909bb071632204bfa49f295c"}
Oct 30 19:16:49.416402 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:49.416281    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t"
Oct 30 19:16:49.463455 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:49.463430    4525 scope.go:122] "RemoveContainer" containerID="1f987361694e0e8596a7c31276cf9bbd2b27d8df7ae6928a9cce99c8c1dd3846"
Oct 30 19:16:49.477256 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:49.477230    4525 scope.go:122] "RemoveContainer" containerID="3bf2d5d3b7b75971e6e70c2687e5ce91006ba56f8cf2121445ce947b7dfb8d08"
Oct 30 19:16:49.480139 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:49.480031    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj"]
Oct 30 19:16:49.493729 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:49.485323    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-2889dd/distinct-w3-2889dd-kserve-5db7f7bcf5-j5jzj"]
Oct 30 19:16:49.500915 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:49.500890    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv"]
Oct 30 19:16:49.504396 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:49.504372    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-2889dd/premium-w3-2889dd-kserve-7f95495776-mfmxv"]
Oct 30 19:16:49.523421 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:49.523305    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t"]
Oct 30 19:16:49.531023 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:49.530986    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-2889dd/simulator-w3-2889dd-kserve-6d78d68bc7-rwp7t"]
Oct 30 19:16:49.551338 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:49.551299    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="424fca14-6d8a-4663-86ec-0c91165f34ee" path="/var/lib/kubelet/pods/424fca14-6d8a-4663-86ec-0c91165f34ee/volumes"
Oct 30 19:16:49.551897 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:49.551875    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="595230e3-55f3-43eb-9a7b-6d89250af9b1" path="/var/lib/kubelet/pods/595230e3-55f3-43eb-9a7b-6d89250af9b1/volumes"
Oct 30 19:16:49.552389 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:49.552370    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84c99c49-1d60-4176-bda4-cad29c7d801c" path="/var/lib/kubelet/pods/84c99c49-1d60-4176-bda4-cad29c7d801c/volumes"
Oct 30 19:16:49.656833 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:49.656801    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp"
Oct 30 19:16:49.703312 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:49.703282    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-h4qrf"
Oct 30 19:16:49.731311 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:49.731271    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99aac9ad-b8dc-46cf-ac3f-69a814014bd2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99aac9ad-b8dc-46cf-ac3f-69a814014bd2-istiod-ca-cert\") pod \"99aac9ad-b8dc-46cf-ac3f-69a814014bd2\" (UID: \"99aac9ad-b8dc-46cf-ac3f-69a814014bd2\") "
Oct 30 19:16:49.731518 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:49.731339    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99aac9ad-b8dc-46cf-ac3f-69a814014bd2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99aac9ad-b8dc-46cf-ac3f-69a814014bd2-istio-envoy\") pod \"99aac9ad-b8dc-46cf-ac3f-69a814014bd2\" (UID: \"99aac9ad-b8dc-46cf-ac3f-69a814014bd2\") "
Oct 30 19:16:49.731518 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:49.731406    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99aac9ad-b8dc-46cf-ac3f-69a814014bd2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99aac9ad-b8dc-46cf-ac3f-69a814014bd2-credential-socket\") pod \"99aac9ad-b8dc-46cf-ac3f-69a814014bd2\" (UID: \"99aac9ad-b8dc-46cf-ac3f-69a814014bd2\") "
Oct 30 19:16:49.731518 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:49.731468    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99aac9ad-b8dc-46cf-ac3f-69a814014bd2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99aac9ad-b8dc-46cf-ac3f-69a814014bd2-workload-socket\") pod \"99aac9ad-b8dc-46cf-ac3f-69a814014bd2\" (UID: \"99aac9ad-b8dc-46cf-ac3f-69a814014bd2\") "
Oct 30 19:16:49.731518 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:49.731507    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99aac9ad-b8dc-46cf-ac3f-69a814014bd2-istio-token\" (UniqueName: \"kubernetes.io/projected/99aac9ad-b8dc-46cf-ac3f-69a814014bd2-istio-token\") pod \"99aac9ad-b8dc-46cf-ac3f-69a814014bd2\" (UID: \"99aac9ad-b8dc-46cf-ac3f-69a814014bd2\") "
Oct 30 19:16:49.731832 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:49.731567    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99aac9ad-b8dc-46cf-ac3f-69a814014bd2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99aac9ad-b8dc-46cf-ac3f-69a814014bd2-workload-certs\") pod \"99aac9ad-b8dc-46cf-ac3f-69a814014bd2\" (UID: \"99aac9ad-b8dc-46cf-ac3f-69a814014bd2\") "
Oct 30 19:16:49.731832 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:49.731604    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/99aac9ad-b8dc-46cf-ac3f-69a814014bd2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99aac9ad-b8dc-46cf-ac3f-69a814014bd2-istio-podinfo\") pod \"99aac9ad-b8dc-46cf-ac3f-69a814014bd2\" (UID: \"99aac9ad-b8dc-46cf-ac3f-69a814014bd2\") "
Oct 30 19:16:49.731832 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:49.731706    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99aac9ad-b8dc-46cf-ac3f-69a814014bd2-kube-api-access-ccjq4\" (UniqueName: \"kubernetes.io/projected/99aac9ad-b8dc-46cf-ac3f-69a814014bd2-kube-api-access-ccjq4\") pod \"99aac9ad-b8dc-46cf-ac3f-69a814014bd2\" (UID: \"99aac9ad-b8dc-46cf-ac3f-69a814014bd2\") "
Oct 30 19:16:49.731832 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:49.731779    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99aac9ad-b8dc-46cf-ac3f-69a814014bd2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99aac9ad-b8dc-46cf-ac3f-69a814014bd2-istio-data\") pod \"99aac9ad-b8dc-46cf-ac3f-69a814014bd2\" (UID: \"99aac9ad-b8dc-46cf-ac3f-69a814014bd2\") "
Oct 30 19:16:49.732228 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:49.732201    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99aac9ad-b8dc-46cf-ac3f-69a814014bd2-workload-socket" pod "99aac9ad-b8dc-46cf-ac3f-69a814014bd2" (UID: "99aac9ad-b8dc-46cf-ac3f-69a814014bd2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:49.732451 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:49.732390    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99aac9ad-b8dc-46cf-ac3f-69a814014bd2-credential-socket" pod "99aac9ad-b8dc-46cf-ac3f-69a814014bd2" (UID: "99aac9ad-b8dc-46cf-ac3f-69a814014bd2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:49.732577 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:49.732518    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99aac9ad-b8dc-46cf-ac3f-69a814014bd2-istio-data" pod "99aac9ad-b8dc-46cf-ac3f-69a814014bd2" (UID: "99aac9ad-b8dc-46cf-ac3f-69a814014bd2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:49.733000 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:49.732961    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99aac9ad-b8dc-46cf-ac3f-69a814014bd2-workload-certs" pod "99aac9ad-b8dc-46cf-ac3f-69a814014bd2" (UID: "99aac9ad-b8dc-46cf-ac3f-69a814014bd2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:49.733000 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:49.732979    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99aac9ad-b8dc-46cf-ac3f-69a814014bd2-istiod-ca-cert" pod "99aac9ad-b8dc-46cf-ac3f-69a814014bd2" (UID: "99aac9ad-b8dc-46cf-ac3f-69a814014bd2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:16:49.734728 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:49.734645    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99aac9ad-b8dc-46cf-ac3f-69a814014bd2-istio-envoy" pod "99aac9ad-b8dc-46cf-ac3f-69a814014bd2" (UID: "99aac9ad-b8dc-46cf-ac3f-69a814014bd2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:49.735160 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:49.735117    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99aac9ad-b8dc-46cf-ac3f-69a814014bd2-kube-api-access-ccjq4" pod "99aac9ad-b8dc-46cf-ac3f-69a814014bd2" (UID: "99aac9ad-b8dc-46cf-ac3f-69a814014bd2"). InnerVolumeSpecName "kube-api-access-ccjq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:49.735306 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:49.735284    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/99aac9ad-b8dc-46cf-ac3f-69a814014bd2-istio-podinfo" pod "99aac9ad-b8dc-46cf-ac3f-69a814014bd2" (UID: "99aac9ad-b8dc-46cf-ac3f-69a814014bd2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:16:49.735487 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:49.735463    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99aac9ad-b8dc-46cf-ac3f-69a814014bd2-istio-token" pod "99aac9ad-b8dc-46cf-ac3f-69a814014bd2" (UID: "99aac9ad-b8dc-46cf-ac3f-69a814014bd2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:49.833506 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:49.833471    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd500131-13db-47b6-a1a7-b377e950a539-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd500131-13db-47b6-a1a7-b377e950a539-istio-envoy\") pod \"cd500131-13db-47b6-a1a7-b377e950a539\" (UID: \"cd500131-13db-47b6-a1a7-b377e950a539\") "
Oct 30 19:16:49.833506 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:49.833513    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd500131-13db-47b6-a1a7-b377e950a539-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd500131-13db-47b6-a1a7-b377e950a539-workload-socket\") pod \"cd500131-13db-47b6-a1a7-b377e950a539\" (UID: \"cd500131-13db-47b6-a1a7-b377e950a539\") "
Oct 30 19:16:49.833839 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:49.833543    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd500131-13db-47b6-a1a7-b377e950a539-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd500131-13db-47b6-a1a7-b377e950a539-workload-certs\") pod \"cd500131-13db-47b6-a1a7-b377e950a539\" (UID: \"cd500131-13db-47b6-a1a7-b377e950a539\") "
Oct 30 19:16:49.833839 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:49.833575    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd500131-13db-47b6-a1a7-b377e950a539-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd500131-13db-47b6-a1a7-b377e950a539-istiod-ca-cert\") pod \"cd500131-13db-47b6-a1a7-b377e950a539\" (UID: \"cd500131-13db-47b6-a1a7-b377e950a539\") "
Oct 30 19:16:49.833839 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:49.833606    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cd500131-13db-47b6-a1a7-b377e950a539-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd500131-13db-47b6-a1a7-b377e950a539-istio-podinfo\") pod \"cd500131-13db-47b6-a1a7-b377e950a539\" (UID: \"cd500131-13db-47b6-a1a7-b377e950a539\") "
Oct 30 19:16:49.833839 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:49.833645    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd500131-13db-47b6-a1a7-b377e950a539-kube-api-access-md8vr\" (UniqueName: \"kubernetes.io/projected/cd500131-13db-47b6-a1a7-b377e950a539-kube-api-access-md8vr\") pod \"cd500131-13db-47b6-a1a7-b377e950a539\" (UID: \"cd500131-13db-47b6-a1a7-b377e950a539\") "
Oct 30 19:16:49.833839 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:49.833760    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd500131-13db-47b6-a1a7-b377e950a539-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd500131-13db-47b6-a1a7-b377e950a539-istio-data\") pod \"cd500131-13db-47b6-a1a7-b377e950a539\" (UID: \"cd500131-13db-47b6-a1a7-b377e950a539\") "
Oct 30 19:16:49.833839 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:49.833822    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd500131-13db-47b6-a1a7-b377e950a539-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd500131-13db-47b6-a1a7-b377e950a539-credential-socket\") pod \"cd500131-13db-47b6-a1a7-b377e950a539\" (UID: \"cd500131-13db-47b6-a1a7-b377e950a539\") "
Oct 30 19:16:49.834191 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:49.833857    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd500131-13db-47b6-a1a7-b377e950a539-istio-token\" (UniqueName: \"kubernetes.io/projected/cd500131-13db-47b6-a1a7-b377e950a539-istio-token\") pod \"cd500131-13db-47b6-a1a7-b377e950a539\" (UID: \"cd500131-13db-47b6-a1a7-b377e950a539\") "
Oct 30 19:16:49.834191 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:49.834156    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd500131-13db-47b6-a1a7-b377e950a539-istio-data" pod "cd500131-13db-47b6-a1a7-b377e950a539" (UID: "cd500131-13db-47b6-a1a7-b377e950a539"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:49.834191 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:49.834175    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd500131-13db-47b6-a1a7-b377e950a539-workload-socket" pod "cd500131-13db-47b6-a1a7-b377e950a539" (UID: "cd500131-13db-47b6-a1a7-b377e950a539"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:49.834367 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:49.834117    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd500131-13db-47b6-a1a7-b377e950a539-istiod-ca-cert" pod "cd500131-13db-47b6-a1a7-b377e950a539" (UID: "cd500131-13db-47b6-a1a7-b377e950a539"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:16:49.834367 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:49.834278    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd500131-13db-47b6-a1a7-b377e950a539-workload-certs" pod "cd500131-13db-47b6-a1a7-b377e950a539" (UID: "cd500131-13db-47b6-a1a7-b377e950a539"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:49.834476 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:49.834421    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd500131-13db-47b6-a1a7-b377e950a539-credential-socket" pod "cd500131-13db-47b6-a1a7-b377e950a539" (UID: "cd500131-13db-47b6-a1a7-b377e950a539"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:49.834570 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:49.834550    4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd500131-13db-47b6-a1a7-b377e950a539-workload-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:49.834633 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:49.834576    4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd500131-13db-47b6-a1a7-b377e950a539-workload-certs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:49.834633 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:49.834591    4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99aac9ad-b8dc-46cf-ac3f-69a814014bd2-workload-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:49.834633 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:49.834606    4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd500131-13db-47b6-a1a7-b377e950a539-istiod-ca-cert\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:49.834633 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:49.834620    4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99aac9ad-b8dc-46cf-ac3f-69a814014bd2-istio-token\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:49.834859 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:49.834634    4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99aac9ad-b8dc-46cf-ac3f-69a814014bd2-workload-certs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:49.834859 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:49.834647    4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99aac9ad-b8dc-46cf-ac3f-69a814014bd2-istio-podinfo\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:49.834859 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:49.834661    4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd500131-13db-47b6-a1a7-b377e950a539-istio-data\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:49.834859 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:49.834704    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccjq4\" (UniqueName: \"kubernetes.io/projected/99aac9ad-b8dc-46cf-ac3f-69a814014bd2-kube-api-access-ccjq4\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:49.834859 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:49.834722    4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd500131-13db-47b6-a1a7-b377e950a539-credential-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:49.834859 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:49.834735    4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99aac9ad-b8dc-46cf-ac3f-69a814014bd2-istio-data\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:49.834859 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:49.834747    4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99aac9ad-b8dc-46cf-ac3f-69a814014bd2-istiod-ca-cert\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:49.834859 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:49.834761    4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99aac9ad-b8dc-46cf-ac3f-69a814014bd2-istio-envoy\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:49.834859 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:49.834775    4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99aac9ad-b8dc-46cf-ac3f-69a814014bd2-credential-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:49.836191 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:49.836162    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cd500131-13db-47b6-a1a7-b377e950a539-istio-podinfo" pod "cd500131-13db-47b6-a1a7-b377e950a539" (UID: "cd500131-13db-47b6-a1a7-b377e950a539"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:16:49.836290 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:49.836269    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd500131-13db-47b6-a1a7-b377e950a539-istio-token" pod "cd500131-13db-47b6-a1a7-b377e950a539" (UID: "cd500131-13db-47b6-a1a7-b377e950a539"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:49.836591 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:49.836572    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd500131-13db-47b6-a1a7-b377e950a539-istio-envoy" pod "cd500131-13db-47b6-a1a7-b377e950a539" (UID: "cd500131-13db-47b6-a1a7-b377e950a539"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:49.836661 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:49.836601    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd500131-13db-47b6-a1a7-b377e950a539-kube-api-access-md8vr" pod "cd500131-13db-47b6-a1a7-b377e950a539" (UID: "cd500131-13db-47b6-a1a7-b377e950a539"). InnerVolumeSpecName "kube-api-access-md8vr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:49.937503 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:49.936906    4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd500131-13db-47b6-a1a7-b377e950a539-istio-token\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:49.937503 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:49.936949    4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd500131-13db-47b6-a1a7-b377e950a539-istio-envoy\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:49.939045 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:49.938971    4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd500131-13db-47b6-a1a7-b377e950a539-istio-podinfo\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:49.939045 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:49.939007    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-md8vr\" (UniqueName: \"kubernetes.io/projected/cd500131-13db-47b6-a1a7-b377e950a539-kube-api-access-md8vr\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:50.434141 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:50.434100    4525 generic.go:356] "Generic (PLEG): container finished" podID="cd500131-13db-47b6-a1a7-b377e950a539" containerID="43ebfce1803a281dc319f4fb9c1ed5af0ec0a5c21c5a2daf6f78e1adc22e8a74" exitCode=0
Oct 30 19:16:50.434141 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:50.434139    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-h4qrf" event={"ID":"cd500131-13db-47b6-a1a7-b377e950a539","Type":"ContainerDied","Data":"43ebfce1803a281dc319f4fb9c1ed5af0ec0a5c21c5a2daf6f78e1adc22e8a74"}
Oct 30 19:16:50.434547 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:50.434167    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-h4qrf"
Oct 30 19:16:50.434547 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:50.434212    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-h4qrf" event={"ID":"cd500131-13db-47b6-a1a7-b377e950a539","Type":"ContainerDied","Data":"2caf11bff9ead7b3d364b6494ca142d61f4cfbfa680532295cf07296a2cf601f"}
Oct 30 19:16:50.434547 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:50.434246    4525 scope.go:122] "RemoveContainer" containerID="43ebfce1803a281dc319f4fb9c1ed5af0ec0a5c21c5a2daf6f78e1adc22e8a74"
Oct 30 19:16:50.442566 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:50.442529    4525 generic.go:356] "Generic (PLEG): container finished" podID="99aac9ad-b8dc-46cf-ac3f-69a814014bd2" containerID="491e4e8e384983e5e1bc92aa47ce905b5bd8c20fe88b7dcb3668f37382e7100c" exitCode=0
Oct 30 19:16:50.442879 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:50.442709    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp" event={"ID":"99aac9ad-b8dc-46cf-ac3f-69a814014bd2","Type":"ContainerDied","Data":"491e4e8e384983e5e1bc92aa47ce905b5bd8c20fe88b7dcb3668f37382e7100c"}
Oct 30 19:16:50.442879 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:50.442763    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp" event={"ID":"99aac9ad-b8dc-46cf-ac3f-69a814014bd2","Type":"ContainerDied","Data":"12a5024eb19a90cd471f14953d730849f3acd42d98ebd3540324639b2c6fb9e2"}
Oct 30 19:16:50.442879 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:50.442859    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp"
Oct 30 19:16:50.456914 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:50.456894    4525 scope.go:122] "RemoveContainer" containerID="43ebfce1803a281dc319f4fb9c1ed5af0ec0a5c21c5a2daf6f78e1adc22e8a74"
Oct 30 19:16:50.457204 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:16:50.457184    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43ebfce1803a281dc319f4fb9c1ed5af0ec0a5c21c5a2daf6f78e1adc22e8a74\": container with ID starting with 43ebfce1803a281dc319f4fb9c1ed5af0ec0a5c21c5a2daf6f78e1adc22e8a74 not found: ID does not exist" containerID="43ebfce1803a281dc319f4fb9c1ed5af0ec0a5c21c5a2daf6f78e1adc22e8a74"
Oct 30 19:16:50.457258 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:50.457213    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43ebfce1803a281dc319f4fb9c1ed5af0ec0a5c21c5a2daf6f78e1adc22e8a74"} err="failed to get container status \"43ebfce1803a281dc319f4fb9c1ed5af0ec0a5c21c5a2daf6f78e1adc22e8a74\": rpc error: code = NotFound desc = could not find container \"43ebfce1803a281dc319f4fb9c1ed5af0ec0a5c21c5a2daf6f78e1adc22e8a74\": container with ID starting with 43ebfce1803a281dc319f4fb9c1ed5af0ec0a5c21c5a2daf6f78e1adc22e8a74 not found: ID does not exist"
Oct 30 19:16:50.457258 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:50.457227    4525 scope.go:122] "RemoveContainer" containerID="491e4e8e384983e5e1bc92aa47ce905b5bd8c20fe88b7dcb3668f37382e7100c"
Oct 30 19:16:50.467722 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:50.467705    4525 scope.go:122] "RemoveContainer" containerID="a10ca651e5feab589cc19423939c8ccde1c0e0ea87a11276221f4529ff26c8d0"
Oct 30 19:16:50.479942 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:50.479786    4525 scope.go:122] "RemoveContainer" containerID="491e4e8e384983e5e1bc92aa47ce905b5bd8c20fe88b7dcb3668f37382e7100c"
Oct 30 19:16:50.480573 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:50.480556    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp"]
Oct 30 19:16:50.480787 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:16:50.480757    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"491e4e8e384983e5e1bc92aa47ce905b5bd8c20fe88b7dcb3668f37382e7100c\": container with ID starting with 491e4e8e384983e5e1bc92aa47ce905b5bd8c20fe88b7dcb3668f37382e7100c not found: ID does not exist" containerID="491e4e8e384983e5e1bc92aa47ce905b5bd8c20fe88b7dcb3668f37382e7100c"
Oct 30 19:16:50.480874 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:50.480802    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"491e4e8e384983e5e1bc92aa47ce905b5bd8c20fe88b7dcb3668f37382e7100c"} err="failed to get container status \"491e4e8e384983e5e1bc92aa47ce905b5bd8c20fe88b7dcb3668f37382e7100c\": rpc error: code = NotFound desc = could not find container \"491e4e8e384983e5e1bc92aa47ce905b5bd8c20fe88b7dcb3668f37382e7100c\": container with ID starting with 491e4e8e384983e5e1bc92aa47ce905b5bd8c20fe88b7dcb3668f37382e7100c not found: ID does not exist"
Oct 30 19:16:50.480874 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:50.480827    4525 scope.go:122] "RemoveContainer" containerID="a10ca651e5feab589cc19423939c8ccde1c0e0ea87a11276221f4529ff26c8d0"
Oct 30 19:16:50.481146 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:16:50.481127    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a10ca651e5feab589cc19423939c8ccde1c0e0ea87a11276221f4529ff26c8d0\": container with ID starting with a10ca651e5feab589cc19423939c8ccde1c0e0ea87a11276221f4529ff26c8d0 not found: ID does not exist" containerID="a10ca651e5feab589cc19423939c8ccde1c0e0ea87a11276221f4529ff26c8d0"
Oct 30 19:16:50.481223 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:50.481154    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a10ca651e5feab589cc19423939c8ccde1c0e0ea87a11276221f4529ff26c8d0"} err="failed to get container status \"a10ca651e5feab589cc19423939c8ccde1c0e0ea87a11276221f4529ff26c8d0\": rpc error: code = NotFound desc = could not find container \"a10ca651e5feab589cc19423939c8ccde1c0e0ea87a11276221f4529ff26c8d0\": container with ID starting with a10ca651e5feab589cc19423939c8ccde1c0e0ea87a11276221f4529ff26c8d0 not found: ID does not exist"
Oct 30 19:16:50.484842 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:50.484818    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-nrbmp"]
Oct 30 19:16:50.499361 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:50.499336    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-h4qrf"]
Oct 30 19:16:50.504174 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:50.504150    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-0458df-openshift-default-7c86d79f79-h4qrf"]
Oct 30 19:16:51.543972 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:51.543935    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99aac9ad-b8dc-46cf-ac3f-69a814014bd2" path="/var/lib/kubelet/pods/99aac9ad-b8dc-46cf-ac3f-69a814014bd2/volumes"
Oct 30 19:16:51.544491 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:51.544475    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd500131-13db-47b6-a1a7-b377e950a539" path="/var/lib/kubelet/pods/cd500131-13db-47b6-a1a7-b377e950a539/volumes"
Oct 30 19:16:52.576446 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:52.576412    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f3d6a5-dsrdc"]
Oct 30 19:16:52.582856 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:52.582823    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f3d6a5-dsrdc"]
Oct 30 19:16:52.793672 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:52.793638    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-7ttp9"]
Oct 30 19:16:52.793933 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:52.793887    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-7ttp9" podUID="960cce33-8574-42ab-8d16-c4fb2f0b3202" containerName="istio-proxy" containerID="cri-o://751fa8df5468e8cb96d391b1299dffa22a214e78bf156730654cd06fe3c2700a" gracePeriod=30
Oct 30 19:16:52.798851 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:52.798827    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-qlfgx"]
Oct 30 19:16:52.799027 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:52.799007    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-qlfgx" podUID="9a3d6a70-b055-40dc-98a7-d0f5890293c8" containerName="istio-proxy" containerID="cri-o://043728e1c4c8db9a53c02c0fdf9069f6c450cc50fef4da5da344e8ccf891eb85" gracePeriod=30
Oct 30 19:16:52.803436 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:52.803411    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-t8s45"]
Oct 30 19:16:52.803620 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:52.803599    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-t8s45" podUID="06b7dec0-c44b-4aea-a753-53ec832dfd24" containerName="istio-proxy" containerID="cri-o://a54b4418f6fea83f1a4e0b6cbd3057eeda100829b0edf905d2215e1d3c909fca" gracePeriod=30
Oct 30 19:16:52.936753 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:52.935923    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn"]
Oct 30 19:16:52.936753 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:52.936261    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn" podUID="e77e530a-8e80-4371-840e-b2345bf68449" containerName="istio-proxy" containerID="cri-o://d740cc14106fe5bc6d83a0eb11e4e9b6d757b4e0a0ff175aea9962d2509198e7" gracePeriod=30
Oct 30 19:16:53.545352 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:53.545314    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6a3bc4f-a918-46c9-9e3f-cf58fe1b499e" path="/var/lib/kubelet/pods/c6a3bc4f-a918-46c9-9e3f-cf58fe1b499e/volumes"
Oct 30 19:16:53.659412 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:53.659381    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-39bde3-kcldj"]
Oct 30 19:16:53.659881 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:53.659868    4525 cpu_manager.go:414] "containerMap: removing container" podUID="424fca14-6d8a-4663-86ec-0c91165f34ee" containerName="main"
Oct 30 19:16:53.659929 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:53.659883    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="424fca14-6d8a-4663-86ec-0c91165f34ee" containerName="main"
Oct 30 19:16:53.659929 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:53.659897    4525 cpu_manager.go:414] "containerMap: removing container" podUID="84c99c49-1d60-4176-bda4-cad29c7d801c" containerName="main"
Oct 30 19:16:53.659929 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:53.659903    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84c99c49-1d60-4176-bda4-cad29c7d801c" containerName="main"
Oct 30 19:16:53.659929 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:53.659909    4525 cpu_manager.go:414] "containerMap: removing container" podUID="f2ae470b-e56c-49f2-ae97-f0a4ec3a98fa" containerName="revoke-keys"
Oct 30 19:16:53.659929 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:53.659915    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2ae470b-e56c-49f2-ae97-f0a4ec3a98fa" containerName="revoke-keys"
Oct 30 19:16:53.659929 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:53.659926    4525 cpu_manager.go:414] "containerMap: removing container" podUID="cd500131-13db-47b6-a1a7-b377e950a539" containerName="istio-proxy"
Oct 30 19:16:53.659929 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:53.659931    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd500131-13db-47b6-a1a7-b377e950a539" containerName="istio-proxy"
Oct 30 19:16:53.660168 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:53.659943    4525 cpu_manager.go:414] "containerMap: removing container" podUID="595230e3-55f3-43eb-9a7b-6d89250af9b1" containerName="main"
Oct 30 19:16:53.660168 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:53.659947    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="595230e3-55f3-43eb-9a7b-6d89250af9b1" containerName="main"
Oct 30 19:16:53.660168 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:53.659959    4525 cpu_manager.go:414] "containerMap: removing container" podUID="84c44eb7-6483-4f83-ab59-447f04ba5d1d" containerName="istio-proxy"
Oct 30 19:16:53.660168 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:53.659964    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84c44eb7-6483-4f83-ab59-447f04ba5d1d" containerName="istio-proxy"
Oct 30 19:16:53.660168 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:53.659972    4525 cpu_manager.go:414] "containerMap: removing container" podUID="399c5bbf-b0b3-49a1-ae99-6bc544052d99" containerName="revoke-keys"
Oct 30 19:16:53.660168 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:53.659976    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="399c5bbf-b0b3-49a1-ae99-6bc544052d99" containerName="revoke-keys"
Oct 30 19:16:53.660168 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:53.659982    4525 cpu_manager.go:414] "containerMap: removing container" podUID="99aac9ad-b8dc-46cf-ac3f-69a814014bd2" containerName="istio-proxy"
Oct 30 19:16:53.660168 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:53.659986    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99aac9ad-b8dc-46cf-ac3f-69a814014bd2" containerName="istio-proxy"
Oct 30 19:16:53.660168 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:53.659995    4525 cpu_manager.go:414] "containerMap: removing container" podUID="84c44eb7-6483-4f83-ab59-447f04ba5d1d" containerName="istio-proxy"
Oct 30 19:16:53.660168 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:53.659999    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84c44eb7-6483-4f83-ab59-447f04ba5d1d" containerName="istio-proxy"
Oct 30 19:16:53.660168 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:53.660007    4525 cpu_manager.go:414] "containerMap: removing container" podUID="99aac9ad-b8dc-46cf-ac3f-69a814014bd2" containerName="istio-proxy"
Oct 30 19:16:53.660168 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:53.660011    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99aac9ad-b8dc-46cf-ac3f-69a814014bd2" containerName="istio-proxy"
Oct 30 19:16:53.660168 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:53.660088    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="424fca14-6d8a-4663-86ec-0c91165f34ee" containerName="main"
Oct 30 19:16:53.660168 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:53.660098    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="84c99c49-1d60-4176-bda4-cad29c7d801c" containerName="main"
Oct 30 19:16:53.660168 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:53.660104    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2ae470b-e56c-49f2-ae97-f0a4ec3a98fa" containerName="revoke-keys"
Oct 30 19:16:53.660168 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:53.660109    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd500131-13db-47b6-a1a7-b377e950a539" containerName="istio-proxy"
Oct 30 19:16:53.660168 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:53.660115    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="595230e3-55f3-43eb-9a7b-6d89250af9b1" containerName="main"
Oct 30 19:16:53.660168 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:53.660121    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="84c44eb7-6483-4f83-ab59-447f04ba5d1d" containerName="istio-proxy"
Oct 30 19:16:53.660168 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:53.660126    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="399c5bbf-b0b3-49a1-ae99-6bc544052d99" containerName="revoke-keys"
Oct 30 19:16:53.660168 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:53.660131    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="99aac9ad-b8dc-46cf-ac3f-69a814014bd2" containerName="istio-proxy"
Oct 30 19:16:53.660168 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:53.660137    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="99aac9ad-b8dc-46cf-ac3f-69a814014bd2" containerName="istio-proxy"
Oct 30 19:16:53.665001 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:53.664984    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-39bde3-kcldj"
Oct 30 19:16:53.674650 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:53.674610    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:16:53.674995 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:53.674977    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94hcn\""
Oct 30 19:16:53.677116 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:53.677089    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-39bde3-kcldj"]
Oct 30 19:16:53.809270 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:53.809176    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hdjj\" (UniqueName: \"kubernetes.io/projected/9f3d341b-70f0-415d-b624-e691ecc6db16-kube-api-access-2hdjj\") pod \"maas-api-revoke-keys-e2e-worker-w2-39bde3-kcldj\" (UID: \"9f3d341b-70f0-415d-b624-e691ecc6db16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-39bde3-kcldj"
Oct 30 19:16:53.809445 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:53.809304    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f3d341b-70f0-415d-b624-e691ecc6db16-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-39bde3-kcldj\" (UID: \"9f3d341b-70f0-415d-b624-e691ecc6db16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-39bde3-kcldj"
Oct 30 19:16:53.809445 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:53.809399    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f3d341b-70f0-415d-b624-e691ecc6db16-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-39bde3-kcldj\" (UID: \"9f3d341b-70f0-415d-b624-e691ecc6db16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-39bde3-kcldj"
Oct 30 19:16:53.910951 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:53.910911    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f3d341b-70f0-415d-b624-e691ecc6db16-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-39bde3-kcldj\" (UID: \"9f3d341b-70f0-415d-b624-e691ecc6db16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-39bde3-kcldj"
Oct 30 19:16:53.911130 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:53.911010    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hdjj\" (UniqueName: \"kubernetes.io/projected/9f3d341b-70f0-415d-b624-e691ecc6db16-kube-api-access-2hdjj\") pod \"maas-api-revoke-keys-e2e-worker-w2-39bde3-kcldj\" (UID: \"9f3d341b-70f0-415d-b624-e691ecc6db16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-39bde3-kcldj"
Oct 30 19:16:53.911130 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:53.911066    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f3d341b-70f0-415d-b624-e691ecc6db16-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-39bde3-kcldj\" (UID: \"9f3d341b-70f0-415d-b624-e691ecc6db16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-39bde3-kcldj"
Oct 30 19:16:53.911716 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:53.911671    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f3d341b-70f0-415d-b624-e691ecc6db16-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-39bde3-kcldj\" (UID: \"9f3d341b-70f0-415d-b624-e691ecc6db16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-39bde3-kcldj"
Oct 30 19:16:53.914327 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:53.914304    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f3d341b-70f0-415d-b624-e691ecc6db16-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-39bde3-kcldj\" (UID: \"9f3d341b-70f0-415d-b624-e691ecc6db16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-39bde3-kcldj"
Oct 30 19:16:53.919871 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:53.919843    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hdjj\" (UniqueName: \"kubernetes.io/projected/9f3d341b-70f0-415d-b624-e691ecc6db16-kube-api-access-2hdjj\") pod \"maas-api-revoke-keys-e2e-worker-w2-39bde3-kcldj\" (UID: \"9f3d341b-70f0-415d-b624-e691ecc6db16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-39bde3-kcldj"
Oct 30 19:16:53.983660 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:53.983628    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-39bde3-kcldj"
Oct 30 19:16:54.265793 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:54.265761    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-39bde3-kcldj"]
Oct 30 19:16:54.266826 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:16:54.266800    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f3d341b_70f0_415d_b624_e691ecc6db16.slice/crio-758c987a1a032d83184072418062b5d9dea90f133359875251d02765d7deac2b WatchSource:0}: Error finding container 758c987a1a032d83184072418062b5d9dea90f133359875251d02765d7deac2b: Status 404 returned error can't find the container with id 758c987a1a032d83184072418062b5d9dea90f133359875251d02765d7deac2b
Oct 30 19:16:54.512332 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:54.512294    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-39bde3-kcldj" event={"ID":"9f3d341b-70f0-415d-b624-e691ecc6db16","Type":"ContainerStarted","Data":"758c987a1a032d83184072418062b5d9dea90f133359875251d02765d7deac2b"}
Oct 30 19:16:55.521255 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:55.521167    4525 generic.go:356] "Generic (PLEG): container finished" podID="9f3d341b-70f0-415d-b624-e691ecc6db16" containerID="4f0b6e2e1f6f55d698e95836013917f7ef65260d566d3b3170100a32d60eede0" exitCode=0
Oct 30 19:16:55.521647 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:55.521257    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-39bde3-kcldj" event={"ID":"9f3d341b-70f0-415d-b624-e691ecc6db16","Type":"ContainerDied","Data":"4f0b6e2e1f6f55d698e95836013917f7ef65260d566d3b3170100a32d60eede0"}
Oct 30 19:16:56.713322 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:56.713288    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-39bde3-kcldj"
Oct 30 19:16:56.847075 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:56.846986    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f3d341b-70f0-415d-b624-e691ecc6db16-kube-api-access-2hdjj\" (UniqueName: \"kubernetes.io/projected/9f3d341b-70f0-415d-b624-e691ecc6db16-kube-api-access-2hdjj\") pod \"9f3d341b-70f0-415d-b624-e691ecc6db16\" (UID: \"9f3d341b-70f0-415d-b624-e691ecc6db16\") "
Oct 30 19:16:56.847075 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:56.847070    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f3d341b-70f0-415d-b624-e691ecc6db16-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f3d341b-70f0-415d-b624-e691ecc6db16-maas-api-xks-ca\") pod \"9f3d341b-70f0-415d-b624-e691ecc6db16\" (UID: \"9f3d341b-70f0-415d-b624-e691ecc6db16\") "
Oct 30 19:16:56.847339 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:56.847205    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f3d341b-70f0-415d-b624-e691ecc6db16-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f3d341b-70f0-415d-b624-e691ecc6db16-maas-api-openshift-ca\") pod \"9f3d341b-70f0-415d-b624-e691ecc6db16\" (UID: \"9f3d341b-70f0-415d-b624-e691ecc6db16\") "
Oct 30 19:16:56.847563 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:56.847537    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f3d341b-70f0-415d-b624-e691ecc6db16-maas-api-openshift-ca" pod "9f3d341b-70f0-415d-b624-e691ecc6db16" (UID: "9f3d341b-70f0-415d-b624-e691ecc6db16"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:16:56.849339 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:56.849318    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f3d341b-70f0-415d-b624-e691ecc6db16-maas-api-xks-ca" pod "9f3d341b-70f0-415d-b624-e691ecc6db16" (UID: "9f3d341b-70f0-415d-b624-e691ecc6db16"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:16:56.849339 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:56.849324    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f3d341b-70f0-415d-b624-e691ecc6db16-kube-api-access-2hdjj" pod "9f3d341b-70f0-415d-b624-e691ecc6db16" (UID: "9f3d341b-70f0-415d-b624-e691ecc6db16"). InnerVolumeSpecName "kube-api-access-2hdjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:56.949191 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:56.949152    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hdjj\" (UniqueName: \"kubernetes.io/projected/9f3d341b-70f0-415d-b624-e691ecc6db16-kube-api-access-2hdjj\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:56.949191 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:56.949184    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f3d341b-70f0-415d-b624-e691ecc6db16-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:56.949191 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:56.949197    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f3d341b-70f0-415d-b624-e691ecc6db16-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:57.541415 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:57.541384    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-39bde3-kcldj"
Oct 30 19:16:57.544239 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:57.544210    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-39bde3-kcldj" event={"ID":"9f3d341b-70f0-415d-b624-e691ecc6db16","Type":"ContainerDied","Data":"758c987a1a032d83184072418062b5d9dea90f133359875251d02765d7deac2b"}
Oct 30 19:16:57.544356 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:57.544254    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="758c987a1a032d83184072418062b5d9dea90f133359875251d02765d7deac2b"
Oct 30 19:16:58.115271 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.115217    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2889dd-2fxrn"]
Oct 30 19:16:58.122237 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.122209    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2889dd-2fxrn"]
Oct 30 19:16:58.153643 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.153297    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-7ttp9"
Oct 30 19:16:58.242760 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.242730    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-qlfgx"
Oct 30 19:16:58.264960 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.264929    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/960cce33-8574-42ab-8d16-c4fb2f0b3202-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/960cce33-8574-42ab-8d16-c4fb2f0b3202-credential-socket\") pod \"960cce33-8574-42ab-8d16-c4fb2f0b3202\" (UID: \"960cce33-8574-42ab-8d16-c4fb2f0b3202\") "
Oct 30 19:16:58.265160 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.264975    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/960cce33-8574-42ab-8d16-c4fb2f0b3202-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/960cce33-8574-42ab-8d16-c4fb2f0b3202-istio-envoy\") pod \"960cce33-8574-42ab-8d16-c4fb2f0b3202\" (UID: \"960cce33-8574-42ab-8d16-c4fb2f0b3202\") "
Oct 30 19:16:58.265286 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.265262    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/960cce33-8574-42ab-8d16-c4fb2f0b3202-credential-socket" pod "960cce33-8574-42ab-8d16-c4fb2f0b3202" (UID: "960cce33-8574-42ab-8d16-c4fb2f0b3202"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:58.265483 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.265458    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/960cce33-8574-42ab-8d16-c4fb2f0b3202-istio-token\" (UniqueName: \"kubernetes.io/projected/960cce33-8574-42ab-8d16-c4fb2f0b3202-istio-token\") pod \"960cce33-8574-42ab-8d16-c4fb2f0b3202\" (UID: \"960cce33-8574-42ab-8d16-c4fb2f0b3202\") "
Oct 30 19:16:58.265634 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.265524    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/960cce33-8574-42ab-8d16-c4fb2f0b3202-kube-api-access-2bjhw\" (UniqueName: \"kubernetes.io/projected/960cce33-8574-42ab-8d16-c4fb2f0b3202-kube-api-access-2bjhw\") pod \"960cce33-8574-42ab-8d16-c4fb2f0b3202\" (UID: \"960cce33-8574-42ab-8d16-c4fb2f0b3202\") "
Oct 30 19:16:58.265634 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.265597    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/960cce33-8574-42ab-8d16-c4fb2f0b3202-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/960cce33-8574-42ab-8d16-c4fb2f0b3202-istiod-ca-cert\") pod \"960cce33-8574-42ab-8d16-c4fb2f0b3202\" (UID: \"960cce33-8574-42ab-8d16-c4fb2f0b3202\") "
Oct 30 19:16:58.265809 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.265650    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/960cce33-8574-42ab-8d16-c4fb2f0b3202-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/960cce33-8574-42ab-8d16-c4fb2f0b3202-istio-data\") pod \"960cce33-8574-42ab-8d16-c4fb2f0b3202\" (UID: \"960cce33-8574-42ab-8d16-c4fb2f0b3202\") "
Oct 30 19:16:58.265809 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.265787    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/960cce33-8574-42ab-8d16-c4fb2f0b3202-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/960cce33-8574-42ab-8d16-c4fb2f0b3202-workload-certs\") pod \"960cce33-8574-42ab-8d16-c4fb2f0b3202\" (UID: \"960cce33-8574-42ab-8d16-c4fb2f0b3202\") "
Oct 30 19:16:58.265921 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.265845    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/960cce33-8574-42ab-8d16-c4fb2f0b3202-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/960cce33-8574-42ab-8d16-c4fb2f0b3202-istio-podinfo\") pod \"960cce33-8574-42ab-8d16-c4fb2f0b3202\" (UID: \"960cce33-8574-42ab-8d16-c4fb2f0b3202\") "
Oct 30 19:16:58.265921 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.265887    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/960cce33-8574-42ab-8d16-c4fb2f0b3202-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/960cce33-8574-42ab-8d16-c4fb2f0b3202-workload-socket\") pod \"960cce33-8574-42ab-8d16-c4fb2f0b3202\" (UID: \"960cce33-8574-42ab-8d16-c4fb2f0b3202\") "
Oct 30 19:16:58.266426 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.266398    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/960cce33-8574-42ab-8d16-c4fb2f0b3202-workload-certs" pod "960cce33-8574-42ab-8d16-c4fb2f0b3202" (UID: "960cce33-8574-42ab-8d16-c4fb2f0b3202"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:58.266613 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.266590    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/960cce33-8574-42ab-8d16-c4fb2f0b3202-istio-data" pod "960cce33-8574-42ab-8d16-c4fb2f0b3202" (UID: "960cce33-8574-42ab-8d16-c4fb2f0b3202"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:58.266772 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.266752    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/960cce33-8574-42ab-8d16-c4fb2f0b3202-istiod-ca-cert" pod "960cce33-8574-42ab-8d16-c4fb2f0b3202" (UID: "960cce33-8574-42ab-8d16-c4fb2f0b3202"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:16:58.267182 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.267164    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/960cce33-8574-42ab-8d16-c4fb2f0b3202-workload-socket" pod "960cce33-8574-42ab-8d16-c4fb2f0b3202" (UID: "960cce33-8574-42ab-8d16-c4fb2f0b3202"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:58.267321 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.267301    4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/960cce33-8574-42ab-8d16-c4fb2f0b3202-istiod-ca-cert\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:58.267401 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.267330    4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/960cce33-8574-42ab-8d16-c4fb2f0b3202-istio-data\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:58.267401 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.267346    4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/960cce33-8574-42ab-8d16-c4fb2f0b3202-workload-certs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:58.267401 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.267360    4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/960cce33-8574-42ab-8d16-c4fb2f0b3202-credential-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:58.267778 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.267760    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/960cce33-8574-42ab-8d16-c4fb2f0b3202-istio-envoy" pod "960cce33-8574-42ab-8d16-c4fb2f0b3202" (UID: "960cce33-8574-42ab-8d16-c4fb2f0b3202"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:58.269166 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.269134    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/960cce33-8574-42ab-8d16-c4fb2f0b3202-istio-podinfo" pod "960cce33-8574-42ab-8d16-c4fb2f0b3202" (UID: "960cce33-8574-42ab-8d16-c4fb2f0b3202"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:16:58.269263 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.269206    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/960cce33-8574-42ab-8d16-c4fb2f0b3202-kube-api-access-2bjhw" pod "960cce33-8574-42ab-8d16-c4fb2f0b3202" (UID: "960cce33-8574-42ab-8d16-c4fb2f0b3202"). InnerVolumeSpecName "kube-api-access-2bjhw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:58.276165 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.275719    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/960cce33-8574-42ab-8d16-c4fb2f0b3202-istio-token" pod "960cce33-8574-42ab-8d16-c4fb2f0b3202" (UID: "960cce33-8574-42ab-8d16-c4fb2f0b3202"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:58.287172 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.287152    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-t8s45"
Oct 30 19:16:58.367906 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.367823    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a3d6a70-b055-40dc-98a7-d0f5890293c8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a3d6a70-b055-40dc-98a7-d0f5890293c8-credential-socket\") pod \"9a3d6a70-b055-40dc-98a7-d0f5890293c8\" (UID: \"9a3d6a70-b055-40dc-98a7-d0f5890293c8\") "
Oct 30 19:16:58.367906 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.367885    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06b7dec0-c44b-4aea-a753-53ec832dfd24-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06b7dec0-c44b-4aea-a753-53ec832dfd24-workload-socket\") pod \"06b7dec0-c44b-4aea-a753-53ec832dfd24\" (UID: \"06b7dec0-c44b-4aea-a753-53ec832dfd24\") "
Oct 30 19:16:58.368124 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.367944    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06b7dec0-c44b-4aea-a753-53ec832dfd24-kube-api-access-gv7vb\" (UniqueName: \"kubernetes.io/projected/06b7dec0-c44b-4aea-a753-53ec832dfd24-kube-api-access-gv7vb\") pod \"06b7dec0-c44b-4aea-a753-53ec832dfd24\" (UID: \"06b7dec0-c44b-4aea-a753-53ec832dfd24\") "
Oct 30 19:16:58.368124 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.367983    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a3d6a70-b055-40dc-98a7-d0f5890293c8-kube-api-access-v7dzb\" (UniqueName: \"kubernetes.io/projected/9a3d6a70-b055-40dc-98a7-d0f5890293c8-kube-api-access-v7dzb\") pod \"9a3d6a70-b055-40dc-98a7-d0f5890293c8\" (UID: \"9a3d6a70-b055-40dc-98a7-d0f5890293c8\") "
Oct 30 19:16:58.368124 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.368014    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a3d6a70-b055-40dc-98a7-d0f5890293c8-istio-token\" (UniqueName: \"kubernetes.io/projected/9a3d6a70-b055-40dc-98a7-d0f5890293c8-istio-token\") pod \"9a3d6a70-b055-40dc-98a7-d0f5890293c8\" (UID: \"9a3d6a70-b055-40dc-98a7-d0f5890293c8\") "
Oct 30 19:16:58.368124 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.368068    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a3d6a70-b055-40dc-98a7-d0f5890293c8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a3d6a70-b055-40dc-98a7-d0f5890293c8-istio-data\") pod \"9a3d6a70-b055-40dc-98a7-d0f5890293c8\" (UID: \"9a3d6a70-b055-40dc-98a7-d0f5890293c8\") "
Oct 30 19:16:58.368358 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.368150    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a3d6a70-b055-40dc-98a7-d0f5890293c8-credential-socket" pod "9a3d6a70-b055-40dc-98a7-d0f5890293c8" (UID: "9a3d6a70-b055-40dc-98a7-d0f5890293c8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:58.368358 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.368174    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06b7dec0-c44b-4aea-a753-53ec832dfd24-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06b7dec0-c44b-4aea-a753-53ec832dfd24-istio-envoy\") pod \"06b7dec0-c44b-4aea-a753-53ec832dfd24\" (UID: \"06b7dec0-c44b-4aea-a753-53ec832dfd24\") "
Oct 30 19:16:58.368358 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.368219    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a3d6a70-b055-40dc-98a7-d0f5890293c8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a3d6a70-b055-40dc-98a7-d0f5890293c8-workload-certs\") pod \"9a3d6a70-b055-40dc-98a7-d0f5890293c8\" (UID: \"9a3d6a70-b055-40dc-98a7-d0f5890293c8\") "
Oct 30 19:16:58.368358 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.368257    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06b7dec0-c44b-4aea-a753-53ec832dfd24-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06b7dec0-c44b-4aea-a753-53ec832dfd24-istio-data\") pod \"06b7dec0-c44b-4aea-a753-53ec832dfd24\" (UID: \"06b7dec0-c44b-4aea-a753-53ec832dfd24\") "
Oct 30 19:16:58.368358 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.368307    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a3d6a70-b055-40dc-98a7-d0f5890293c8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a3d6a70-b055-40dc-98a7-d0f5890293c8-istiod-ca-cert\") pod \"9a3d6a70-b055-40dc-98a7-d0f5890293c8\" (UID: \"9a3d6a70-b055-40dc-98a7-d0f5890293c8\") "
Oct 30 19:16:58.368631 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.368366    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a3d6a70-b055-40dc-98a7-d0f5890293c8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a3d6a70-b055-40dc-98a7-d0f5890293c8-istio-envoy\") pod \"9a3d6a70-b055-40dc-98a7-d0f5890293c8\" (UID: \"9a3d6a70-b055-40dc-98a7-d0f5890293c8\") "
Oct 30 19:16:58.368631 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.368399    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/06b7dec0-c44b-4aea-a753-53ec832dfd24-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06b7dec0-c44b-4aea-a753-53ec832dfd24-istio-podinfo\") pod \"06b7dec0-c44b-4aea-a753-53ec832dfd24\" (UID: \"06b7dec0-c44b-4aea-a753-53ec832dfd24\") "
Oct 30 19:16:58.368631 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.368430    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9a3d6a70-b055-40dc-98a7-d0f5890293c8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a3d6a70-b055-40dc-98a7-d0f5890293c8-istio-podinfo\") pod \"9a3d6a70-b055-40dc-98a7-d0f5890293c8\" (UID: \"9a3d6a70-b055-40dc-98a7-d0f5890293c8\") "
Oct 30 19:16:58.368631 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.368460    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06b7dec0-c44b-4aea-a753-53ec832dfd24-istio-token\" (UniqueName: \"kubernetes.io/projected/06b7dec0-c44b-4aea-a753-53ec832dfd24-istio-token\") pod \"06b7dec0-c44b-4aea-a753-53ec832dfd24\" (UID: \"06b7dec0-c44b-4aea-a753-53ec832dfd24\") "
Oct 30 19:16:58.368631 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.368215    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06b7dec0-c44b-4aea-a753-53ec832dfd24-workload-socket" pod "06b7dec0-c44b-4aea-a753-53ec832dfd24" (UID: "06b7dec0-c44b-4aea-a753-53ec832dfd24"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:58.368631 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.368444    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a3d6a70-b055-40dc-98a7-d0f5890293c8-istio-data" pod "9a3d6a70-b055-40dc-98a7-d0f5890293c8" (UID: "9a3d6a70-b055-40dc-98a7-d0f5890293c8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:58.368631 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.368508    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a3d6a70-b055-40dc-98a7-d0f5890293c8-workload-certs" pod "9a3d6a70-b055-40dc-98a7-d0f5890293c8" (UID: "9a3d6a70-b055-40dc-98a7-d0f5890293c8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:58.368631 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.368559    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06b7dec0-c44b-4aea-a753-53ec832dfd24-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06b7dec0-c44b-4aea-a753-53ec832dfd24-credential-socket\") pod \"06b7dec0-c44b-4aea-a753-53ec832dfd24\" (UID: \"06b7dec0-c44b-4aea-a753-53ec832dfd24\") "
Oct 30 19:16:58.368631 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.368588    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06b7dec0-c44b-4aea-a753-53ec832dfd24-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06b7dec0-c44b-4aea-a753-53ec832dfd24-workload-certs\") pod \"06b7dec0-c44b-4aea-a753-53ec832dfd24\" (UID: \"06b7dec0-c44b-4aea-a753-53ec832dfd24\") "
Oct 30 19:16:58.368631 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.368635    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a3d6a70-b055-40dc-98a7-d0f5890293c8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a3d6a70-b055-40dc-98a7-d0f5890293c8-workload-socket\") pod \"9a3d6a70-b055-40dc-98a7-d0f5890293c8\" (UID: \"9a3d6a70-b055-40dc-98a7-d0f5890293c8\") "
Oct 30 19:16:58.369399 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.368662    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06b7dec0-c44b-4aea-a753-53ec832dfd24-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06b7dec0-c44b-4aea-a753-53ec832dfd24-istiod-ca-cert\") pod \"06b7dec0-c44b-4aea-a753-53ec832dfd24\" (UID: \"06b7dec0-c44b-4aea-a753-53ec832dfd24\") "
Oct 30 19:16:58.369399 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.368818    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a3d6a70-b055-40dc-98a7-d0f5890293c8-istiod-ca-cert" pod "9a3d6a70-b055-40dc-98a7-d0f5890293c8" (UID: "9a3d6a70-b055-40dc-98a7-d0f5890293c8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:16:58.369399 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.369241    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06b7dec0-c44b-4aea-a753-53ec832dfd24-istio-data" pod "06b7dec0-c44b-4aea-a753-53ec832dfd24" (UID: "06b7dec0-c44b-4aea-a753-53ec832dfd24"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:58.369572 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.369480    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bjhw\" (UniqueName: \"kubernetes.io/projected/960cce33-8574-42ab-8d16-c4fb2f0b3202-kube-api-access-2bjhw\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:58.369572 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.369502    4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a3d6a70-b055-40dc-98a7-d0f5890293c8-credential-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:58.369572 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.369522    4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06b7dec0-c44b-4aea-a753-53ec832dfd24-workload-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:58.369572 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.369538    4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/960cce33-8574-42ab-8d16-c4fb2f0b3202-istio-podinfo\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:58.369572 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.369554    4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/960cce33-8574-42ab-8d16-c4fb2f0b3202-workload-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:58.369572 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.369557    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06b7dec0-c44b-4aea-a753-53ec832dfd24-istiod-ca-cert" pod "06b7dec0-c44b-4aea-a753-53ec832dfd24" (UID: "06b7dec0-c44b-4aea-a753-53ec832dfd24"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:16:58.369927 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.369568    4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a3d6a70-b055-40dc-98a7-d0f5890293c8-workload-certs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:58.369927 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.369602    4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06b7dec0-c44b-4aea-a753-53ec832dfd24-istio-data\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:58.369927 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.369622    4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/960cce33-8574-42ab-8d16-c4fb2f0b3202-istio-envoy\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:58.369927 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.369636    4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a3d6a70-b055-40dc-98a7-d0f5890293c8-istiod-ca-cert\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:58.369927 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.369652    4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/960cce33-8574-42ab-8d16-c4fb2f0b3202-istio-token\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:58.370306 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.369938    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06b7dec0-c44b-4aea-a753-53ec832dfd24-workload-certs" pod "06b7dec0-c44b-4aea-a753-53ec832dfd24" (UID: "06b7dec0-c44b-4aea-a753-53ec832dfd24"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:58.370306 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.370141    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a3d6a70-b055-40dc-98a7-d0f5890293c8-workload-socket" pod "9a3d6a70-b055-40dc-98a7-d0f5890293c8" (UID: "9a3d6a70-b055-40dc-98a7-d0f5890293c8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:58.370306 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.370262    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06b7dec0-c44b-4aea-a753-53ec832dfd24-credential-socket" pod "06b7dec0-c44b-4aea-a753-53ec832dfd24" (UID: "06b7dec0-c44b-4aea-a753-53ec832dfd24"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:58.371245 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.371214    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06b7dec0-c44b-4aea-a753-53ec832dfd24-istio-envoy" pod "06b7dec0-c44b-4aea-a753-53ec832dfd24" (UID: "06b7dec0-c44b-4aea-a753-53ec832dfd24"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:58.371397 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.371294    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a3d6a70-b055-40dc-98a7-d0f5890293c8-istio-podinfo" pod "9a3d6a70-b055-40dc-98a7-d0f5890293c8" (UID: "9a3d6a70-b055-40dc-98a7-d0f5890293c8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:16:58.371397 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.371313    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a3d6a70-b055-40dc-98a7-d0f5890293c8-kube-api-access-v7dzb" pod "9a3d6a70-b055-40dc-98a7-d0f5890293c8" (UID: "9a3d6a70-b055-40dc-98a7-d0f5890293c8"). InnerVolumeSpecName "kube-api-access-v7dzb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:58.371665 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.371542    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06b7dec0-c44b-4aea-a753-53ec832dfd24-kube-api-access-gv7vb" pod "06b7dec0-c44b-4aea-a753-53ec832dfd24" (UID: "06b7dec0-c44b-4aea-a753-53ec832dfd24"). InnerVolumeSpecName "kube-api-access-gv7vb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:58.371665 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.371653    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a3d6a70-b055-40dc-98a7-d0f5890293c8-istio-token" pod "9a3d6a70-b055-40dc-98a7-d0f5890293c8" (UID: "9a3d6a70-b055-40dc-98a7-d0f5890293c8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:58.372329 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.372304    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06b7dec0-c44b-4aea-a753-53ec832dfd24-istio-token" pod "06b7dec0-c44b-4aea-a753-53ec832dfd24" (UID: "06b7dec0-c44b-4aea-a753-53ec832dfd24"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:58.372966 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.372938    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a3d6a70-b055-40dc-98a7-d0f5890293c8-istio-envoy" pod "9a3d6a70-b055-40dc-98a7-d0f5890293c8" (UID: "9a3d6a70-b055-40dc-98a7-d0f5890293c8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:58.373049 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.372999    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/06b7dec0-c44b-4aea-a753-53ec832dfd24-istio-podinfo" pod "06b7dec0-c44b-4aea-a753-53ec832dfd24" (UID: "06b7dec0-c44b-4aea-a753-53ec832dfd24"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:16:58.391483 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.391456    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr"]
Oct 30 19:16:58.404292 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.404259    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw"]
Oct 30 19:16:58.471243 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.471214    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gv7vb\" (UniqueName: \"kubernetes.io/projected/06b7dec0-c44b-4aea-a753-53ec832dfd24-kube-api-access-gv7vb\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:58.471243 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.471244    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7dzb\" (UniqueName: \"kubernetes.io/projected/9a3d6a70-b055-40dc-98a7-d0f5890293c8-kube-api-access-v7dzb\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:58.471393 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.471254    4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a3d6a70-b055-40dc-98a7-d0f5890293c8-istio-token\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:58.471393 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.471263    4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a3d6a70-b055-40dc-98a7-d0f5890293c8-istio-data\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:58.471393 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.471272    4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06b7dec0-c44b-4aea-a753-53ec832dfd24-istio-envoy\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:58.471393 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.471281    4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a3d6a70-b055-40dc-98a7-d0f5890293c8-istio-envoy\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:58.471393 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.471291    4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06b7dec0-c44b-4aea-a753-53ec832dfd24-istio-podinfo\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:58.471393 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.471300    4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a3d6a70-b055-40dc-98a7-d0f5890293c8-istio-podinfo\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:58.471393 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.471313    4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06b7dec0-c44b-4aea-a753-53ec832dfd24-istio-token\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:58.471393 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.471328    4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06b7dec0-c44b-4aea-a753-53ec832dfd24-credential-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:58.471393 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.471340    4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06b7dec0-c44b-4aea-a753-53ec832dfd24-workload-certs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:58.471393 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.471350    4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a3d6a70-b055-40dc-98a7-d0f5890293c8-workload-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:58.471393 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.471359    4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06b7dec0-c44b-4aea-a753-53ec832dfd24-istiod-ca-cert\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:58.613471 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.613434    4525 generic.go:356] "Generic (PLEG): container finished" podID="06b7dec0-c44b-4aea-a753-53ec832dfd24" containerID="a54b4418f6fea83f1a4e0b6cbd3057eeda100829b0edf905d2215e1d3c909fca" exitCode=0
Oct 30 19:16:58.613728 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.613541    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-t8s45" event={"ID":"06b7dec0-c44b-4aea-a753-53ec832dfd24","Type":"ContainerDied","Data":"a54b4418f6fea83f1a4e0b6cbd3057eeda100829b0edf905d2215e1d3c909fca"}
Oct 30 19:16:58.613728 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.613617    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-t8s45" event={"ID":"06b7dec0-c44b-4aea-a753-53ec832dfd24","Type":"ContainerDied","Data":"5bd2b9f381a44f100acebf19497d7987aa65a38205e463b06c661867519d8e45"}
Oct 30 19:16:58.613728 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.613651    4525 scope.go:122] "RemoveContainer" containerID="a54b4418f6fea83f1a4e0b6cbd3057eeda100829b0edf905d2215e1d3c909fca"
Oct 30 19:16:58.616344 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.613859    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-t8s45"
Oct 30 19:16:58.620977 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.620946    4525 generic.go:356] "Generic (PLEG): container finished" podID="e77e530a-8e80-4371-840e-b2345bf68449" containerID="d740cc14106fe5bc6d83a0eb11e4e9b6d757b4e0a0ff175aea9962d2509198e7" exitCode=0
Oct 30 19:16:58.621117 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.621054    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn" event={"ID":"e77e530a-8e80-4371-840e-b2345bf68449","Type":"ContainerDied","Data":"d740cc14106fe5bc6d83a0eb11e4e9b6d757b4e0a0ff175aea9962d2509198e7"}
Oct 30 19:16:58.623003 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.622975    4525 generic.go:356] "Generic (PLEG): container finished" podID="960cce33-8574-42ab-8d16-c4fb2f0b3202" containerID="751fa8df5468e8cb96d391b1299dffa22a214e78bf156730654cd06fe3c2700a" exitCode=0
Oct 30 19:16:58.623137 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.623103    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-7ttp9" event={"ID":"960cce33-8574-42ab-8d16-c4fb2f0b3202","Type":"ContainerDied","Data":"751fa8df5468e8cb96d391b1299dffa22a214e78bf156730654cd06fe3c2700a"}
Oct 30 19:16:58.623266 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.623150    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-7ttp9" event={"ID":"960cce33-8574-42ab-8d16-c4fb2f0b3202","Type":"ContainerDied","Data":"8828279bae189514301e4d950df713ca0d1a177cd42dcbbb8e15b469a3049701"}
Oct 30 19:16:58.624809 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.624789    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-7ttp9"
Oct 30 19:16:58.625309 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.625287    4525 generic.go:356] "Generic (PLEG): container finished" podID="9a3d6a70-b055-40dc-98a7-d0f5890293c8" containerID="043728e1c4c8db9a53c02c0fdf9069f6c450cc50fef4da5da344e8ccf891eb85" exitCode=0
Oct 30 19:16:58.625410 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.625370    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-qlfgx"
Oct 30 19:16:58.625465 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.625368    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-qlfgx" event={"ID":"9a3d6a70-b055-40dc-98a7-d0f5890293c8","Type":"ContainerDied","Data":"043728e1c4c8db9a53c02c0fdf9069f6c450cc50fef4da5da344e8ccf891eb85"}
Oct 30 19:16:58.625513 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.625485    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-qlfgx" event={"ID":"9a3d6a70-b055-40dc-98a7-d0f5890293c8","Type":"ContainerDied","Data":"7ba1f869f50ef1fea758fee34fba182f12d57cd14196da00eda76aa266e6fe59"}
Oct 30 19:16:58.637067 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.635547    4525 scope.go:122] "RemoveContainer" containerID="a54b4418f6fea83f1a4e0b6cbd3057eeda100829b0edf905d2215e1d3c909fca"
Oct 30 19:16:58.637067 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:16:58.635952    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a54b4418f6fea83f1a4e0b6cbd3057eeda100829b0edf905d2215e1d3c909fca\": container with ID starting with a54b4418f6fea83f1a4e0b6cbd3057eeda100829b0edf905d2215e1d3c909fca not found: ID does not exist" containerID="a54b4418f6fea83f1a4e0b6cbd3057eeda100829b0edf905d2215e1d3c909fca"
Oct 30 19:16:58.637067 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.635990    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a54b4418f6fea83f1a4e0b6cbd3057eeda100829b0edf905d2215e1d3c909fca"} err="failed to get container status \"a54b4418f6fea83f1a4e0b6cbd3057eeda100829b0edf905d2215e1d3c909fca\": rpc error: code = NotFound desc = could not find container \"a54b4418f6fea83f1a4e0b6cbd3057eeda100829b0edf905d2215e1d3c909fca\": container with ID starting with a54b4418f6fea83f1a4e0b6cbd3057eeda100829b0edf905d2215e1d3c909fca not found: ID does not exist"
Oct 30 19:16:58.637067 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.636019    4525 scope.go:122] "RemoveContainer" containerID="bc1fc55950a91a7538ea3218d9447d112374babdbb61184d2633be1fa66bd903"
Oct 30 19:16:58.675085 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.675054    4525 scope.go:122] "RemoveContainer" containerID="751fa8df5468e8cb96d391b1299dffa22a214e78bf156730654cd06fe3c2700a"
Oct 30 19:16:58.681968 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.681943    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-7ttp9"]
Oct 30 19:16:58.694159 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.691168    4525 scope.go:122] "RemoveContainer" containerID="751fa8df5468e8cb96d391b1299dffa22a214e78bf156730654cd06fe3c2700a"
Oct 30 19:16:58.694159 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:16:58.691585    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"751fa8df5468e8cb96d391b1299dffa22a214e78bf156730654cd06fe3c2700a\": container with ID starting with 751fa8df5468e8cb96d391b1299dffa22a214e78bf156730654cd06fe3c2700a not found: ID does not exist" containerID="751fa8df5468e8cb96d391b1299dffa22a214e78bf156730654cd06fe3c2700a"
Oct 30 19:16:58.694159 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.691638    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"751fa8df5468e8cb96d391b1299dffa22a214e78bf156730654cd06fe3c2700a"} err="failed to get container status \"751fa8df5468e8cb96d391b1299dffa22a214e78bf156730654cd06fe3c2700a\": rpc error: code = NotFound desc = could not find container \"751fa8df5468e8cb96d391b1299dffa22a214e78bf156730654cd06fe3c2700a\": container with ID starting with 751fa8df5468e8cb96d391b1299dffa22a214e78bf156730654cd06fe3c2700a not found: ID does not exist"
Oct 30 19:16:58.694159 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.691662    4525 scope.go:122] "RemoveContainer" containerID="043728e1c4c8db9a53c02c0fdf9069f6c450cc50fef4da5da344e8ccf891eb85"
Oct 30 19:16:58.696451 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.696426    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-7ttp9"]
Oct 30 19:16:58.703066 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.703040    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn"
Oct 30 19:16:58.704801 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.704777    4525 scope.go:122] "RemoveContainer" containerID="043728e1c4c8db9a53c02c0fdf9069f6c450cc50fef4da5da344e8ccf891eb85"
Oct 30 19:16:58.705121 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:16:58.705094    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"043728e1c4c8db9a53c02c0fdf9069f6c450cc50fef4da5da344e8ccf891eb85\": container with ID starting with 043728e1c4c8db9a53c02c0fdf9069f6c450cc50fef4da5da344e8ccf891eb85 not found: ID does not exist" containerID="043728e1c4c8db9a53c02c0fdf9069f6c450cc50fef4da5da344e8ccf891eb85"
Oct 30 19:16:58.705204 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.705130    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"043728e1c4c8db9a53c02c0fdf9069f6c450cc50fef4da5da344e8ccf891eb85"} err="failed to get container status \"043728e1c4c8db9a53c02c0fdf9069f6c450cc50fef4da5da344e8ccf891eb85\": rpc error: code = NotFound desc = could not find container \"043728e1c4c8db9a53c02c0fdf9069f6c450cc50fef4da5da344e8ccf891eb85\": container with ID starting with 043728e1c4c8db9a53c02c0fdf9069f6c450cc50fef4da5da344e8ccf891eb85 not found: ID does not exist"
Oct 30 19:16:58.709058 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.709038    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-t8s45"]
Oct 30 19:16:58.718790 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.718761    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-t8s45"]
Oct 30 19:16:58.728911 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.728887    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-qlfgx"]
Oct 30 19:16:58.739283 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.739236    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-qlfgx"]
Oct 30 19:16:58.790810 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.790776    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e77e530a-8e80-4371-840e-b2345bf68449-kube-api-access-zcb67\" (UniqueName: \"kubernetes.io/projected/e77e530a-8e80-4371-840e-b2345bf68449-kube-api-access-zcb67\") pod \"e77e530a-8e80-4371-840e-b2345bf68449\" (UID: \"e77e530a-8e80-4371-840e-b2345bf68449\") "
Oct 30 19:16:58.791012 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.790835    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e77e530a-8e80-4371-840e-b2345bf68449-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e77e530a-8e80-4371-840e-b2345bf68449-istio-data\") pod \"e77e530a-8e80-4371-840e-b2345bf68449\" (UID: \"e77e530a-8e80-4371-840e-b2345bf68449\") "
Oct 30 19:16:58.791012 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.790874    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e77e530a-8e80-4371-840e-b2345bf68449-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e77e530a-8e80-4371-840e-b2345bf68449-istio-podinfo\") pod \"e77e530a-8e80-4371-840e-b2345bf68449\" (UID: \"e77e530a-8e80-4371-840e-b2345bf68449\") "
Oct 30 19:16:58.791012 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.790953    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e77e530a-8e80-4371-840e-b2345bf68449-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e77e530a-8e80-4371-840e-b2345bf68449-workload-certs\") pod \"e77e530a-8e80-4371-840e-b2345bf68449\" (UID: \"e77e530a-8e80-4371-840e-b2345bf68449\") "
Oct 30 19:16:58.791192 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.791023    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e77e530a-8e80-4371-840e-b2345bf68449-istio-token\" (UniqueName: \"kubernetes.io/projected/e77e530a-8e80-4371-840e-b2345bf68449-istio-token\") pod \"e77e530a-8e80-4371-840e-b2345bf68449\" (UID: \"e77e530a-8e80-4371-840e-b2345bf68449\") "
Oct 30 19:16:58.791192 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.791100    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e77e530a-8e80-4371-840e-b2345bf68449-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e77e530a-8e80-4371-840e-b2345bf68449-workload-socket\") pod \"e77e530a-8e80-4371-840e-b2345bf68449\" (UID: \"e77e530a-8e80-4371-840e-b2345bf68449\") "
Oct 30 19:16:58.791192 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.791167    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e77e530a-8e80-4371-840e-b2345bf68449-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e77e530a-8e80-4371-840e-b2345bf68449-credential-socket\") pod \"e77e530a-8e80-4371-840e-b2345bf68449\" (UID: \"e77e530a-8e80-4371-840e-b2345bf68449\") "
Oct 30 19:16:58.791319 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.791211    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e77e530a-8e80-4371-840e-b2345bf68449-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e77e530a-8e80-4371-840e-b2345bf68449-istiod-ca-cert\") pod \"e77e530a-8e80-4371-840e-b2345bf68449\" (UID: \"e77e530a-8e80-4371-840e-b2345bf68449\") "
Oct 30 19:16:58.791319 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.791244    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e77e530a-8e80-4371-840e-b2345bf68449-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e77e530a-8e80-4371-840e-b2345bf68449-istio-envoy\") pod \"e77e530a-8e80-4371-840e-b2345bf68449\" (UID: \"e77e530a-8e80-4371-840e-b2345bf68449\") "
Oct 30 19:16:58.791319 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.791262    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e77e530a-8e80-4371-840e-b2345bf68449-istio-data" pod "e77e530a-8e80-4371-840e-b2345bf68449" (UID: "e77e530a-8e80-4371-840e-b2345bf68449"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:58.791493 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.791465    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e77e530a-8e80-4371-840e-b2345bf68449-workload-certs" pod "e77e530a-8e80-4371-840e-b2345bf68449" (UID: "e77e530a-8e80-4371-840e-b2345bf68449"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:58.791660 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.791622    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e77e530a-8e80-4371-840e-b2345bf68449-workload-socket" pod "e77e530a-8e80-4371-840e-b2345bf68449" (UID: "e77e530a-8e80-4371-840e-b2345bf68449"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:58.792017 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.791821    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e77e530a-8e80-4371-840e-b2345bf68449-credential-socket" pod "e77e530a-8e80-4371-840e-b2345bf68449" (UID: "e77e530a-8e80-4371-840e-b2345bf68449"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:58.792017 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.791888    4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e77e530a-8e80-4371-840e-b2345bf68449-workload-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:58.792017 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.791910    4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e77e530a-8e80-4371-840e-b2345bf68449-istio-data\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:58.792017 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.791923    4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e77e530a-8e80-4371-840e-b2345bf68449-workload-certs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:58.792205 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.792084    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e77e530a-8e80-4371-840e-b2345bf68449-istiod-ca-cert" pod "e77e530a-8e80-4371-840e-b2345bf68449" (UID: "e77e530a-8e80-4371-840e-b2345bf68449"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:16:58.796724 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.794195    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e77e530a-8e80-4371-840e-b2345bf68449-kube-api-access-zcb67" pod "e77e530a-8e80-4371-840e-b2345bf68449" (UID: "e77e530a-8e80-4371-840e-b2345bf68449"). InnerVolumeSpecName "kube-api-access-zcb67". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:58.796724 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.794334    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e77e530a-8e80-4371-840e-b2345bf68449-istio-envoy" pod "e77e530a-8e80-4371-840e-b2345bf68449" (UID: "e77e530a-8e80-4371-840e-b2345bf68449"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:58.796724 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.794586    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e77e530a-8e80-4371-840e-b2345bf68449-istio-token" pod "e77e530a-8e80-4371-840e-b2345bf68449" (UID: "e77e530a-8e80-4371-840e-b2345bf68449"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:58.796724 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.794653    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e77e530a-8e80-4371-840e-b2345bf68449-istio-podinfo" pod "e77e530a-8e80-4371-840e-b2345bf68449" (UID: "e77e530a-8e80-4371-840e-b2345bf68449"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:16:58.893068 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.892980    4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e77e530a-8e80-4371-840e-b2345bf68449-istio-token\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:58.893068 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.893015    4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e77e530a-8e80-4371-840e-b2345bf68449-credential-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:58.893068 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.893024    4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e77e530a-8e80-4371-840e-b2345bf68449-istiod-ca-cert\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:58.893068 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.893033    4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e77e530a-8e80-4371-840e-b2345bf68449-istio-envoy\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:58.893068 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.893043    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcb67\" (UniqueName: \"kubernetes.io/projected/e77e530a-8e80-4371-840e-b2345bf68449-kube-api-access-zcb67\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:58.893068 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:58.893055    4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e77e530a-8e80-4371-840e-b2345bf68449-istio-podinfo\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:59.548705 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:59.548054    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0128f9e4-fbac-46b9-a7ec-2c19cac0d30a" path="/var/lib/kubelet/pods/0128f9e4-fbac-46b9-a7ec-2c19cac0d30a/volumes"
Oct 30 19:16:59.548705 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:59.548663    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06b7dec0-c44b-4aea-a753-53ec832dfd24" path="/var/lib/kubelet/pods/06b7dec0-c44b-4aea-a753-53ec832dfd24/volumes"
Oct 30 19:16:59.549244 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:59.549222    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="960cce33-8574-42ab-8d16-c4fb2f0b3202" path="/var/lib/kubelet/pods/960cce33-8574-42ab-8d16-c4fb2f0b3202/volumes"
Oct 30 19:16:59.550229 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:59.550205    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a3d6a70-b055-40dc-98a7-d0f5890293c8" path="/var/lib/kubelet/pods/9a3d6a70-b055-40dc-98a7-d0f5890293c8/volumes"
Oct 30 19:16:59.648060 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:59.648023    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn"
Oct 30 19:16:59.648242 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:59.648024    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn" event={"ID":"e77e530a-8e80-4371-840e-b2345bf68449","Type":"ContainerDied","Data":"00921869f6c1d6c2ecdca138d2be84ab793f54199fa37ea497455ed2a7cbe0bf"}
Oct 30 19:16:59.648242 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:59.648192    4525 scope.go:122] "RemoveContainer" containerID="d740cc14106fe5bc6d83a0eb11e4e9b6d757b4e0a0ff175aea9962d2509198e7"
Oct 30 19:16:59.697760 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:59.697722    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn"]
Oct 30 19:16:59.702804 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:16:59.702775    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ef6be7-openshift-default-bfc5fb79f-hv6qn"]
Oct 30 19:17:01.544942 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:01.544905    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e77e530a-8e80-4371-840e-b2345bf68449" path="/var/lib/kubelet/pods/e77e530a-8e80-4371-840e-b2345bf68449/volumes"
Oct 30 19:17:02.555842 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:02.555812    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-f5ct2"]
Oct 30 19:17:02.556243 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:02.556016    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-f5ct2" podUID="d9094695-97bc-4924-9526-45b9e8892662" containerName="istio-proxy" containerID="cri-o://f9abfb3593f30d2a2f3db76bff3f94bc481e32e4fa4b29f164b78571af541af2" gracePeriod=30
Oct 30 19:17:03.049010 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:03.048974    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-dcd326-p9pwl"]
Oct 30 19:17:03.049515 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:03.049495    4525 cpu_manager.go:414] "containerMap: removing container" podUID="960cce33-8574-42ab-8d16-c4fb2f0b3202" containerName="istio-proxy"
Oct 30 19:17:03.049610 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:03.049517    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="960cce33-8574-42ab-8d16-c4fb2f0b3202" containerName="istio-proxy"
Oct 30 19:17:03.049610 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:03.049533    4525 cpu_manager.go:414] "containerMap: removing container" podUID="e77e530a-8e80-4371-840e-b2345bf68449" containerName="istio-proxy"
Oct 30 19:17:03.049610 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:03.049540    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e77e530a-8e80-4371-840e-b2345bf68449" containerName="istio-proxy"
Oct 30 19:17:03.049610 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:03.049558    4525 cpu_manager.go:414] "containerMap: removing container" podUID="9a3d6a70-b055-40dc-98a7-d0f5890293c8" containerName="istio-proxy"
Oct 30 19:17:03.049610 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:03.049568    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a3d6a70-b055-40dc-98a7-d0f5890293c8" containerName="istio-proxy"
Oct 30 19:17:03.049610 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:03.049588    4525 cpu_manager.go:414] "containerMap: removing container" podUID="9f3d341b-70f0-415d-b624-e691ecc6db16" containerName="revoke-keys"
Oct 30 19:17:03.049610 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:03.049595    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f3d341b-70f0-415d-b624-e691ecc6db16" containerName="revoke-keys"
Oct 30 19:17:03.050063 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:03.049620    4525 cpu_manager.go:414] "containerMap: removing container" podUID="06b7dec0-c44b-4aea-a753-53ec832dfd24" containerName="istio-proxy"
Oct 30 19:17:03.050063 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:03.049626    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06b7dec0-c44b-4aea-a753-53ec832dfd24" containerName="istio-proxy"
Oct 30 19:17:03.050063 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:03.049770    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="e77e530a-8e80-4371-840e-b2345bf68449" containerName="istio-proxy"
Oct 30 19:17:03.050063 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:03.049786    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f3d341b-70f0-415d-b624-e691ecc6db16" containerName="revoke-keys"
Oct 30 19:17:03.050063 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:03.049798    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="84c44eb7-6483-4f83-ab59-447f04ba5d1d" containerName="istio-proxy"
Oct 30 19:17:03.050063 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:03.049810    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="06b7dec0-c44b-4aea-a753-53ec832dfd24" containerName="istio-proxy"
Oct 30 19:17:03.050063 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:03.049825    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="960cce33-8574-42ab-8d16-c4fb2f0b3202" containerName="istio-proxy"
Oct 30 19:17:03.050063 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:03.049834    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="e77e530a-8e80-4371-840e-b2345bf68449" containerName="istio-proxy"
Oct 30 19:17:03.050063 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:03.049845    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a3d6a70-b055-40dc-98a7-d0f5890293c8" containerName="istio-proxy"
Oct 30 19:17:03.054893 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:03.054869    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-dcd326-p9pwl"
Oct 30 19:17:03.066053 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:03.066022    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-dcd326-p9pwl"]
Oct 30 19:17:03.125614 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:03.125579    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:17:03.125896 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:03.125876    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94hcn\""
Oct 30 19:17:03.152455 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:03.152418    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d64f0ea3-54ce-4d6d-b2fd-43bbd7eb3b7e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-dcd326-p9pwl\" (UID: \"d64f0ea3-54ce-4d6d-b2fd-43bbd7eb3b7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-dcd326-p9pwl"
Oct 30 19:17:03.152663 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:03.152466    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d64f0ea3-54ce-4d6d-b2fd-43bbd7eb3b7e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-dcd326-p9pwl\" (UID: \"d64f0ea3-54ce-4d6d-b2fd-43bbd7eb3b7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-dcd326-p9pwl"
Oct 30 19:17:03.152663 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:03.152594    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ckh5\" (UniqueName: \"kubernetes.io/projected/d64f0ea3-54ce-4d6d-b2fd-43bbd7eb3b7e-kube-api-access-6ckh5\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-dcd326-p9pwl\" (UID: \"d64f0ea3-54ce-4d6d-b2fd-43bbd7eb3b7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-dcd326-p9pwl"
Oct 30 19:17:03.254198 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:03.254155    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d64f0ea3-54ce-4d6d-b2fd-43bbd7eb3b7e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-dcd326-p9pwl\" (UID: \"d64f0ea3-54ce-4d6d-b2fd-43bbd7eb3b7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-dcd326-p9pwl"
Oct 30 19:17:03.254390 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:03.254274    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ckh5\" (UniqueName: \"kubernetes.io/projected/d64f0ea3-54ce-4d6d-b2fd-43bbd7eb3b7e-kube-api-access-6ckh5\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-dcd326-p9pwl\" (UID: \"d64f0ea3-54ce-4d6d-b2fd-43bbd7eb3b7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-dcd326-p9pwl"
Oct 30 19:17:03.254390 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:03.254370    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d64f0ea3-54ce-4d6d-b2fd-43bbd7eb3b7e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-dcd326-p9pwl\" (UID: \"d64f0ea3-54ce-4d6d-b2fd-43bbd7eb3b7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-dcd326-p9pwl"
Oct 30 19:17:03.254980 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:03.254955    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d64f0ea3-54ce-4d6d-b2fd-43bbd7eb3b7e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-dcd326-p9pwl\" (UID: \"d64f0ea3-54ce-4d6d-b2fd-43bbd7eb3b7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-dcd326-p9pwl"
Oct 30 19:17:03.256875 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:03.256849    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d64f0ea3-54ce-4d6d-b2fd-43bbd7eb3b7e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-dcd326-p9pwl\" (UID: \"d64f0ea3-54ce-4d6d-b2fd-43bbd7eb3b7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-dcd326-p9pwl"
Oct 30 19:17:03.263045 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:03.263015    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ckh5\" (UniqueName: \"kubernetes.io/projected/d64f0ea3-54ce-4d6d-b2fd-43bbd7eb3b7e-kube-api-access-6ckh5\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-dcd326-p9pwl\" (UID: \"d64f0ea3-54ce-4d6d-b2fd-43bbd7eb3b7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-dcd326-p9pwl"
Oct 30 19:17:03.382770 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:03.382729    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-dcd326-p9pwl"
Oct 30 19:17:03.671064 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:17:03.671032    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd64f0ea3_54ce_4d6d_b2fd_43bbd7eb3b7e.slice/crio-fcbc5c935fdf83ac3755228bd56b2a09099914485f1c66992c5816dd58b0d667 WatchSource:0}: Error finding container fcbc5c935fdf83ac3755228bd56b2a09099914485f1c66992c5816dd58b0d667: Status 404 returned error can't find the container with id fcbc5c935fdf83ac3755228bd56b2a09099914485f1c66992c5816dd58b0d667
Oct 30 19:17:03.672235 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:03.672208    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-dcd326-p9pwl"]
Oct 30 19:17:03.706148 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:03.706116    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-dcd326-p9pwl" event={"ID":"d64f0ea3-54ce-4d6d-b2fd-43bbd7eb3b7e","Type":"ContainerStarted","Data":"fcbc5c935fdf83ac3755228bd56b2a09099914485f1c66992c5816dd58b0d667"}
Oct 30 19:17:04.016887 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:04.016850    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2b0c944-ctpw6"]
Oct 30 19:17:04.017335 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:04.017318    4525 cpu_manager.go:414] "containerMap: removing container" podUID="e77e530a-8e80-4371-840e-b2345bf68449" containerName="istio-proxy"
Oct 30 19:17:04.017335 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:04.017334    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e77e530a-8e80-4371-840e-b2345bf68449" containerName="istio-proxy"
Oct 30 19:17:04.020930 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:04.020911    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b0c944-ctpw6"
Oct 30 19:17:04.034542 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:04.034507    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2b0c944-ctpw6"]
Oct 30 19:17:04.171543 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:04.171489    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8kkr\" (UniqueName: \"kubernetes.io/projected/b0781052-ad77-4573-bfea-16b96fbc3c5b-kube-api-access-f8kkr\") pod \"maas-api-revoke-sub-f2b0c944-ctpw6\" (UID: \"b0781052-ad77-4573-bfea-16b96fbc3c5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b0c944-ctpw6"
Oct 30 19:17:04.171766 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:04.171732    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0781052-ad77-4573-bfea-16b96fbc3c5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2b0c944-ctpw6\" (UID: \"b0781052-ad77-4573-bfea-16b96fbc3c5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b0c944-ctpw6"
Oct 30 19:17:04.171823 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:04.171806    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0781052-ad77-4573-bfea-16b96fbc3c5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2b0c944-ctpw6\" (UID: \"b0781052-ad77-4573-bfea-16b96fbc3c5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b0c944-ctpw6"
Oct 30 19:17:04.273530 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:04.273453    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0781052-ad77-4573-bfea-16b96fbc3c5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2b0c944-ctpw6\" (UID: \"b0781052-ad77-4573-bfea-16b96fbc3c5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b0c944-ctpw6"
Oct 30 19:17:04.273530 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:04.273517    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8kkr\" (UniqueName: \"kubernetes.io/projected/b0781052-ad77-4573-bfea-16b96fbc3c5b-kube-api-access-f8kkr\") pod \"maas-api-revoke-sub-f2b0c944-ctpw6\" (UID: \"b0781052-ad77-4573-bfea-16b96fbc3c5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b0c944-ctpw6"
Oct 30 19:17:04.273883 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:04.273625    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0781052-ad77-4573-bfea-16b96fbc3c5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2b0c944-ctpw6\" (UID: \"b0781052-ad77-4573-bfea-16b96fbc3c5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b0c944-ctpw6"
Oct 30 19:17:04.274404 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:04.274380    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0781052-ad77-4573-bfea-16b96fbc3c5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2b0c944-ctpw6\" (UID: \"b0781052-ad77-4573-bfea-16b96fbc3c5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b0c944-ctpw6"
Oct 30 19:17:04.276279 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:04.276249    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0781052-ad77-4573-bfea-16b96fbc3c5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2b0c944-ctpw6\" (UID: \"b0781052-ad77-4573-bfea-16b96fbc3c5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b0c944-ctpw6"
Oct 30 19:17:04.281527 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:04.281500    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8kkr\" (UniqueName: \"kubernetes.io/projected/b0781052-ad77-4573-bfea-16b96fbc3c5b-kube-api-access-f8kkr\") pod \"maas-api-revoke-sub-f2b0c944-ctpw6\" (UID: \"b0781052-ad77-4573-bfea-16b96fbc3c5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b0c944-ctpw6"
Oct 30 19:17:04.345881 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:04.345828    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b0c944-ctpw6"
Oct 30 19:17:04.635485 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:04.635454    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2b0c944-ctpw6"]
Oct 30 19:17:04.635700 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:17:04.635663    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0781052_ad77_4573_bfea_16b96fbc3c5b.slice/crio-38bf56cd2754ecb38c341bb13b629d83542cb114933a8db420a15ee8d8725c97 WatchSource:0}: Error finding container 38bf56cd2754ecb38c341bb13b629d83542cb114933a8db420a15ee8d8725c97: Status 404 returned error can't find the container with id 38bf56cd2754ecb38c341bb13b629d83542cb114933a8db420a15ee8d8725c97
Oct 30 19:17:04.716345 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:04.716310    4525 generic.go:356] "Generic (PLEG): container finished" podID="d64f0ea3-54ce-4d6d-b2fd-43bbd7eb3b7e" containerID="0709c1e6be3296aed9cb8ad88565aa7cdb8999573e612964680013687f318ec1" exitCode=0
Oct 30 19:17:04.716822 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:04.716399    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-dcd326-p9pwl" event={"ID":"d64f0ea3-54ce-4d6d-b2fd-43bbd7eb3b7e","Type":"ContainerDied","Data":"0709c1e6be3296aed9cb8ad88565aa7cdb8999573e612964680013687f318ec1"}
Oct 30 19:17:04.718011 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:04.717979    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b0c944-ctpw6" event={"ID":"b0781052-ad77-4573-bfea-16b96fbc3c5b","Type":"ContainerStarted","Data":"38bf56cd2754ecb38c341bb13b629d83542cb114933a8db420a15ee8d8725c97"}
Oct 30 19:17:05.728326 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:05.728287    4525 generic.go:356] "Generic (PLEG): container finished" podID="b0781052-ad77-4573-bfea-16b96fbc3c5b" containerID="a6b19b9db89e6a67feefe521df0df515df7c5015eaf9fb4b287fc86251e2bb25" exitCode=0
Oct 30 19:17:05.728807 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:05.728385    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b0c944-ctpw6" event={"ID":"b0781052-ad77-4573-bfea-16b96fbc3c5b","Type":"ContainerDied","Data":"a6b19b9db89e6a67feefe521df0df515df7c5015eaf9fb4b287fc86251e2bb25"}
Oct 30 19:17:05.902432 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:05.902395    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-dcd326-p9pwl"
Oct 30 19:17:05.995071 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:05.994992    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d64f0ea3-54ce-4d6d-b2fd-43bbd7eb3b7e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d64f0ea3-54ce-4d6d-b2fd-43bbd7eb3b7e-maas-api-openshift-ca\") pod \"d64f0ea3-54ce-4d6d-b2fd-43bbd7eb3b7e\" (UID: \"d64f0ea3-54ce-4d6d-b2fd-43bbd7eb3b7e\") "
Oct 30 19:17:05.995071 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:05.995048    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d64f0ea3-54ce-4d6d-b2fd-43bbd7eb3b7e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d64f0ea3-54ce-4d6d-b2fd-43bbd7eb3b7e-maas-api-xks-ca\") pod \"d64f0ea3-54ce-4d6d-b2fd-43bbd7eb3b7e\" (UID: \"d64f0ea3-54ce-4d6d-b2fd-43bbd7eb3b7e\") "
Oct 30 19:17:05.995295 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:05.995222    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d64f0ea3-54ce-4d6d-b2fd-43bbd7eb3b7e-kube-api-access-6ckh5\" (UniqueName: \"kubernetes.io/projected/d64f0ea3-54ce-4d6d-b2fd-43bbd7eb3b7e-kube-api-access-6ckh5\") pod \"d64f0ea3-54ce-4d6d-b2fd-43bbd7eb3b7e\" (UID: \"d64f0ea3-54ce-4d6d-b2fd-43bbd7eb3b7e\") "
Oct 30 19:17:05.995361 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:05.995339    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d64f0ea3-54ce-4d6d-b2fd-43bbd7eb3b7e-maas-api-openshift-ca" pod "d64f0ea3-54ce-4d6d-b2fd-43bbd7eb3b7e" (UID: "d64f0ea3-54ce-4d6d-b2fd-43bbd7eb3b7e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:17:05.995484 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:05.995467    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d64f0ea3-54ce-4d6d-b2fd-43bbd7eb3b7e-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:05.997295 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:05.997273    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d64f0ea3-54ce-4d6d-b2fd-43bbd7eb3b7e-maas-api-xks-ca" pod "d64f0ea3-54ce-4d6d-b2fd-43bbd7eb3b7e" (UID: "d64f0ea3-54ce-4d6d-b2fd-43bbd7eb3b7e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:17:05.997518 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:05.997497    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d64f0ea3-54ce-4d6d-b2fd-43bbd7eb3b7e-kube-api-access-6ckh5" pod "d64f0ea3-54ce-4d6d-b2fd-43bbd7eb3b7e" (UID: "d64f0ea3-54ce-4d6d-b2fd-43bbd7eb3b7e"). InnerVolumeSpecName "kube-api-access-6ckh5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:06.096486 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:06.096440    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ckh5\" (UniqueName: \"kubernetes.io/projected/d64f0ea3-54ce-4d6d-b2fd-43bbd7eb3b7e-kube-api-access-6ckh5\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:06.096486 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:06.096480    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d64f0ea3-54ce-4d6d-b2fd-43bbd7eb3b7e-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:06.333602 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:06.333517    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j"]
Oct 30 19:17:06.333788 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:06.333754    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j" podUID="6061ede7-981c-4c07-acfc-04be1cae252d" containerName="istio-proxy" containerID="cri-o://c805a23922cfcabad21322c61935bbb611eb1011d79b925c872c6c66f018364b" gracePeriod=30
Oct 30 19:17:06.337593 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:06.337565    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-gtdr6"]
Oct 30 19:17:06.337806 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:06.337785    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-gtdr6" podUID="01a5c107-dcf4-4ca5-8c9d-57c40814206d" containerName="istio-proxy" containerID="cri-o://d06820e7a9f2c469f042b89ef9b369d6078f46df9a49292147b2e8e81b1c55d4" gracePeriod=30
Oct 30 19:17:06.738240 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:06.738207    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-dcd326-p9pwl"
Oct 30 19:17:06.738699 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:06.738263    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-dcd326-p9pwl" event={"ID":"d64f0ea3-54ce-4d6d-b2fd-43bbd7eb3b7e","Type":"ContainerDied","Data":"fcbc5c935fdf83ac3755228bd56b2a09099914485f1c66992c5816dd58b0d667"}
Oct 30 19:17:06.738699 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:06.738308    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fcbc5c935fdf83ac3755228bd56b2a09099914485f1c66992c5816dd58b0d667"
Oct 30 19:17:06.894951 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:06.894922    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b0c944-ctpw6"
Oct 30 19:17:07.005933 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:07.005837    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0781052-ad77-4573-bfea-16b96fbc3c5b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0781052-ad77-4573-bfea-16b96fbc3c5b-maas-api-openshift-ca\") pod \"b0781052-ad77-4573-bfea-16b96fbc3c5b\" (UID: \"b0781052-ad77-4573-bfea-16b96fbc3c5b\") "
Oct 30 19:17:07.006098 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:07.005938    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0781052-ad77-4573-bfea-16b96fbc3c5b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0781052-ad77-4573-bfea-16b96fbc3c5b-maas-api-xks-ca\") pod \"b0781052-ad77-4573-bfea-16b96fbc3c5b\" (UID: \"b0781052-ad77-4573-bfea-16b96fbc3c5b\") "
Oct 30 19:17:07.006152 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:07.006102    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0781052-ad77-4573-bfea-16b96fbc3c5b-kube-api-access-f8kkr\" (UniqueName: \"kubernetes.io/projected/b0781052-ad77-4573-bfea-16b96fbc3c5b-kube-api-access-f8kkr\") pod \"b0781052-ad77-4573-bfea-16b96fbc3c5b\" (UID: \"b0781052-ad77-4573-bfea-16b96fbc3c5b\") "
Oct 30 19:17:07.006306 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:07.006265    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0781052-ad77-4573-bfea-16b96fbc3c5b-maas-api-openshift-ca" pod "b0781052-ad77-4573-bfea-16b96fbc3c5b" (UID: "b0781052-ad77-4573-bfea-16b96fbc3c5b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:17:07.006571 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:07.006550    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0781052-ad77-4573-bfea-16b96fbc3c5b-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:07.008292 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:07.008271    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0781052-ad77-4573-bfea-16b96fbc3c5b-maas-api-xks-ca" pod "b0781052-ad77-4573-bfea-16b96fbc3c5b" (UID: "b0781052-ad77-4573-bfea-16b96fbc3c5b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:17:07.008418 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:07.008403    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0781052-ad77-4573-bfea-16b96fbc3c5b-kube-api-access-f8kkr" pod "b0781052-ad77-4573-bfea-16b96fbc3c5b" (UID: "b0781052-ad77-4573-bfea-16b96fbc3c5b"). InnerVolumeSpecName "kube-api-access-f8kkr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:07.107555 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:07.107513    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8kkr\" (UniqueName: \"kubernetes.io/projected/b0781052-ad77-4573-bfea-16b96fbc3c5b-kube-api-access-f8kkr\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:07.107555 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:07.107548    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0781052-ad77-4573-bfea-16b96fbc3c5b-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:07.751620 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:07.751581    4525 generic.go:356] "Generic (PLEG): container finished" podID="d9094695-97bc-4924-9526-45b9e8892662" containerID="f9abfb3593f30d2a2f3db76bff3f94bc481e32e4fa4b29f164b78571af541af2" exitCode=0
Oct 30 19:17:07.752078 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:07.751768    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-f5ct2" event={"ID":"d9094695-97bc-4924-9526-45b9e8892662","Type":"ContainerDied","Data":"f9abfb3593f30d2a2f3db76bff3f94bc481e32e4fa4b29f164b78571af541af2"}
Oct 30 19:17:07.752078 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:07.751826    4525 scope.go:122] "RemoveContainer" containerID="e2d446f40cf1dd6c12c1badf1f879711f7cb234dd8dca27e13a442572a339ac4"
Oct 30 19:17:07.754771 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:07.754740    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b0c944-ctpw6" event={"ID":"b0781052-ad77-4573-bfea-16b96fbc3c5b","Type":"ContainerDied","Data":"38bf56cd2754ecb38c341bb13b629d83542cb114933a8db420a15ee8d8725c97"}
Oct 30 19:17:07.754898 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:07.754785    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b0c944-ctpw6"
Oct 30 19:17:07.754898 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:07.754795    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="38bf56cd2754ecb38c341bb13b629d83542cb114933a8db420a15ee8d8725c97"
Oct 30 19:17:07.820265 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:07.820197    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-f5ct2"
Oct 30 19:17:07.916470 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:07.916435    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9094695-97bc-4924-9526-45b9e8892662-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9094695-97bc-4924-9526-45b9e8892662-istio-data\") pod \"d9094695-97bc-4924-9526-45b9e8892662\" (UID: \"d9094695-97bc-4924-9526-45b9e8892662\") "
Oct 30 19:17:07.916666 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:07.916541    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9094695-97bc-4924-9526-45b9e8892662-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9094695-97bc-4924-9526-45b9e8892662-istio-envoy\") pod \"d9094695-97bc-4924-9526-45b9e8892662\" (UID: \"d9094695-97bc-4924-9526-45b9e8892662\") "
Oct 30 19:17:07.916666 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:07.916570    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9094695-97bc-4924-9526-45b9e8892662-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9094695-97bc-4924-9526-45b9e8892662-istiod-ca-cert\") pod \"d9094695-97bc-4924-9526-45b9e8892662\" (UID: \"d9094695-97bc-4924-9526-45b9e8892662\") "
Oct 30 19:17:07.916666 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:07.916608    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9094695-97bc-4924-9526-45b9e8892662-kube-api-access-gjxkq\" (UniqueName: \"kubernetes.io/projected/d9094695-97bc-4924-9526-45b9e8892662-kube-api-access-gjxkq\") pod \"d9094695-97bc-4924-9526-45b9e8892662\" (UID: \"d9094695-97bc-4924-9526-45b9e8892662\") "
Oct 30 19:17:07.916666 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:07.916641    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9094695-97bc-4924-9526-45b9e8892662-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9094695-97bc-4924-9526-45b9e8892662-workload-socket\") pod \"d9094695-97bc-4924-9526-45b9e8892662\" (UID: \"d9094695-97bc-4924-9526-45b9e8892662\") "
Oct 30 19:17:07.916935 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:07.916782    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9094695-97bc-4924-9526-45b9e8892662-istio-data" pod "d9094695-97bc-4924-9526-45b9e8892662" (UID: "d9094695-97bc-4924-9526-45b9e8892662"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:07.916935 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:07.916828    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9094695-97bc-4924-9526-45b9e8892662-istio-token\" (UniqueName: \"kubernetes.io/projected/d9094695-97bc-4924-9526-45b9e8892662-istio-token\") pod \"d9094695-97bc-4924-9526-45b9e8892662\" (UID: \"d9094695-97bc-4924-9526-45b9e8892662\") "
Oct 30 19:17:07.917041 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:07.916935    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d9094695-97bc-4924-9526-45b9e8892662-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9094695-97bc-4924-9526-45b9e8892662-istio-podinfo\") pod \"d9094695-97bc-4924-9526-45b9e8892662\" (UID: \"d9094695-97bc-4924-9526-45b9e8892662\") "
Oct 30 19:17:07.917041 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:07.917005    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9094695-97bc-4924-9526-45b9e8892662-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9094695-97bc-4924-9526-45b9e8892662-workload-certs\") pod \"d9094695-97bc-4924-9526-45b9e8892662\" (UID: \"d9094695-97bc-4924-9526-45b9e8892662\") "
Oct 30 19:17:07.917150 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:07.917075    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9094695-97bc-4924-9526-45b9e8892662-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9094695-97bc-4924-9526-45b9e8892662-credential-socket\") pod \"d9094695-97bc-4924-9526-45b9e8892662\" (UID: \"d9094695-97bc-4924-9526-45b9e8892662\") "
Oct 30 19:17:07.917150 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:07.917079    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9094695-97bc-4924-9526-45b9e8892662-workload-socket" pod "d9094695-97bc-4924-9526-45b9e8892662" (UID: "d9094695-97bc-4924-9526-45b9e8892662"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:07.917266 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:07.917171    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9094695-97bc-4924-9526-45b9e8892662-istiod-ca-cert" pod "d9094695-97bc-4924-9526-45b9e8892662" (UID: "d9094695-97bc-4924-9526-45b9e8892662"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:17:07.917468 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:07.917443    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9094695-97bc-4924-9526-45b9e8892662-credential-socket" pod "d9094695-97bc-4924-9526-45b9e8892662" (UID: "d9094695-97bc-4924-9526-45b9e8892662"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:07.917560 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:07.917463    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9094695-97bc-4924-9526-45b9e8892662-workload-certs" pod "d9094695-97bc-4924-9526-45b9e8892662" (UID: "d9094695-97bc-4924-9526-45b9e8892662"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:07.917560 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:07.917551    4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9094695-97bc-4924-9526-45b9e8892662-istio-data\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:07.917707 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:07.917572    4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9094695-97bc-4924-9526-45b9e8892662-istiod-ca-cert\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:07.917707 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:07.917588    4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9094695-97bc-4924-9526-45b9e8892662-workload-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:07.917707 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:07.917601    4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9094695-97bc-4924-9526-45b9e8892662-workload-certs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:07.917707 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:07.917615    4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9094695-97bc-4924-9526-45b9e8892662-credential-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:07.919451 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:07.919422    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d9094695-97bc-4924-9526-45b9e8892662-istio-podinfo" pod "d9094695-97bc-4924-9526-45b9e8892662" (UID: "d9094695-97bc-4924-9526-45b9e8892662"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:17:07.919753 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:07.919731    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9094695-97bc-4924-9526-45b9e8892662-istio-token" pod "d9094695-97bc-4924-9526-45b9e8892662" (UID: "d9094695-97bc-4924-9526-45b9e8892662"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:07.920012 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:07.919993    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9094695-97bc-4924-9526-45b9e8892662-istio-envoy" pod "d9094695-97bc-4924-9526-45b9e8892662" (UID: "d9094695-97bc-4924-9526-45b9e8892662"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:07.920078 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:07.920024    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9094695-97bc-4924-9526-45b9e8892662-kube-api-access-gjxkq" pod "d9094695-97bc-4924-9526-45b9e8892662" (UID: "d9094695-97bc-4924-9526-45b9e8892662"). InnerVolumeSpecName "kube-api-access-gjxkq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:08.019180 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:08.019137    4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9094695-97bc-4924-9526-45b9e8892662-istio-envoy\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:08.019180 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:08.019175    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjxkq\" (UniqueName: \"kubernetes.io/projected/d9094695-97bc-4924-9526-45b9e8892662-kube-api-access-gjxkq\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:08.019180 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:08.019187    4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9094695-97bc-4924-9526-45b9e8892662-istio-token\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:08.019406 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:08.019196    4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9094695-97bc-4924-9526-45b9e8892662-istio-podinfo\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:08.764141 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:08.764102    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-f5ct2"
Oct 30 19:17:08.764592 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:08.764138    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-f5ct2" event={"ID":"d9094695-97bc-4924-9526-45b9e8892662","Type":"ContainerDied","Data":"94f38ffacf702d2e801acf0d1dc31eb30c924937b8f973785e6b9202554eb8da"}
Oct 30 19:17:08.764592 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:08.764205    4525 scope.go:122] "RemoveContainer" containerID="f9abfb3593f30d2a2f3db76bff3f94bc481e32e4fa4b29f164b78571af541af2"
Oct 30 19:17:08.789195 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:08.789167    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-f5ct2"]
Oct 30 19:17:08.794854 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:08.794827    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-f3d6a5-openshift-default-79b6db5bf7-f5ct2"]
Oct 30 19:17:09.545160 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:09.545126    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9094695-97bc-4924-9526-45b9e8892662" path="/var/lib/kubelet/pods/d9094695-97bc-4924-9526-45b9e8892662/volumes"
Oct 30 19:17:11.026960 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:11.026922    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e93a3449-tnzzz"]
Oct 30 19:17:11.030501 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:11.030480    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e93a3449-tnzzz"]
Oct 30 19:17:11.566544 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:11.565818    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc053426-d0cc-4c1e-928a-5b7624ae02bb" path="/var/lib/kubelet/pods/bc053426-d0cc-4c1e-928a-5b7624ae02bb/volumes"
Oct 30 19:17:11.619323 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:11.619293    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j"
Oct 30 19:17:11.679599 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:11.679560    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-gtdr6"
Oct 30 19:17:11.766921 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:11.766882    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6061ede7-981c-4c07-acfc-04be1cae252d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6061ede7-981c-4c07-acfc-04be1cae252d-istio-envoy\") pod \"6061ede7-981c-4c07-acfc-04be1cae252d\" (UID: \"6061ede7-981c-4c07-acfc-04be1cae252d\") "
Oct 30 19:17:11.767103 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:11.766967    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6061ede7-981c-4c07-acfc-04be1cae252d-kube-api-access-s2sfg\" (UniqueName: \"kubernetes.io/projected/6061ede7-981c-4c07-acfc-04be1cae252d-kube-api-access-s2sfg\") pod \"6061ede7-981c-4c07-acfc-04be1cae252d\" (UID: \"6061ede7-981c-4c07-acfc-04be1cae252d\") "
Oct 30 19:17:11.767103 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:11.767003    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6061ede7-981c-4c07-acfc-04be1cae252d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6061ede7-981c-4c07-acfc-04be1cae252d-istio-podinfo\") pod \"6061ede7-981c-4c07-acfc-04be1cae252d\" (UID: \"6061ede7-981c-4c07-acfc-04be1cae252d\") "
Oct 30 19:17:11.767190 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:11.767151    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6061ede7-981c-4c07-acfc-04be1cae252d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6061ede7-981c-4c07-acfc-04be1cae252d-workload-certs\") pod \"6061ede7-981c-4c07-acfc-04be1cae252d\" (UID: \"6061ede7-981c-4c07-acfc-04be1cae252d\") "
Oct 30 19:17:11.767250 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:11.767233    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6061ede7-981c-4c07-acfc-04be1cae252d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6061ede7-981c-4c07-acfc-04be1cae252d-workload-socket\") pod \"6061ede7-981c-4c07-acfc-04be1cae252d\" (UID: \"6061ede7-981c-4c07-acfc-04be1cae252d\") "
Oct 30 19:17:11.767308 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:11.767269    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6061ede7-981c-4c07-acfc-04be1cae252d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6061ede7-981c-4c07-acfc-04be1cae252d-istiod-ca-cert\") pod \"6061ede7-981c-4c07-acfc-04be1cae252d\" (UID: \"6061ede7-981c-4c07-acfc-04be1cae252d\") "
Oct 30 19:17:11.767400 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:11.767370    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6061ede7-981c-4c07-acfc-04be1cae252d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6061ede7-981c-4c07-acfc-04be1cae252d-istio-data\") pod \"6061ede7-981c-4c07-acfc-04be1cae252d\" (UID: \"6061ede7-981c-4c07-acfc-04be1cae252d\") "
Oct 30 19:17:11.767469 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:11.767410    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6061ede7-981c-4c07-acfc-04be1cae252d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6061ede7-981c-4c07-acfc-04be1cae252d-credential-socket\") pod \"6061ede7-981c-4c07-acfc-04be1cae252d\" (UID: \"6061ede7-981c-4c07-acfc-04be1cae252d\") "
Oct 30 19:17:11.767534 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:11.767484    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6061ede7-981c-4c07-acfc-04be1cae252d-istio-token\" (UniqueName: \"kubernetes.io/projected/6061ede7-981c-4c07-acfc-04be1cae252d-istio-token\") pod \"6061ede7-981c-4c07-acfc-04be1cae252d\" (UID: \"6061ede7-981c-4c07-acfc-04be1cae252d\") "
Oct 30 19:17:11.767591 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:11.767538    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6061ede7-981c-4c07-acfc-04be1cae252d-workload-socket" pod "6061ede7-981c-4c07-acfc-04be1cae252d" (UID: "6061ede7-981c-4c07-acfc-04be1cae252d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:11.767773 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:11.767672    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6061ede7-981c-4c07-acfc-04be1cae252d-istio-data" pod "6061ede7-981c-4c07-acfc-04be1cae252d" (UID: "6061ede7-981c-4c07-acfc-04be1cae252d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:11.768108 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:11.767844    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6061ede7-981c-4c07-acfc-04be1cae252d-credential-socket" pod "6061ede7-981c-4c07-acfc-04be1cae252d" (UID: "6061ede7-981c-4c07-acfc-04be1cae252d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:11.768108 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:11.768013    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6061ede7-981c-4c07-acfc-04be1cae252d-istiod-ca-cert" pod "6061ede7-981c-4c07-acfc-04be1cae252d" (UID: "6061ede7-981c-4c07-acfc-04be1cae252d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:17:11.768108 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:11.768080    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6061ede7-981c-4c07-acfc-04be1cae252d-workload-certs" pod "6061ede7-981c-4c07-acfc-04be1cae252d" (UID: "6061ede7-981c-4c07-acfc-04be1cae252d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:11.768327 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:11.768184    4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6061ede7-981c-4c07-acfc-04be1cae252d-workload-certs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:11.768327 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:11.768200    4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6061ede7-981c-4c07-acfc-04be1cae252d-workload-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:11.768327 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:11.768215    4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6061ede7-981c-4c07-acfc-04be1cae252d-istiod-ca-cert\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:11.768327 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:11.768225    4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6061ede7-981c-4c07-acfc-04be1cae252d-istio-data\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:11.768327 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:11.768237    4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6061ede7-981c-4c07-acfc-04be1cae252d-credential-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:11.769490 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:11.769465    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6061ede7-981c-4c07-acfc-04be1cae252d-istio-podinfo" pod "6061ede7-981c-4c07-acfc-04be1cae252d" (UID: "6061ede7-981c-4c07-acfc-04be1cae252d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:17:11.769602 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:11.769563    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6061ede7-981c-4c07-acfc-04be1cae252d-kube-api-access-s2sfg" pod "6061ede7-981c-4c07-acfc-04be1cae252d" (UID: "6061ede7-981c-4c07-acfc-04be1cae252d"). InnerVolumeSpecName "kube-api-access-s2sfg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:11.769903 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:11.769877    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6061ede7-981c-4c07-acfc-04be1cae252d-istio-envoy" pod "6061ede7-981c-4c07-acfc-04be1cae252d" (UID: "6061ede7-981c-4c07-acfc-04be1cae252d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:11.770046 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:11.770031    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6061ede7-981c-4c07-acfc-04be1cae252d-istio-token" pod "6061ede7-981c-4c07-acfc-04be1cae252d" (UID: "6061ede7-981c-4c07-acfc-04be1cae252d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:11.791549 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:11.791518    4525 generic.go:356] "Generic (PLEG): container finished" podID="01a5c107-dcf4-4ca5-8c9d-57c40814206d" containerID="d06820e7a9f2c469f042b89ef9b369d6078f46df9a49292147b2e8e81b1c55d4" exitCode=0
Oct 30 19:17:11.791753 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:11.791570    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-gtdr6"
Oct 30 19:17:11.791753 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:11.791612    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-gtdr6" event={"ID":"01a5c107-dcf4-4ca5-8c9d-57c40814206d","Type":"ContainerDied","Data":"d06820e7a9f2c469f042b89ef9b369d6078f46df9a49292147b2e8e81b1c55d4"}
Oct 30 19:17:11.791753 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:11.791666    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-gtdr6" event={"ID":"01a5c107-dcf4-4ca5-8c9d-57c40814206d","Type":"ContainerDied","Data":"55649931f8faa01eae62c9cefcf93ab82bf1b4b3fe384727c91f82c3219b771f"}
Oct 30 19:17:11.791753 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:11.791710    4525 scope.go:122] "RemoveContainer" containerID="d06820e7a9f2c469f042b89ef9b369d6078f46df9a49292147b2e8e81b1c55d4"
Oct 30 19:17:11.793790 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:11.793770    4525 generic.go:356] "Generic (PLEG): container finished" podID="6061ede7-981c-4c07-acfc-04be1cae252d" containerID="c805a23922cfcabad21322c61935bbb611eb1011d79b925c872c6c66f018364b" exitCode=0
Oct 30 19:17:11.793887 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:11.793818    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j"
Oct 30 19:17:11.793887 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:11.793820    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j" event={"ID":"6061ede7-981c-4c07-acfc-04be1cae252d","Type":"ContainerDied","Data":"c805a23922cfcabad21322c61935bbb611eb1011d79b925c872c6c66f018364b"}
Oct 30 19:17:11.793977 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:11.793898    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j" event={"ID":"6061ede7-981c-4c07-acfc-04be1cae252d","Type":"ContainerDied","Data":"a65c38e22d7c9cfe58089211935c7902a4414e3bfeb160f389b4042d9c3cc9cd"}
Oct 30 19:17:11.804016 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:11.803996    4525 scope.go:122] "RemoveContainer" containerID="d06820e7a9f2c469f042b89ef9b369d6078f46df9a49292147b2e8e81b1c55d4"
Oct 30 19:17:11.804305 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:17:11.804287    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d06820e7a9f2c469f042b89ef9b369d6078f46df9a49292147b2e8e81b1c55d4\": container with ID starting with d06820e7a9f2c469f042b89ef9b369d6078f46df9a49292147b2e8e81b1c55d4 not found: ID does not exist" containerID="d06820e7a9f2c469f042b89ef9b369d6078f46df9a49292147b2e8e81b1c55d4"
Oct 30 19:17:11.804371 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:11.804315    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d06820e7a9f2c469f042b89ef9b369d6078f46df9a49292147b2e8e81b1c55d4"} err="failed to get container status \"d06820e7a9f2c469f042b89ef9b369d6078f46df9a49292147b2e8e81b1c55d4\": rpc error: code = NotFound desc = could not find container \"d06820e7a9f2c469f042b89ef9b369d6078f46df9a49292147b2e8e81b1c55d4\": container with ID starting with d06820e7a9f2c469f042b89ef9b369d6078f46df9a49292147b2e8e81b1c55d4 not found: ID does not exist"
Oct 30 19:17:11.804371 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:11.804333    4525 scope.go:122] "RemoveContainer" containerID="c805a23922cfcabad21322c61935bbb611eb1011d79b925c872c6c66f018364b"
Oct 30 19:17:11.814809 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:11.814787    4525 scope.go:122] "RemoveContainer" containerID="95be28c0a2e6e7efa2daa699bd3d87456912a01b2d5df468ae82634e02d7a3b7"
Oct 30 19:17:11.825040 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:11.825016    4525 scope.go:122] "RemoveContainer" containerID="c805a23922cfcabad21322c61935bbb611eb1011d79b925c872c6c66f018364b"
Oct 30 19:17:11.825335 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:17:11.825314    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c805a23922cfcabad21322c61935bbb611eb1011d79b925c872c6c66f018364b\": container with ID starting with c805a23922cfcabad21322c61935bbb611eb1011d79b925c872c6c66f018364b not found: ID does not exist" containerID="c805a23922cfcabad21322c61935bbb611eb1011d79b925c872c6c66f018364b"
Oct 30 19:17:11.825409 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:11.825342    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c805a23922cfcabad21322c61935bbb611eb1011d79b925c872c6c66f018364b"} err="failed to get container status \"c805a23922cfcabad21322c61935bbb611eb1011d79b925c872c6c66f018364b\": rpc error: code = NotFound desc = could not find container \"c805a23922cfcabad21322c61935bbb611eb1011d79b925c872c6c66f018364b\": container with ID starting with c805a23922cfcabad21322c61935bbb611eb1011d79b925c872c6c66f018364b not found: ID does not exist"
Oct 30 19:17:11.825409 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:11.825358    4525 scope.go:122] "RemoveContainer" containerID="95be28c0a2e6e7efa2daa699bd3d87456912a01b2d5df468ae82634e02d7a3b7"
Oct 30 19:17:11.825628 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:17:11.825609    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95be28c0a2e6e7efa2daa699bd3d87456912a01b2d5df468ae82634e02d7a3b7\": container with ID starting with 95be28c0a2e6e7efa2daa699bd3d87456912a01b2d5df468ae82634e02d7a3b7 not found: ID does not exist" containerID="95be28c0a2e6e7efa2daa699bd3d87456912a01b2d5df468ae82634e02d7a3b7"
Oct 30 19:17:11.825668 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:11.825639    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95be28c0a2e6e7efa2daa699bd3d87456912a01b2d5df468ae82634e02d7a3b7"} err="failed to get container status \"95be28c0a2e6e7efa2daa699bd3d87456912a01b2d5df468ae82634e02d7a3b7\": rpc error: code = NotFound desc = could not find container \"95be28c0a2e6e7efa2daa699bd3d87456912a01b2d5df468ae82634e02d7a3b7\": container with ID starting with 95be28c0a2e6e7efa2daa699bd3d87456912a01b2d5df468ae82634e02d7a3b7 not found: ID does not exist"
Oct 30 19:17:11.840085 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:11.840060    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j"]
Oct 30 19:17:11.854543 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:11.854516    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-h627j"]
Oct 30 19:17:11.868873 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:11.868807    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01a5c107-dcf4-4ca5-8c9d-57c40814206d-istio-token\" (UniqueName: \"kubernetes.io/projected/01a5c107-dcf4-4ca5-8c9d-57c40814206d-istio-token\") pod \"01a5c107-dcf4-4ca5-8c9d-57c40814206d\" (UID: \"01a5c107-dcf4-4ca5-8c9d-57c40814206d\") "
Oct 30 19:17:11.868873 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:11.868848    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01a5c107-dcf4-4ca5-8c9d-57c40814206d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a5c107-dcf4-4ca5-8c9d-57c40814206d-workload-socket\") pod \"01a5c107-dcf4-4ca5-8c9d-57c40814206d\" (UID: \"01a5c107-dcf4-4ca5-8c9d-57c40814206d\") "
Oct 30 19:17:11.869023 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:11.868873    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01a5c107-dcf4-4ca5-8c9d-57c40814206d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01a5c107-dcf4-4ca5-8c9d-57c40814206d-istio-data\") pod \"01a5c107-dcf4-4ca5-8c9d-57c40814206d\" (UID: \"01a5c107-dcf4-4ca5-8c9d-57c40814206d\") "
Oct 30 19:17:11.869023 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:11.868909    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01a5c107-dcf4-4ca5-8c9d-57c40814206d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01a5c107-dcf4-4ca5-8c9d-57c40814206d-istio-envoy\") pod \"01a5c107-dcf4-4ca5-8c9d-57c40814206d\" (UID: \"01a5c107-dcf4-4ca5-8c9d-57c40814206d\") "
Oct 30 19:17:11.869023 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:11.868967    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01a5c107-dcf4-4ca5-8c9d-57c40814206d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01a5c107-dcf4-4ca5-8c9d-57c40814206d-workload-certs\") pod \"01a5c107-dcf4-4ca5-8c9d-57c40814206d\" (UID: \"01a5c107-dcf4-4ca5-8c9d-57c40814206d\") "
Oct 30 19:17:11.869157 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:11.869105    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01a5c107-dcf4-4ca5-8c9d-57c40814206d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a5c107-dcf4-4ca5-8c9d-57c40814206d-credential-socket\") pod \"01a5c107-dcf4-4ca5-8c9d-57c40814206d\" (UID: \"01a5c107-dcf4-4ca5-8c9d-57c40814206d\") "
Oct 30 19:17:11.869217 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:11.869205    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/01a5c107-dcf4-4ca5-8c9d-57c40814206d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01a5c107-dcf4-4ca5-8c9d-57c40814206d-istio-podinfo\") pod \"01a5c107-dcf4-4ca5-8c9d-57c40814206d\" (UID: \"01a5c107-dcf4-4ca5-8c9d-57c40814206d\") "
Oct 30 19:17:11.869290 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:11.869263    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01a5c107-dcf4-4ca5-8c9d-57c40814206d-workload-socket" pod "01a5c107-dcf4-4ca5-8c9d-57c40814206d" (UID: "01a5c107-dcf4-4ca5-8c9d-57c40814206d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:11.869354 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:11.869283    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01a5c107-dcf4-4ca5-8c9d-57c40814206d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01a5c107-dcf4-4ca5-8c9d-57c40814206d-istiod-ca-cert\") pod \"01a5c107-dcf4-4ca5-8c9d-57c40814206d\" (UID: \"01a5c107-dcf4-4ca5-8c9d-57c40814206d\") "
Oct 30 19:17:11.869354 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:11.869310    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01a5c107-dcf4-4ca5-8c9d-57c40814206d-credential-socket" pod "01a5c107-dcf4-4ca5-8c9d-57c40814206d" (UID: "01a5c107-dcf4-4ca5-8c9d-57c40814206d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:11.869354 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:11.869289    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01a5c107-dcf4-4ca5-8c9d-57c40814206d-istio-data" pod "01a5c107-dcf4-4ca5-8c9d-57c40814206d" (UID: "01a5c107-dcf4-4ca5-8c9d-57c40814206d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:11.869354 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:11.869328    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01a5c107-dcf4-4ca5-8c9d-57c40814206d-workload-certs" pod "01a5c107-dcf4-4ca5-8c9d-57c40814206d" (UID: "01a5c107-dcf4-4ca5-8c9d-57c40814206d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:11.869354 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:11.869319    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01a5c107-dcf4-4ca5-8c9d-57c40814206d-kube-api-access-trs94\" (UniqueName: \"kubernetes.io/projected/01a5c107-dcf4-4ca5-8c9d-57c40814206d-kube-api-access-trs94\") pod \"01a5c107-dcf4-4ca5-8c9d-57c40814206d\" (UID: \"01a5c107-dcf4-4ca5-8c9d-57c40814206d\") "
Oct 30 19:17:11.869639 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:11.869601    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01a5c107-dcf4-4ca5-8c9d-57c40814206d-istiod-ca-cert" pod "01a5c107-dcf4-4ca5-8c9d-57c40814206d" (UID: "01a5c107-dcf4-4ca5-8c9d-57c40814206d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:17:11.869804 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:11.869784    4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6061ede7-981c-4c07-acfc-04be1cae252d-istio-token\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:11.869804 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:11.869803    4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a5c107-dcf4-4ca5-8c9d-57c40814206d-workload-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:11.870158 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:11.869813    4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01a5c107-dcf4-4ca5-8c9d-57c40814206d-istio-data\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:11.870158 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:11.869827    4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6061ede7-981c-4c07-acfc-04be1cae252d-istio-envoy\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:11.870158 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:11.869842    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2sfg\" (UniqueName: \"kubernetes.io/projected/6061ede7-981c-4c07-acfc-04be1cae252d-kube-api-access-s2sfg\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:11.870158 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:11.869857    4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01a5c107-dcf4-4ca5-8c9d-57c40814206d-workload-certs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:11.870158 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:11.869866    4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6061ede7-981c-4c07-acfc-04be1cae252d-istio-podinfo\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:11.870158 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:11.869874    4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a5c107-dcf4-4ca5-8c9d-57c40814206d-credential-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:11.870158 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:11.869884    4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01a5c107-dcf4-4ca5-8c9d-57c40814206d-istiod-ca-cert\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:11.871524 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:11.871503    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01a5c107-dcf4-4ca5-8c9d-57c40814206d-kube-api-access-trs94" pod "01a5c107-dcf4-4ca5-8c9d-57c40814206d" (UID: "01a5c107-dcf4-4ca5-8c9d-57c40814206d"). InnerVolumeSpecName "kube-api-access-trs94". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:11.871609 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:11.871595    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/01a5c107-dcf4-4ca5-8c9d-57c40814206d-istio-podinfo" pod "01a5c107-dcf4-4ca5-8c9d-57c40814206d" (UID: "01a5c107-dcf4-4ca5-8c9d-57c40814206d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:17:11.871804 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:11.871778    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01a5c107-dcf4-4ca5-8c9d-57c40814206d-istio-envoy" pod "01a5c107-dcf4-4ca5-8c9d-57c40814206d" (UID: "01a5c107-dcf4-4ca5-8c9d-57c40814206d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:11.871804 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:11.871780    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01a5c107-dcf4-4ca5-8c9d-57c40814206d-istio-token" pod "01a5c107-dcf4-4ca5-8c9d-57c40814206d" (UID: "01a5c107-dcf4-4ca5-8c9d-57c40814206d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:11.971157 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:11.971118    4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01a5c107-dcf4-4ca5-8c9d-57c40814206d-istio-token\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:11.971157 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:11.971148    4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01a5c107-dcf4-4ca5-8c9d-57c40814206d-istio-envoy\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:11.971157 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:11.971158    4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01a5c107-dcf4-4ca5-8c9d-57c40814206d-istio-podinfo\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:11.971406 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:11.971169    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trs94\" (UniqueName: \"kubernetes.io/projected/01a5c107-dcf4-4ca5-8c9d-57c40814206d-kube-api-access-trs94\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:12.119446 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:12.119356    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-gtdr6"]
Oct 30 19:17:12.124527 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:12.124503    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-2889dd-openshift-default-84768d69f7-gtdr6"]
Oct 30 19:17:13.420640 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:13.420594    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr" podUID="82bbbf85-ed93-48d2-9f17-49ac1f31c0a8" containerName="main" containerID="cri-o://eb4eecd9e9dc3d03699dafc4fea9e08845b6c8c2814a546b34d8b9de9d79e894" gracePeriod=45
Oct 30 19:17:13.433311 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:13.433267    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw" podUID="ac6852fd-01be-4d69-8c9b-0c591e50b889" containerName="main" containerID="cri-o://9cb591ec249cb9cc35219afe413c4bec64533b92f0bd53f5dae1c45acb0101f3" gracePeriod=45
Oct 30 19:17:13.543023 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:13.542226    4525 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw" podUID="ac6852fd-01be-4d69-8c9b-0c591e50b889" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": dial tcp 10.133.0.65:8000: connect: connection refused"
Oct 30 19:17:13.547006 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:13.546974    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01a5c107-dcf4-4ca5-8c9d-57c40814206d" path="/var/lib/kubelet/pods/01a5c107-dcf4-4ca5-8c9d-57c40814206d/volumes"
Oct 30 19:17:13.547444 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:13.547428    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6061ede7-981c-4c07-acfc-04be1cae252d" path="/var/lib/kubelet/pods/6061ede7-981c-4c07-acfc-04be1cae252d/volumes"
Oct 30 19:17:13.917759 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:13.917719    4525 generic.go:356] "Generic (PLEG): container finished" podID="ac6852fd-01be-4d69-8c9b-0c591e50b889" containerID="9cb591ec249cb9cc35219afe413c4bec64533b92f0bd53f5dae1c45acb0101f3" exitCode=0
Oct 30 19:17:13.918112 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:13.918086    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw" event={"ID":"ac6852fd-01be-4d69-8c9b-0c591e50b889","Type":"ContainerDied","Data":"9cb591ec249cb9cc35219afe413c4bec64533b92f0bd53f5dae1c45acb0101f3"}
Oct 30 19:17:13.962165 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:13.962132    4525 generic.go:356] "Generic (PLEG): container finished" podID="82bbbf85-ed93-48d2-9f17-49ac1f31c0a8" containerID="eb4eecd9e9dc3d03699dafc4fea9e08845b6c8c2814a546b34d8b9de9d79e894" exitCode=0
Oct 30 19:17:13.962440 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:13.962208    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr" event={"ID":"82bbbf85-ed93-48d2-9f17-49ac1f31c0a8","Type":"ContainerDied","Data":"eb4eecd9e9dc3d03699dafc4fea9e08845b6c8c2814a546b34d8b9de9d79e894"}
Oct 30 19:17:14.033646 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:14.033613    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw"
Oct 30 19:17:14.047430 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:14.047393    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr"
Oct 30 19:17:14.120704 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:14.117673    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac6852fd-01be-4d69-8c9b-0c591e50b889-home\" (UniqueName: \"kubernetes.io/empty-dir/ac6852fd-01be-4d69-8c9b-0c591e50b889-home\") pod \"ac6852fd-01be-4d69-8c9b-0c591e50b889\" (UID: \"ac6852fd-01be-4d69-8c9b-0c591e50b889\") "
Oct 30 19:17:14.120704 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:14.117789    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82bbbf85-ed93-48d2-9f17-49ac1f31c0a8-tls-certs\" (UniqueName: \"kubernetes.io/secret/82bbbf85-ed93-48d2-9f17-49ac1f31c0a8-tls-certs\") pod \"82bbbf85-ed93-48d2-9f17-49ac1f31c0a8\" (UID: \"82bbbf85-ed93-48d2-9f17-49ac1f31c0a8\") "
Oct 30 19:17:14.120704 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:14.117814    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac6852fd-01be-4d69-8c9b-0c591e50b889-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac6852fd-01be-4d69-8c9b-0c591e50b889-tmp-dir\") pod \"ac6852fd-01be-4d69-8c9b-0c591e50b889\" (UID: \"ac6852fd-01be-4d69-8c9b-0c591e50b889\") "
Oct 30 19:17:14.120704 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:14.117866    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac6852fd-01be-4d69-8c9b-0c591e50b889-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac6852fd-01be-4d69-8c9b-0c591e50b889-model-cache\") pod \"ac6852fd-01be-4d69-8c9b-0c591e50b889\" (UID: \"ac6852fd-01be-4d69-8c9b-0c591e50b889\") "
Oct 30 19:17:14.120704 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:14.117898    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82bbbf85-ed93-48d2-9f17-49ac1f31c0a8-kube-api-access-6rf92\" (UniqueName: \"kubernetes.io/projected/82bbbf85-ed93-48d2-9f17-49ac1f31c0a8-kube-api-access-6rf92\") pod \"82bbbf85-ed93-48d2-9f17-49ac1f31c0a8\" (UID: \"82bbbf85-ed93-48d2-9f17-49ac1f31c0a8\") "
Oct 30 19:17:14.120704 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:14.117927    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82bbbf85-ed93-48d2-9f17-49ac1f31c0a8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82bbbf85-ed93-48d2-9f17-49ac1f31c0a8-model-cache\") pod \"82bbbf85-ed93-48d2-9f17-49ac1f31c0a8\" (UID: \"82bbbf85-ed93-48d2-9f17-49ac1f31c0a8\") "
Oct 30 19:17:14.120704 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:14.117955    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac6852fd-01be-4d69-8c9b-0c591e50b889-tls-certs\" (UniqueName: \"kubernetes.io/secret/ac6852fd-01be-4d69-8c9b-0c591e50b889-tls-certs\") pod \"ac6852fd-01be-4d69-8c9b-0c591e50b889\" (UID: \"ac6852fd-01be-4d69-8c9b-0c591e50b889\") "
Oct 30 19:17:14.120704 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:14.117969    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82bbbf85-ed93-48d2-9f17-49ac1f31c0a8-home\" (UniqueName: \"kubernetes.io/empty-dir/82bbbf85-ed93-48d2-9f17-49ac1f31c0a8-home\") pod \"82bbbf85-ed93-48d2-9f17-49ac1f31c0a8\" (UID: \"82bbbf85-ed93-48d2-9f17-49ac1f31c0a8\") "
Oct 30 19:17:14.120704 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:14.118013    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac6852fd-01be-4d69-8c9b-0c591e50b889-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac6852fd-01be-4d69-8c9b-0c591e50b889-dshm\") pod \"ac6852fd-01be-4d69-8c9b-0c591e50b889\" (UID: \"ac6852fd-01be-4d69-8c9b-0c591e50b889\") "
Oct 30 19:17:14.120704 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:14.118037    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82bbbf85-ed93-48d2-9f17-49ac1f31c0a8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/82bbbf85-ed93-48d2-9f17-49ac1f31c0a8-dshm\") pod \"82bbbf85-ed93-48d2-9f17-49ac1f31c0a8\" (UID: \"82bbbf85-ed93-48d2-9f17-49ac1f31c0a8\") "
Oct 30 19:17:14.120704 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:14.118110    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82bbbf85-ed93-48d2-9f17-49ac1f31c0a8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82bbbf85-ed93-48d2-9f17-49ac1f31c0a8-tmp-dir\") pod \"82bbbf85-ed93-48d2-9f17-49ac1f31c0a8\" (UID: \"82bbbf85-ed93-48d2-9f17-49ac1f31c0a8\") "
Oct 30 19:17:14.120704 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:14.118136    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac6852fd-01be-4d69-8c9b-0c591e50b889-kube-api-access-csn4h\" (UniqueName: \"kubernetes.io/projected/ac6852fd-01be-4d69-8c9b-0c591e50b889-kube-api-access-csn4h\") pod \"ac6852fd-01be-4d69-8c9b-0c591e50b889\" (UID: \"ac6852fd-01be-4d69-8c9b-0c591e50b889\") "
Oct 30 19:17:14.121504 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:14.120912    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac6852fd-01be-4d69-8c9b-0c591e50b889-kube-api-access-csn4h" pod "ac6852fd-01be-4d69-8c9b-0c591e50b889" (UID: "ac6852fd-01be-4d69-8c9b-0c591e50b889"). InnerVolumeSpecName "kube-api-access-csn4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:14.121504 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:14.121175    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac6852fd-01be-4d69-8c9b-0c591e50b889-home" pod "ac6852fd-01be-4d69-8c9b-0c591e50b889" (UID: "ac6852fd-01be-4d69-8c9b-0c591e50b889"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:14.122009 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:14.121967    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac6852fd-01be-4d69-8c9b-0c591e50b889-model-cache" pod "ac6852fd-01be-4d69-8c9b-0c591e50b889" (UID: "ac6852fd-01be-4d69-8c9b-0c591e50b889"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:14.123756 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:14.123713    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82bbbf85-ed93-48d2-9f17-49ac1f31c0a8-tls-certs" pod "82bbbf85-ed93-48d2-9f17-49ac1f31c0a8" (UID: "82bbbf85-ed93-48d2-9f17-49ac1f31c0a8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:17:14.128708 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:14.124273    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82bbbf85-ed93-48d2-9f17-49ac1f31c0a8-home" pod "82bbbf85-ed93-48d2-9f17-49ac1f31c0a8" (UID: "82bbbf85-ed93-48d2-9f17-49ac1f31c0a8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:14.128708 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:14.125266    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82bbbf85-ed93-48d2-9f17-49ac1f31c0a8-tmp-dir" pod "82bbbf85-ed93-48d2-9f17-49ac1f31c0a8" (UID: "82bbbf85-ed93-48d2-9f17-49ac1f31c0a8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:14.128708 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:14.125443    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82bbbf85-ed93-48d2-9f17-49ac1f31c0a8-model-cache" pod "82bbbf85-ed93-48d2-9f17-49ac1f31c0a8" (UID: "82bbbf85-ed93-48d2-9f17-49ac1f31c0a8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:14.128708 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:14.126892    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac6852fd-01be-4d69-8c9b-0c591e50b889-dshm" pod "ac6852fd-01be-4d69-8c9b-0c591e50b889" (UID: "ac6852fd-01be-4d69-8c9b-0c591e50b889"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:14.129022 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:14.128993    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac6852fd-01be-4d69-8c9b-0c591e50b889-tmp-dir" pod "ac6852fd-01be-4d69-8c9b-0c591e50b889" (UID: "ac6852fd-01be-4d69-8c9b-0c591e50b889"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:14.129854 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:14.129828    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82bbbf85-ed93-48d2-9f17-49ac1f31c0a8-kube-api-access-6rf92" pod "82bbbf85-ed93-48d2-9f17-49ac1f31c0a8" (UID: "82bbbf85-ed93-48d2-9f17-49ac1f31c0a8"). InnerVolumeSpecName "kube-api-access-6rf92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:14.130019 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:14.129985    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82bbbf85-ed93-48d2-9f17-49ac1f31c0a8-dshm" pod "82bbbf85-ed93-48d2-9f17-49ac1f31c0a8" (UID: "82bbbf85-ed93-48d2-9f17-49ac1f31c0a8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:14.138708 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:14.138653    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac6852fd-01be-4d69-8c9b-0c591e50b889-tls-certs" pod "ac6852fd-01be-4d69-8c9b-0c591e50b889" (UID: "ac6852fd-01be-4d69-8c9b-0c591e50b889"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:17:14.223057 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:14.222972    4525 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac6852fd-01be-4d69-8c9b-0c591e50b889-home\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:14.223265 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:14.223249    4525 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82bbbf85-ed93-48d2-9f17-49ac1f31c0a8-tls-certs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:14.223353 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:14.223307    4525 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac6852fd-01be-4d69-8c9b-0c591e50b889-tmp-dir\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:14.223353 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:14.223325    4525 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac6852fd-01be-4d69-8c9b-0c591e50b889-model-cache\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:14.223353 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:14.223344    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rf92\" (UniqueName: \"kubernetes.io/projected/82bbbf85-ed93-48d2-9f17-49ac1f31c0a8-kube-api-access-6rf92\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:14.223535 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:14.223359    4525 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82bbbf85-ed93-48d2-9f17-49ac1f31c0a8-model-cache\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:14.223535 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:14.223374    4525 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac6852fd-01be-4d69-8c9b-0c591e50b889-tls-certs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:14.223535 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:14.223388    4525 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82bbbf85-ed93-48d2-9f17-49ac1f31c0a8-home\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:14.223535 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:14.223401    4525 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac6852fd-01be-4d69-8c9b-0c591e50b889-dshm\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:14.223535 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:14.223414    4525 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82bbbf85-ed93-48d2-9f17-49ac1f31c0a8-dshm\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:14.223535 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:14.223428    4525 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82bbbf85-ed93-48d2-9f17-49ac1f31c0a8-tmp-dir\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:14.223535 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:14.223441    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csn4h\" (UniqueName: \"kubernetes.io/projected/ac6852fd-01be-4d69-8c9b-0c591e50b889-kube-api-access-csn4h\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:14.973926 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:14.973882    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw" event={"ID":"ac6852fd-01be-4d69-8c9b-0c591e50b889","Type":"ContainerDied","Data":"12350b62126b9d0eed32b1d142c342ca8e20502912b9c9449e84c0ac34290110"}
Oct 30 19:17:14.973926 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:14.973916    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw"
Oct 30 19:17:14.974534 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:14.973952    4525 scope.go:122] "RemoveContainer" containerID="9cb591ec249cb9cc35219afe413c4bec64533b92f0bd53f5dae1c45acb0101f3"
Oct 30 19:17:14.975737 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:14.975705    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr" event={"ID":"82bbbf85-ed93-48d2-9f17-49ac1f31c0a8","Type":"ContainerDied","Data":"c34e3b4b2e73cedde3d7ce39e0a90a06d0e5d68478ef285a3ec7812e36fd3f8a"}
Oct 30 19:17:14.975842 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:14.975771    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr"
Oct 30 19:17:14.993017 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:14.992993    4525 scope.go:122] "RemoveContainer" containerID="eb4eecd9e9dc3d03699dafc4fea9e08845b6c8c2814a546b34d8b9de9d79e894"
Oct 30 19:17:15.008925 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:15.008895    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr"]
Oct 30 19:17:15.019166 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:15.019136    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-39bde3/premium-w2-39bde3-kserve-8448cb9bdb-46npr"]
Oct 30 19:17:15.036492 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:15.036459    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw"]
Oct 30 19:17:15.040855 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:15.040821    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-39bde3/simulator-w2-39bde3-kserve-58dd74ff99-ld5lw"]
Oct 30 19:17:15.554093 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:15.551513    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82bbbf85-ed93-48d2-9f17-49ac1f31c0a8" path="/var/lib/kubelet/pods/82bbbf85-ed93-48d2-9f17-49ac1f31c0a8/volumes"
Oct 30 19:17:15.554093 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:15.552384    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac6852fd-01be-4d69-8c9b-0c591e50b889" path="/var/lib/kubelet/pods/ac6852fd-01be-4d69-8c9b-0c591e50b889/volumes"
Oct 30 19:17:17.063072 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:17.063034    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-39bde3-kcldj"]
Oct 30 19:17:17.071846 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:17.071820    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-39bde3-kcldj"]
Oct 30 19:17:17.387435 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:17.387393    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-dcd326-p9pwl"]
Oct 30 19:17:17.392196 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:17.392168    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-dcd326-p9pwl"]
Oct 30 19:17:17.398633 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:17:17.398601    4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d49l" containerName="extractor"
Oct 30 19:17:17.544527 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:17.544495    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f3d341b-70f0-415d-b624-e691ecc6db16" path="/var/lib/kubelet/pods/9f3d341b-70f0-415d-b624-e691ecc6db16/volumes"
Oct 30 19:17:17.545036 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:17.545016    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d64f0ea3-54ce-4d6d-b2fd-43bbd7eb3b7e" path="/var/lib/kubelet/pods/d64f0ea3-54ce-4d6d-b2fd-43bbd7eb3b7e/volumes"
Oct 30 19:17:18.028541 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:18.028505    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6708ccfa-txvfl"]
Oct 30 19:17:18.033424 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:18.033396    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6708ccfa-txvfl"]
Oct 30 19:17:19.544240 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:19.544206    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0fe7254-bbf9-44ed-93b3-52df9ca26921" path="/var/lib/kubelet/pods/f0fe7254-bbf9-44ed-93b3-52df9ca26921/volumes"
Oct 30 19:17:22.046244 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:22.046210    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28c95bf3-7b55d"]
Oct 30 19:17:22.049311 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:22.049289    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28c95bf3-7b55d"]
Oct 30 19:17:23.543700 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:23.543644    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="783874cd-5bd5-40f4-829a-842dd6d9cafd" path="/var/lib/kubelet/pods/783874cd-5bd5-40f4-829a-842dd6d9cafd/volumes"
Oct 30 19:17:26.035315 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:26.035225    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-k7bpd"]
Oct 30 19:17:26.035825 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:26.035478    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-k7bpd" podUID="7a57dc1f-182b-4a20-9ad7-bb38b9552e4d" containerName="istio-proxy" containerID="cri-o://b61cb9e0d01a0e8a8b1f03182ddce95b4b5ea018efd5df3974ffad103dcf5ddb" gracePeriod=30
Oct 30 19:17:27.048985 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:27.048940    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-46385c-lvd87"]
Oct 30 19:17:27.049560 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:27.049538    4525 cpu_manager.go:414] "containerMap: removing container" podUID="d9094695-97bc-4924-9526-45b9e8892662" containerName="istio-proxy"
Oct 30 19:17:27.049560 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:27.049559    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9094695-97bc-4924-9526-45b9e8892662" containerName="istio-proxy"
Oct 30 19:17:27.049721 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:27.049571    4525 cpu_manager.go:414] "containerMap: removing container" podUID="b0781052-ad77-4573-bfea-16b96fbc3c5b" containerName="revoke-keys"
Oct 30 19:17:27.049721 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:27.049578    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0781052-ad77-4573-bfea-16b96fbc3c5b" containerName="revoke-keys"
Oct 30 19:17:27.049721 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:27.049592    4525 cpu_manager.go:414] "containerMap: removing container" podUID="01a5c107-dcf4-4ca5-8c9d-57c40814206d" containerName="istio-proxy"
Oct 30 19:17:27.049721 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:27.049600    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01a5c107-dcf4-4ca5-8c9d-57c40814206d" containerName="istio-proxy"
Oct 30 19:17:27.049721 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:27.049614    4525 cpu_manager.go:414] "containerMap: removing container" podUID="d9094695-97bc-4924-9526-45b9e8892662" containerName="istio-proxy"
Oct 30 19:17:27.049721 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:27.049621    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9094695-97bc-4924-9526-45b9e8892662" containerName="istio-proxy"
Oct 30 19:17:27.049721 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:27.049634    4525 cpu_manager.go:414] "containerMap: removing container" podUID="6061ede7-981c-4c07-acfc-04be1cae252d" containerName="istio-proxy"
Oct 30 19:17:27.049721 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:27.049641    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6061ede7-981c-4c07-acfc-04be1cae252d" containerName="istio-proxy"
Oct 30 19:17:27.049721 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:27.049649    4525 cpu_manager.go:414] "containerMap: removing container" podUID="6061ede7-981c-4c07-acfc-04be1cae252d" containerName="istio-proxy"
Oct 30 19:17:27.049721 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:27.049658    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6061ede7-981c-4c07-acfc-04be1cae252d" containerName="istio-proxy"
Oct 30 19:17:27.049721 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:27.049697    4525 cpu_manager.go:414] "containerMap: removing container" podUID="d64f0ea3-54ce-4d6d-b2fd-43bbd7eb3b7e" containerName="revoke-keys"
Oct 30 19:17:27.049721 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:27.049704    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d64f0ea3-54ce-4d6d-b2fd-43bbd7eb3b7e" containerName="revoke-keys"
Oct 30 19:17:27.049721 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:27.049721    4525 cpu_manager.go:414] "containerMap: removing container" podUID="82bbbf85-ed93-48d2-9f17-49ac1f31c0a8" containerName="main"
Oct 30 19:17:27.049721 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:27.049727    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82bbbf85-ed93-48d2-9f17-49ac1f31c0a8" containerName="main"
Oct 30 19:17:27.050531 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:27.049737    4525 cpu_manager.go:414] "containerMap: removing container" podUID="ac6852fd-01be-4d69-8c9b-0c591e50b889" containerName="main"
Oct 30 19:17:27.050531 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:27.049743    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac6852fd-01be-4d69-8c9b-0c591e50b889" containerName="main"
Oct 30 19:17:27.050531 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:27.049851    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="6061ede7-981c-4c07-acfc-04be1cae252d" containerName="istio-proxy"
Oct 30 19:17:27.050531 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:27.049862    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="6061ede7-981c-4c07-acfc-04be1cae252d" containerName="istio-proxy"
Oct 30 19:17:27.050531 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:27.049877    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="d64f0ea3-54ce-4d6d-b2fd-43bbd7eb3b7e" containerName="revoke-keys"
Oct 30 19:17:27.050531 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:27.049886    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="82bbbf85-ed93-48d2-9f17-49ac1f31c0a8" containerName="main"
Oct 30 19:17:27.050531 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:27.049894    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac6852fd-01be-4d69-8c9b-0c591e50b889" containerName="main"
Oct 30 19:17:27.050531 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:27.049907    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9094695-97bc-4924-9526-45b9e8892662" containerName="istio-proxy"
Oct 30 19:17:27.050531 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:27.049914    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0781052-ad77-4573-bfea-16b96fbc3c5b" containerName="revoke-keys"
Oct 30 19:17:27.050531 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:27.049922    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="01a5c107-dcf4-4ca5-8c9d-57c40814206d" containerName="istio-proxy"
Oct 30 19:17:27.050531 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:27.049930    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9094695-97bc-4924-9526-45b9e8892662" containerName="istio-proxy"
Oct 30 19:17:27.056128 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:27.056099    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-46385c-lvd87"
Oct 30 19:17:27.058569 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:27.058541    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-46385c-lvd87"]
Oct 30 19:17:27.060159 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:27.060132    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks4pq\" (UniqueName: \"kubernetes.io/projected/566e4d0f-697c-4b5d-a39d-91a36a940441-kube-api-access-ks4pq\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-46385c-lvd87\" (UID: \"566e4d0f-697c-4b5d-a39d-91a36a940441\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-46385c-lvd87"
Oct 30 19:17:27.060295 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:27.060174    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:17:27.060295 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:27.060272    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/566e4d0f-697c-4b5d-a39d-91a36a940441-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-46385c-lvd87\" (UID: \"566e4d0f-697c-4b5d-a39d-91a36a940441\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-46385c-lvd87"
Oct 30 19:17:27.060373 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:27.060347    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/566e4d0f-697c-4b5d-a39d-91a36a940441-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-46385c-lvd87\" (UID: \"566e4d0f-697c-4b5d-a39d-91a36a940441\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-46385c-lvd87"
Oct 30 19:17:27.061192 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:27.061175    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94hcn\""
Oct 30 19:17:27.161186 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:27.161150    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/566e4d0f-697c-4b5d-a39d-91a36a940441-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-46385c-lvd87\" (UID: \"566e4d0f-697c-4b5d-a39d-91a36a940441\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-46385c-lvd87"
Oct 30 19:17:27.161373 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:27.161196    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/566e4d0f-697c-4b5d-a39d-91a36a940441-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-46385c-lvd87\" (UID: \"566e4d0f-697c-4b5d-a39d-91a36a940441\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-46385c-lvd87"
Oct 30 19:17:27.161373 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:27.161253    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks4pq\" (UniqueName: \"kubernetes.io/projected/566e4d0f-697c-4b5d-a39d-91a36a940441-kube-api-access-ks4pq\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-46385c-lvd87\" (UID: \"566e4d0f-697c-4b5d-a39d-91a36a940441\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-46385c-lvd87"
Oct 30 19:17:27.161855 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:27.161825    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/566e4d0f-697c-4b5d-a39d-91a36a940441-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-46385c-lvd87\" (UID: \"566e4d0f-697c-4b5d-a39d-91a36a940441\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-46385c-lvd87"
Oct 30 19:17:27.163710 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:27.163669    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/566e4d0f-697c-4b5d-a39d-91a36a940441-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-46385c-lvd87\" (UID: \"566e4d0f-697c-4b5d-a39d-91a36a940441\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-46385c-lvd87"
Oct 30 19:17:27.177648 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:27.177620    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks4pq\" (UniqueName: \"kubernetes.io/projected/566e4d0f-697c-4b5d-a39d-91a36a940441-kube-api-access-ks4pq\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-46385c-lvd87\" (UID: \"566e4d0f-697c-4b5d-a39d-91a36a940441\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-46385c-lvd87"
Oct 30 19:17:27.371190 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:27.371152    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-46385c-lvd87"
Oct 30 19:17:27.652377 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:27.652346    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-46385c-lvd87"]
Oct 30 19:17:27.653001 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:17:27.652975    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod566e4d0f_697c_4b5d_a39d_91a36a940441.slice/crio-b0e0ec832fb6eedf46a33dfc16b0f6f570343bc089e15de3b321d638c019c03b WatchSource:0}: Error finding container b0e0ec832fb6eedf46a33dfc16b0f6f570343bc089e15de3b321d638c019c03b: Status 404 returned error can't find the container with id b0e0ec832fb6eedf46a33dfc16b0f6f570343bc089e15de3b321d638c019c03b
Oct 30 19:17:28.089203 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:28.089155    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-46385c-lvd87" event={"ID":"566e4d0f-697c-4b5d-a39d-91a36a940441","Type":"ContainerStarted","Data":"b359571ff6b5802445c519926e646b3204ff64f76bd46281c3e8b78258d15d0f"}
Oct 30 19:17:28.089630 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:28.089214    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-46385c-lvd87" event={"ID":"566e4d0f-697c-4b5d-a39d-91a36a940441","Type":"ContainerStarted","Data":"b0e0ec832fb6eedf46a33dfc16b0f6f570343bc089e15de3b321d638c019c03b"}
Oct 30 19:17:28.107977 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:28.107924    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-46385c-lvd87" podStartSLOduration=1.107909814 podStartE2EDuration="1.107909814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:17:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:17:28.106233329 +0000 UTC m=+2470.731903296" watchObservedRunningTime="2026-10-30 19:17:28.107909814 +0000 UTC m=+2470.733579818"
Oct 30 19:17:29.110203 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:29.110162    4525 generic.go:356] "Generic (PLEG): container finished" podID="566e4d0f-697c-4b5d-a39d-91a36a940441" containerID="b359571ff6b5802445c519926e646b3204ff64f76bd46281c3e8b78258d15d0f" exitCode=0
Oct 30 19:17:29.110627 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:29.110252    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-46385c-lvd87" event={"ID":"566e4d0f-697c-4b5d-a39d-91a36a940441","Type":"ContainerDied","Data":"b359571ff6b5802445c519926e646b3204ff64f76bd46281c3e8b78258d15d0f"}
Oct 30 19:17:30.274185 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:30.274159    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-46385c-lvd87"
Oct 30 19:17:30.289773 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:30.289744    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/566e4d0f-697c-4b5d-a39d-91a36a940441-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/566e4d0f-697c-4b5d-a39d-91a36a940441-maas-api-openshift-ca\") pod \"566e4d0f-697c-4b5d-a39d-91a36a940441\" (UID: \"566e4d0f-697c-4b5d-a39d-91a36a940441\") "
Oct 30 19:17:30.289931 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:30.289804    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/566e4d0f-697c-4b5d-a39d-91a36a940441-kube-api-access-ks4pq\" (UniqueName: \"kubernetes.io/projected/566e4d0f-697c-4b5d-a39d-91a36a940441-kube-api-access-ks4pq\") pod \"566e4d0f-697c-4b5d-a39d-91a36a940441\" (UID: \"566e4d0f-697c-4b5d-a39d-91a36a940441\") "
Oct 30 19:17:30.289931 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:30.289828    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/566e4d0f-697c-4b5d-a39d-91a36a940441-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/566e4d0f-697c-4b5d-a39d-91a36a940441-maas-api-xks-ca\") pod \"566e4d0f-697c-4b5d-a39d-91a36a940441\" (UID: \"566e4d0f-697c-4b5d-a39d-91a36a940441\") "
Oct 30 19:17:30.290177 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:30.290152    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/566e4d0f-697c-4b5d-a39d-91a36a940441-maas-api-openshift-ca" pod "566e4d0f-697c-4b5d-a39d-91a36a940441" (UID: "566e4d0f-697c-4b5d-a39d-91a36a940441"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:17:30.292547 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:30.292508    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/566e4d0f-697c-4b5d-a39d-91a36a940441-kube-api-access-ks4pq" pod "566e4d0f-697c-4b5d-a39d-91a36a940441" (UID: "566e4d0f-697c-4b5d-a39d-91a36a940441"). InnerVolumeSpecName "kube-api-access-ks4pq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:30.293076 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:30.293037    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/566e4d0f-697c-4b5d-a39d-91a36a940441-maas-api-xks-ca" pod "566e4d0f-697c-4b5d-a39d-91a36a940441" (UID: "566e4d0f-697c-4b5d-a39d-91a36a940441"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:17:30.391406 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:30.391324    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/566e4d0f-697c-4b5d-a39d-91a36a940441-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:30.391406 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:30.391353    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ks4pq\" (UniqueName: \"kubernetes.io/projected/566e4d0f-697c-4b5d-a39d-91a36a940441-kube-api-access-ks4pq\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:30.391406 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:30.391363    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/566e4d0f-697c-4b5d-a39d-91a36a940441-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:31.028712 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:31.028664    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09072f0e-8bsjp"]
Oct 30 19:17:31.032873 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:31.032845    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09072f0e-8bsjp"]
Oct 30 19:17:31.125314 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:31.125281    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-46385c-lvd87"
Oct 30 19:17:31.125483 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:31.125325    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-46385c-lvd87" event={"ID":"566e4d0f-697c-4b5d-a39d-91a36a940441","Type":"ContainerDied","Data":"b0e0ec832fb6eedf46a33dfc16b0f6f570343bc089e15de3b321d638c019c03b"}
Oct 30 19:17:31.125483 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:31.125383    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b0e0ec832fb6eedf46a33dfc16b0f6f570343bc089e15de3b321d638c019c03b"
Oct 30 19:17:31.303038 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:31.303006    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-k7bpd"
Oct 30 19:17:31.403236 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:31.403197    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a57dc1f-182b-4a20-9ad7-bb38b9552e4d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a57dc1f-182b-4a20-9ad7-bb38b9552e4d-istio-data\") pod \"7a57dc1f-182b-4a20-9ad7-bb38b9552e4d\" (UID: \"7a57dc1f-182b-4a20-9ad7-bb38b9552e4d\") "
Oct 30 19:17:31.403438 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:31.403250    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a57dc1f-182b-4a20-9ad7-bb38b9552e4d-istio-token\" (UniqueName: \"kubernetes.io/projected/7a57dc1f-182b-4a20-9ad7-bb38b9552e4d-istio-token\") pod \"7a57dc1f-182b-4a20-9ad7-bb38b9552e4d\" (UID: \"7a57dc1f-182b-4a20-9ad7-bb38b9552e4d\") "
Oct 30 19:17:31.403438 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:31.403321    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a57dc1f-182b-4a20-9ad7-bb38b9552e4d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a57dc1f-182b-4a20-9ad7-bb38b9552e4d-istio-envoy\") pod \"7a57dc1f-182b-4a20-9ad7-bb38b9552e4d\" (UID: \"7a57dc1f-182b-4a20-9ad7-bb38b9552e4d\") "
Oct 30 19:17:31.403438 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:31.403365    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a57dc1f-182b-4a20-9ad7-bb38b9552e4d-kube-api-access-mszfs\" (UniqueName: \"kubernetes.io/projected/7a57dc1f-182b-4a20-9ad7-bb38b9552e4d-kube-api-access-mszfs\") pod \"7a57dc1f-182b-4a20-9ad7-bb38b9552e4d\" (UID: \"7a57dc1f-182b-4a20-9ad7-bb38b9552e4d\") "
Oct 30 19:17:31.403438 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:31.403405    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a57dc1f-182b-4a20-9ad7-bb38b9552e4d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a57dc1f-182b-4a20-9ad7-bb38b9552e4d-istiod-ca-cert\") pod \"7a57dc1f-182b-4a20-9ad7-bb38b9552e4d\" (UID: \"7a57dc1f-182b-4a20-9ad7-bb38b9552e4d\") "
Oct 30 19:17:31.403728 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:31.403438    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a57dc1f-182b-4a20-9ad7-bb38b9552e4d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a57dc1f-182b-4a20-9ad7-bb38b9552e4d-workload-certs\") pod \"7a57dc1f-182b-4a20-9ad7-bb38b9552e4d\" (UID: \"7a57dc1f-182b-4a20-9ad7-bb38b9552e4d\") "
Oct 30 19:17:31.403728 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:31.403521    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7a57dc1f-182b-4a20-9ad7-bb38b9552e4d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a57dc1f-182b-4a20-9ad7-bb38b9552e4d-istio-podinfo\") pod \"7a57dc1f-182b-4a20-9ad7-bb38b9552e4d\" (UID: \"7a57dc1f-182b-4a20-9ad7-bb38b9552e4d\") "
Oct 30 19:17:31.403728 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:31.403527    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a57dc1f-182b-4a20-9ad7-bb38b9552e4d-istio-data" pod "7a57dc1f-182b-4a20-9ad7-bb38b9552e4d" (UID: "7a57dc1f-182b-4a20-9ad7-bb38b9552e4d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:31.403728 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:31.403575    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a57dc1f-182b-4a20-9ad7-bb38b9552e4d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a57dc1f-182b-4a20-9ad7-bb38b9552e4d-workload-socket\") pod \"7a57dc1f-182b-4a20-9ad7-bb38b9552e4d\" (UID: \"7a57dc1f-182b-4a20-9ad7-bb38b9552e4d\") "
Oct 30 19:17:31.403728 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:31.403634    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a57dc1f-182b-4a20-9ad7-bb38b9552e4d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a57dc1f-182b-4a20-9ad7-bb38b9552e4d-credential-socket\") pod \"7a57dc1f-182b-4a20-9ad7-bb38b9552e4d\" (UID: \"7a57dc1f-182b-4a20-9ad7-bb38b9552e4d\") "
Oct 30 19:17:31.404028 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:31.404005    4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a57dc1f-182b-4a20-9ad7-bb38b9552e4d-istio-data\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:31.404085 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:31.404018    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a57dc1f-182b-4a20-9ad7-bb38b9552e4d-istiod-ca-cert" pod "7a57dc1f-182b-4a20-9ad7-bb38b9552e4d" (UID: "7a57dc1f-182b-4a20-9ad7-bb38b9552e4d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:17:31.404193 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:31.404163    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a57dc1f-182b-4a20-9ad7-bb38b9552e4d-workload-socket" pod "7a57dc1f-182b-4a20-9ad7-bb38b9552e4d" (UID: "7a57dc1f-182b-4a20-9ad7-bb38b9552e4d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:31.404280 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:31.404261    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a57dc1f-182b-4a20-9ad7-bb38b9552e4d-workload-certs" pod "7a57dc1f-182b-4a20-9ad7-bb38b9552e4d" (UID: "7a57dc1f-182b-4a20-9ad7-bb38b9552e4d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:31.404355 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:31.404277    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a57dc1f-182b-4a20-9ad7-bb38b9552e4d-credential-socket" pod "7a57dc1f-182b-4a20-9ad7-bb38b9552e4d" (UID: "7a57dc1f-182b-4a20-9ad7-bb38b9552e4d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:31.406406 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:31.406379    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a57dc1f-182b-4a20-9ad7-bb38b9552e4d-istio-envoy" pod "7a57dc1f-182b-4a20-9ad7-bb38b9552e4d" (UID: "7a57dc1f-182b-4a20-9ad7-bb38b9552e4d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:31.406406 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:31.406384    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a57dc1f-182b-4a20-9ad7-bb38b9552e4d-istio-token" pod "7a57dc1f-182b-4a20-9ad7-bb38b9552e4d" (UID: "7a57dc1f-182b-4a20-9ad7-bb38b9552e4d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:31.406608 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:31.406437    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a57dc1f-182b-4a20-9ad7-bb38b9552e4d-kube-api-access-mszfs" pod "7a57dc1f-182b-4a20-9ad7-bb38b9552e4d" (UID: "7a57dc1f-182b-4a20-9ad7-bb38b9552e4d"). InnerVolumeSpecName "kube-api-access-mszfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:31.406767 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:31.406662    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7a57dc1f-182b-4a20-9ad7-bb38b9552e4d-istio-podinfo" pod "7a57dc1f-182b-4a20-9ad7-bb38b9552e4d" (UID: "7a57dc1f-182b-4a20-9ad7-bb38b9552e4d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:17:31.504731 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:31.504662    4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a57dc1f-182b-4a20-9ad7-bb38b9552e4d-istio-podinfo\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:31.504731 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:31.504726    4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a57dc1f-182b-4a20-9ad7-bb38b9552e4d-workload-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:31.504731 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:31.504736    4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a57dc1f-182b-4a20-9ad7-bb38b9552e4d-credential-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:31.504986 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:31.504744    4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a57dc1f-182b-4a20-9ad7-bb38b9552e4d-istio-token\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:31.504986 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:31.504753    4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a57dc1f-182b-4a20-9ad7-bb38b9552e4d-istio-envoy\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:31.504986 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:31.504761    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mszfs\" (UniqueName: \"kubernetes.io/projected/7a57dc1f-182b-4a20-9ad7-bb38b9552e4d-kube-api-access-mszfs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:31.504986 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:31.504770    4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a57dc1f-182b-4a20-9ad7-bb38b9552e4d-istiod-ca-cert\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:31.504986 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:31.504778    4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a57dc1f-182b-4a20-9ad7-bb38b9552e4d-workload-certs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:31.543409 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:31.543326    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e9005e6-75e2-4244-8f75-3a2c7eaeee08" path="/var/lib/kubelet/pods/1e9005e6-75e2-4244-8f75-3a2c7eaeee08/volumes"
Oct 30 19:17:32.134848 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:32.134811    4525 generic.go:356] "Generic (PLEG): container finished" podID="7a57dc1f-182b-4a20-9ad7-bb38b9552e4d" containerID="b61cb9e0d01a0e8a8b1f03182ddce95b4b5ea018efd5df3974ffad103dcf5ddb" exitCode=0
Oct 30 19:17:32.135072 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:32.134868    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-k7bpd"
Oct 30 19:17:32.135072 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:32.134888    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-k7bpd" event={"ID":"7a57dc1f-182b-4a20-9ad7-bb38b9552e4d","Type":"ContainerDied","Data":"b61cb9e0d01a0e8a8b1f03182ddce95b4b5ea018efd5df3974ffad103dcf5ddb"}
Oct 30 19:17:32.135072 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:32.134941    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-k7bpd" event={"ID":"7a57dc1f-182b-4a20-9ad7-bb38b9552e4d","Type":"ContainerDied","Data":"e2a62632942103e867e6e00131f5048dc55f4dda8d7202beb316a15a9a7b0348"}
Oct 30 19:17:32.135072 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:32.134963    4525 scope.go:122] "RemoveContainer" containerID="b61cb9e0d01a0e8a8b1f03182ddce95b4b5ea018efd5df3974ffad103dcf5ddb"
Oct 30 19:17:32.146207 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:32.146160    4525 scope.go:122] "RemoveContainer" containerID="b61cb9e0d01a0e8a8b1f03182ddce95b4b5ea018efd5df3974ffad103dcf5ddb"
Oct 30 19:17:32.146480 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:17:32.146461    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b61cb9e0d01a0e8a8b1f03182ddce95b4b5ea018efd5df3974ffad103dcf5ddb\": container with ID starting with b61cb9e0d01a0e8a8b1f03182ddce95b4b5ea018efd5df3974ffad103dcf5ddb not found: ID does not exist" containerID="b61cb9e0d01a0e8a8b1f03182ddce95b4b5ea018efd5df3974ffad103dcf5ddb"
Oct 30 19:17:32.146536 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:32.146489    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b61cb9e0d01a0e8a8b1f03182ddce95b4b5ea018efd5df3974ffad103dcf5ddb"} err="failed to get container status \"b61cb9e0d01a0e8a8b1f03182ddce95b4b5ea018efd5df3974ffad103dcf5ddb\": rpc error: code = NotFound desc = could not find container \"b61cb9e0d01a0e8a8b1f03182ddce95b4b5ea018efd5df3974ffad103dcf5ddb\": container with ID starting with b61cb9e0d01a0e8a8b1f03182ddce95b4b5ea018efd5df3974ffad103dcf5ddb not found: ID does not exist"
Oct 30 19:17:32.172451 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:32.172418    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-k7bpd"]
Oct 30 19:17:32.186268 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:32.186233    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-39bde3-openshift-default-5d45cdbd49-k7bpd"]
Oct 30 19:17:33.543599 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:33.543561    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a57dc1f-182b-4a20-9ad7-bb38b9552e4d" path="/var/lib/kubelet/pods/7a57dc1f-182b-4a20-9ad7-bb38b9552e4d/volumes"
Oct 30 19:17:39.710326 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:39.710282    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-46385c-lvd87"]
Oct 30 19:17:39.713324 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:39.713299    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-46385c-lvd87"]
Oct 30 19:17:41.543436 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:41.543386    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="566e4d0f-697c-4b5d-a39d-91a36a940441" path="/var/lib/kubelet/pods/566e4d0f-697c-4b5d-a39d-91a36a940441/volumes"
Oct 30 19:17:47.025792 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:47.025753    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b16e2a6-7z2nj"]
Oct 30 19:17:47.029397 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:47.029377    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b16e2a6-7z2nj"]
Oct 30 19:17:47.152576 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:47.152543    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-46385c-openshift-default-6ccdb978d5-4gmh4"]
Oct 30 19:17:47.152862 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:47.152805    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-46385c-openshift-default-6ccdb978d5-4gmh4" podUID="7ec64e5e-2c2b-47d8-8070-296ce41ada09" containerName="istio-proxy" containerID="cri-o://fbd1aac2be1a8e31597e2f8ead67cba4c839c6c061cd745c17b63ed13c0d0f2a" gracePeriod=30
Oct 30 19:17:47.398122 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:17:47.398078    4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d49l" containerName="extractor"
Oct 30 19:17:47.543768 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:47.543734    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85f38f2b-bce9-4d9f-9cfb-a0a6222f8d78" path="/var/lib/kubelet/pods/85f38f2b-bce9-4d9f-9cfb-a0a6222f8d78/volumes"
Oct 30 19:17:47.871758 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:47.871724    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-85cb41-5f5wm"]
Oct 30 19:17:47.872137 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:47.872125    4525 cpu_manager.go:414] "containerMap: removing container" podUID="7a57dc1f-182b-4a20-9ad7-bb38b9552e4d" containerName="istio-proxy"
Oct 30 19:17:47.872188 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:47.872138    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a57dc1f-182b-4a20-9ad7-bb38b9552e4d" containerName="istio-proxy"
Oct 30 19:17:47.872188 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:47.872165    4525 cpu_manager.go:414] "containerMap: removing container" podUID="566e4d0f-697c-4b5d-a39d-91a36a940441" containerName="revoke-keys"
Oct 30 19:17:47.872188 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:47.872170    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="566e4d0f-697c-4b5d-a39d-91a36a940441" containerName="revoke-keys"
Oct 30 19:17:47.872286 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:47.872232    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="566e4d0f-697c-4b5d-a39d-91a36a940441" containerName="revoke-keys"
Oct 30 19:17:47.872286 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:47.872244    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a57dc1f-182b-4a20-9ad7-bb38b9552e4d" containerName="istio-proxy"
Oct 30 19:17:47.875637 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:47.875620    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-85cb41-5f5wm"
Oct 30 19:17:47.880642 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:47.880610    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:17:47.886093 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:47.886067    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-85cb41-5f5wm"]
Oct 30 19:17:47.913075 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:47.912975    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94hcn\""
Oct 30 19:17:47.984592 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:47.984565    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqcw8\" (UniqueName: \"kubernetes.io/projected/d0c4cdba-b018-48f1-9db4-8c9a802632ef-kube-api-access-wqcw8\") pod \"maas-api-revoke-keys-e2e-worker-w1-85cb41-5f5wm\" (UID: \"d0c4cdba-b018-48f1-9db4-8c9a802632ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-85cb41-5f5wm"
Oct 30 19:17:47.984775 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:47.984723    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0c4cdba-b018-48f1-9db4-8c9a802632ef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-85cb41-5f5wm\" (UID: \"d0c4cdba-b018-48f1-9db4-8c9a802632ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-85cb41-5f5wm"
Oct 30 19:17:47.984775 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:47.984757    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0c4cdba-b018-48f1-9db4-8c9a802632ef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-85cb41-5f5wm\" (UID: \"d0c4cdba-b018-48f1-9db4-8c9a802632ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-85cb41-5f5wm"
Oct 30 19:17:48.086191 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:48.086161    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0c4cdba-b018-48f1-9db4-8c9a802632ef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-85cb41-5f5wm\" (UID: \"d0c4cdba-b018-48f1-9db4-8c9a802632ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-85cb41-5f5wm"
Oct 30 19:17:48.086568 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:48.086205    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0c4cdba-b018-48f1-9db4-8c9a802632ef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-85cb41-5f5wm\" (UID: \"d0c4cdba-b018-48f1-9db4-8c9a802632ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-85cb41-5f5wm"
Oct 30 19:17:48.086568 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:48.086260    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqcw8\" (UniqueName: \"kubernetes.io/projected/d0c4cdba-b018-48f1-9db4-8c9a802632ef-kube-api-access-wqcw8\") pod \"maas-api-revoke-keys-e2e-worker-w1-85cb41-5f5wm\" (UID: \"d0c4cdba-b018-48f1-9db4-8c9a802632ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-85cb41-5f5wm"
Oct 30 19:17:48.086830 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:48.086809    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0c4cdba-b018-48f1-9db4-8c9a802632ef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-85cb41-5f5wm\" (UID: \"d0c4cdba-b018-48f1-9db4-8c9a802632ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-85cb41-5f5wm"
Oct 30 19:17:48.088694 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:48.088664    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0c4cdba-b018-48f1-9db4-8c9a802632ef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-85cb41-5f5wm\" (UID: \"d0c4cdba-b018-48f1-9db4-8c9a802632ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-85cb41-5f5wm"
Oct 30 19:17:48.095443 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:48.095419    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqcw8\" (UniqueName: \"kubernetes.io/projected/d0c4cdba-b018-48f1-9db4-8c9a802632ef-kube-api-access-wqcw8\") pod \"maas-api-revoke-keys-e2e-worker-w1-85cb41-5f5wm\" (UID: \"d0c4cdba-b018-48f1-9db4-8c9a802632ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-85cb41-5f5wm"
Oct 30 19:17:48.189964 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:48.189892    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-85cb41-5f5wm"
Oct 30 19:17:48.475890 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:48.475858    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-85cb41-5f5wm"]
Oct 30 19:17:48.477063 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:17:48.477041    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0c4cdba_b018_48f1_9db4_8c9a802632ef.slice/crio-c3245ca6661bdd981bdca5f0b31a1c62b122e4f23bd8e11ede0eb248e5f3073b WatchSource:0}: Error finding container c3245ca6661bdd981bdca5f0b31a1c62b122e4f23bd8e11ede0eb248e5f3073b: Status 404 returned error can't find the container with id c3245ca6661bdd981bdca5f0b31a1c62b122e4f23bd8e11ede0eb248e5f3073b
Oct 30 19:17:49.271433 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:49.271350    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-85cb41-5f5wm" event={"ID":"d0c4cdba-b018-48f1-9db4-8c9a802632ef","Type":"ContainerStarted","Data":"a355cd10c627a7795d10e390e0a30a0a98681d53d3ee78e37734d9259ac0ac96"}
Oct 30 19:17:49.271433 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:49.271403    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-85cb41-5f5wm" event={"ID":"d0c4cdba-b018-48f1-9db4-8c9a802632ef","Type":"ContainerStarted","Data":"c3245ca6661bdd981bdca5f0b31a1c62b122e4f23bd8e11ede0eb248e5f3073b"}
Oct 30 19:17:49.290376 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:49.290323    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-85cb41-5f5wm" podStartSLOduration=2.290308062 podStartE2EDuration="2.290308062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:17:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:17:49.287875015 +0000 UTC m=+2491.913544995" watchObservedRunningTime="2026-10-30 19:17:49.290308062 +0000 UTC m=+2491.915978074"
Oct 30 19:17:50.280325 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:50.280292    4525 generic.go:356] "Generic (PLEG): container finished" podID="d0c4cdba-b018-48f1-9db4-8c9a802632ef" containerID="a355cd10c627a7795d10e390e0a30a0a98681d53d3ee78e37734d9259ac0ac96" exitCode=0
Oct 30 19:17:50.280753 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:50.280377    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-85cb41-5f5wm" event={"ID":"d0c4cdba-b018-48f1-9db4-8c9a802632ef","Type":"ContainerDied","Data":"a355cd10c627a7795d10e390e0a30a0a98681d53d3ee78e37734d9259ac0ac96"}
Oct 30 19:17:51.422099 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:51.422068    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-85cb41-5f5wm"
Oct 30 19:17:51.518516 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:51.518480    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0c4cdba-b018-48f1-9db4-8c9a802632ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0c4cdba-b018-48f1-9db4-8c9a802632ef-maas-api-xks-ca\") pod \"d0c4cdba-b018-48f1-9db4-8c9a802632ef\" (UID: \"d0c4cdba-b018-48f1-9db4-8c9a802632ef\") "
Oct 30 19:17:51.518740 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:51.518585    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0c4cdba-b018-48f1-9db4-8c9a802632ef-kube-api-access-wqcw8\" (UniqueName: \"kubernetes.io/projected/d0c4cdba-b018-48f1-9db4-8c9a802632ef-kube-api-access-wqcw8\") pod \"d0c4cdba-b018-48f1-9db4-8c9a802632ef\" (UID: \"d0c4cdba-b018-48f1-9db4-8c9a802632ef\") "
Oct 30 19:17:51.518740 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:51.518605    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0c4cdba-b018-48f1-9db4-8c9a802632ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0c4cdba-b018-48f1-9db4-8c9a802632ef-maas-api-openshift-ca\") pod \"d0c4cdba-b018-48f1-9db4-8c9a802632ef\" (UID: \"d0c4cdba-b018-48f1-9db4-8c9a802632ef\") "
Oct 30 19:17:51.519028 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:51.519004    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0c4cdba-b018-48f1-9db4-8c9a802632ef-maas-api-openshift-ca" pod "d0c4cdba-b018-48f1-9db4-8c9a802632ef" (UID: "d0c4cdba-b018-48f1-9db4-8c9a802632ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:17:51.520943 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:51.520923    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0c4cdba-b018-48f1-9db4-8c9a802632ef-maas-api-xks-ca" pod "d0c4cdba-b018-48f1-9db4-8c9a802632ef" (UID: "d0c4cdba-b018-48f1-9db4-8c9a802632ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:17:51.521030 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:51.521007    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0c4cdba-b018-48f1-9db4-8c9a802632ef-kube-api-access-wqcw8" pod "d0c4cdba-b018-48f1-9db4-8c9a802632ef" (UID: "d0c4cdba-b018-48f1-9db4-8c9a802632ef"). InnerVolumeSpecName "kube-api-access-wqcw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:51.619783 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:51.619747    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0c4cdba-b018-48f1-9db4-8c9a802632ef-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:51.619783 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:51.619779    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqcw8\" (UniqueName: \"kubernetes.io/projected/d0c4cdba-b018-48f1-9db4-8c9a802632ef-kube-api-access-wqcw8\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:51.619783 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:51.619790    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0c4cdba-b018-48f1-9db4-8c9a802632ef-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:52.296590 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:52.296548    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-85cb41-5f5wm"
Oct 30 19:17:52.296802 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:52.296557    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-85cb41-5f5wm" event={"ID":"d0c4cdba-b018-48f1-9db4-8c9a802632ef","Type":"ContainerDied","Data":"c3245ca6661bdd981bdca5f0b31a1c62b122e4f23bd8e11ede0eb248e5f3073b"}
Oct 30 19:17:52.296802 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:52.296726    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3245ca6661bdd981bdca5f0b31a1c62b122e4f23bd8e11ede0eb248e5f3073b"
Oct 30 19:17:52.299231 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:52.299194    4525 generic.go:356] "Generic (PLEG): container finished" podID="7ec64e5e-2c2b-47d8-8070-296ce41ada09" containerID="fbd1aac2be1a8e31597e2f8ead67cba4c839c6c061cd745c17b63ed13c0d0f2a" exitCode=0
Oct 30 19:17:52.299355 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:52.299245    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-46385c-openshift-default-6ccdb978d5-4gmh4" event={"ID":"7ec64e5e-2c2b-47d8-8070-296ce41ada09","Type":"ContainerDied","Data":"fbd1aac2be1a8e31597e2f8ead67cba4c839c6c061cd745c17b63ed13c0d0f2a"}
Oct 30 19:17:52.299355 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:52.299298    4525 scope.go:122] "RemoveContainer" containerID="6370674b13dd336f02fd228a6ea078f74a5fbbdef9f26eedd9354ada3ed2fdbd"
Oct 30 19:17:52.430079 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:52.430022    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-46385c-openshift-default-6ccdb978d5-4gmh4"
Oct 30 19:17:52.528566 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:52.528536    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ec64e5e-2c2b-47d8-8070-296ce41ada09-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ec64e5e-2c2b-47d8-8070-296ce41ada09-istio-data\") pod \"7ec64e5e-2c2b-47d8-8070-296ce41ada09\" (UID: \"7ec64e5e-2c2b-47d8-8070-296ce41ada09\") "
Oct 30 19:17:52.528766 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:52.528585    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ec64e5e-2c2b-47d8-8070-296ce41ada09-kube-api-access-sncml\" (UniqueName: \"kubernetes.io/projected/7ec64e5e-2c2b-47d8-8070-296ce41ada09-kube-api-access-sncml\") pod \"7ec64e5e-2c2b-47d8-8070-296ce41ada09\" (UID: \"7ec64e5e-2c2b-47d8-8070-296ce41ada09\") "
Oct 30 19:17:52.528766 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:52.528621    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ec64e5e-2c2b-47d8-8070-296ce41ada09-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ec64e5e-2c2b-47d8-8070-296ce41ada09-istio-envoy\") pod \"7ec64e5e-2c2b-47d8-8070-296ce41ada09\" (UID: \"7ec64e5e-2c2b-47d8-8070-296ce41ada09\") "
Oct 30 19:17:52.528766 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:52.528650    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ec64e5e-2c2b-47d8-8070-296ce41ada09-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ec64e5e-2c2b-47d8-8070-296ce41ada09-workload-certs\") pod \"7ec64e5e-2c2b-47d8-8070-296ce41ada09\" (UID: \"7ec64e5e-2c2b-47d8-8070-296ce41ada09\") "
Oct 30 19:17:52.528766 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:52.528717    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ec64e5e-2c2b-47d8-8070-296ce41ada09-istio-token\" (UniqueName: \"kubernetes.io/projected/7ec64e5e-2c2b-47d8-8070-296ce41ada09-istio-token\") pod \"7ec64e5e-2c2b-47d8-8070-296ce41ada09\" (UID: \"7ec64e5e-2c2b-47d8-8070-296ce41ada09\") "
Oct 30 19:17:52.529012 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:52.528895    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ec64e5e-2c2b-47d8-8070-296ce41ada09-istio-data" pod "7ec64e5e-2c2b-47d8-8070-296ce41ada09" (UID: "7ec64e5e-2c2b-47d8-8070-296ce41ada09"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:52.529012 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:52.528903    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ec64e5e-2c2b-47d8-8070-296ce41ada09-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ec64e5e-2c2b-47d8-8070-296ce41ada09-credential-socket\") pod \"7ec64e5e-2c2b-47d8-8070-296ce41ada09\" (UID: \"7ec64e5e-2c2b-47d8-8070-296ce41ada09\") "
Oct 30 19:17:52.529129 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:52.529029    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7ec64e5e-2c2b-47d8-8070-296ce41ada09-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ec64e5e-2c2b-47d8-8070-296ce41ada09-istio-podinfo\") pod \"7ec64e5e-2c2b-47d8-8070-296ce41ada09\" (UID: \"7ec64e5e-2c2b-47d8-8070-296ce41ada09\") "
Oct 30 19:17:52.529129 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:52.529097    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ec64e5e-2c2b-47d8-8070-296ce41ada09-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ec64e5e-2c2b-47d8-8070-296ce41ada09-istiod-ca-cert\") pod \"7ec64e5e-2c2b-47d8-8070-296ce41ada09\" (UID: \"7ec64e5e-2c2b-47d8-8070-296ce41ada09\") "
Oct 30 19:17:52.529257 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:52.529133    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ec64e5e-2c2b-47d8-8070-296ce41ada09-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ec64e5e-2c2b-47d8-8070-296ce41ada09-workload-socket\") pod \"7ec64e5e-2c2b-47d8-8070-296ce41ada09\" (UID: \"7ec64e5e-2c2b-47d8-8070-296ce41ada09\") "
Oct 30 19:17:52.529411 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:52.529131    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ec64e5e-2c2b-47d8-8070-296ce41ada09-credential-socket" pod "7ec64e5e-2c2b-47d8-8070-296ce41ada09" (UID: "7ec64e5e-2c2b-47d8-8070-296ce41ada09"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:52.529411 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:52.529146    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ec64e5e-2c2b-47d8-8070-296ce41ada09-workload-certs" pod "7ec64e5e-2c2b-47d8-8070-296ce41ada09" (UID: "7ec64e5e-2c2b-47d8-8070-296ce41ada09"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:52.529411 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:52.529356    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ec64e5e-2c2b-47d8-8070-296ce41ada09-workload-socket" pod "7ec64e5e-2c2b-47d8-8070-296ce41ada09" (UID: "7ec64e5e-2c2b-47d8-8070-296ce41ada09"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:52.529878 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:52.529445    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ec64e5e-2c2b-47d8-8070-296ce41ada09-istiod-ca-cert" pod "7ec64e5e-2c2b-47d8-8070-296ce41ada09" (UID: "7ec64e5e-2c2b-47d8-8070-296ce41ada09"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:17:52.529878 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:52.529784    4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ec64e5e-2c2b-47d8-8070-296ce41ada09-istio-data\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:52.529878 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:52.529809    4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ec64e5e-2c2b-47d8-8070-296ce41ada09-workload-certs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:52.529878 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:52.529825    4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ec64e5e-2c2b-47d8-8070-296ce41ada09-credential-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:52.529878 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:52.529842    4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ec64e5e-2c2b-47d8-8070-296ce41ada09-istiod-ca-cert\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:52.529878 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:52.529855    4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ec64e5e-2c2b-47d8-8070-296ce41ada09-workload-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:52.531323 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:52.531294    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7ec64e5e-2c2b-47d8-8070-296ce41ada09-istio-podinfo" pod "7ec64e5e-2c2b-47d8-8070-296ce41ada09" (UID: "7ec64e5e-2c2b-47d8-8070-296ce41ada09"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:17:52.531441 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:52.531342    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ec64e5e-2c2b-47d8-8070-296ce41ada09-istio-envoy" pod "7ec64e5e-2c2b-47d8-8070-296ce41ada09" (UID: "7ec64e5e-2c2b-47d8-8070-296ce41ada09"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:52.531441 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:52.531407    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ec64e5e-2c2b-47d8-8070-296ce41ada09-istio-token" pod "7ec64e5e-2c2b-47d8-8070-296ce41ada09" (UID: "7ec64e5e-2c2b-47d8-8070-296ce41ada09"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:52.531526 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:52.531439    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ec64e5e-2c2b-47d8-8070-296ce41ada09-kube-api-access-sncml" pod "7ec64e5e-2c2b-47d8-8070-296ce41ada09" (UID: "7ec64e5e-2c2b-47d8-8070-296ce41ada09"). InnerVolumeSpecName "kube-api-access-sncml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:52.631250 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:52.631215    4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ec64e5e-2c2b-47d8-8070-296ce41ada09-istio-token\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:52.631250 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:52.631248    4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ec64e5e-2c2b-47d8-8070-296ce41ada09-istio-podinfo\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:52.631479 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:52.631267    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sncml\" (UniqueName: \"kubernetes.io/projected/7ec64e5e-2c2b-47d8-8070-296ce41ada09-kube-api-access-sncml\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:52.631479 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:52.631281    4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ec64e5e-2c2b-47d8-8070-296ce41ada09-istio-envoy\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:52.691398 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:52.691313    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts"]
Oct 30 19:17:53.307864 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:53.307831    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-46385c-openshift-default-6ccdb978d5-4gmh4"
Oct 30 19:17:53.307864 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:53.307847    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-46385c-openshift-default-6ccdb978d5-4gmh4" event={"ID":"7ec64e5e-2c2b-47d8-8070-296ce41ada09","Type":"ContainerDied","Data":"b4ee24bd41e3c25e7d8fec1ba1ffe7abf2e47ef0393452146a58fdbfcc4f4d28"}
Oct 30 19:17:53.308116 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:53.307907    4525 scope.go:122] "RemoveContainer" containerID="fbd1aac2be1a8e31597e2f8ead67cba4c839c6c061cd745c17b63ed13c0d0f2a"
Oct 30 19:17:53.332647 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:53.332624    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-46385c-openshift-default-6ccdb978d5-4gmh4"]
Oct 30 19:17:53.337476 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:53.337453    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-46385c-openshift-default-6ccdb978d5-4gmh4"]
Oct 30 19:17:53.543730 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:53.543701    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ec64e5e-2c2b-47d8-8070-296ce41ada09" path="/var/lib/kubelet/pods/7ec64e5e-2c2b-47d8-8070-296ce41ada09/volumes"
Oct 30 19:17:58.638531 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:58.638502    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c3362e0-mn964"]
Oct 30 19:17:58.638951 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:58.638933    4525 cpu_manager.go:414] "containerMap: removing container" podUID="7ec64e5e-2c2b-47d8-8070-296ce41ada09" containerName="istio-proxy"
Oct 30 19:17:58.638951 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:58.638944    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ec64e5e-2c2b-47d8-8070-296ce41ada09" containerName="istio-proxy"
Oct 30 19:17:58.639055 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:58.638957    4525 cpu_manager.go:414] "containerMap: removing container" podUID="d0c4cdba-b018-48f1-9db4-8c9a802632ef" containerName="revoke-keys"
Oct 30 19:17:58.639055 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:58.638962    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0c4cdba-b018-48f1-9db4-8c9a802632ef" containerName="revoke-keys"
Oct 30 19:17:58.639055 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:58.638968    4525 cpu_manager.go:414] "containerMap: removing container" podUID="7ec64e5e-2c2b-47d8-8070-296ce41ada09" containerName="istio-proxy"
Oct 30 19:17:58.639055 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:58.638973    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ec64e5e-2c2b-47d8-8070-296ce41ada09" containerName="istio-proxy"
Oct 30 19:17:58.639055 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:58.639033    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ec64e5e-2c2b-47d8-8070-296ce41ada09" containerName="istio-proxy"
Oct 30 19:17:58.639055 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:58.639040    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0c4cdba-b018-48f1-9db4-8c9a802632ef" containerName="revoke-keys"
Oct 30 19:17:58.643426 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:58.643395    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c3362e0-mn964"
Oct 30 19:17:58.653553 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:58.653528    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c3362e0-mn964"]
Oct 30 19:17:58.686382 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:58.686355    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:17:58.686533 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:58.686435    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-94hcn\""
Oct 30 19:17:58.795021 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:58.794987    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrc9q\" (UniqueName: \"kubernetes.io/projected/fc981614-0ce4-468c-9457-c972e4caf4d7-kube-api-access-nrc9q\") pod \"maas-api-revoke-sub-1c3362e0-mn964\" (UID: \"fc981614-0ce4-468c-9457-c972e4caf4d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c3362e0-mn964"
Oct 30 19:17:58.795209 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:58.795117    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc981614-0ce4-468c-9457-c972e4caf4d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c3362e0-mn964\" (UID: \"fc981614-0ce4-468c-9457-c972e4caf4d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c3362e0-mn964"
Oct 30 19:17:58.795209 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:58.795155    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc981614-0ce4-468c-9457-c972e4caf4d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c3362e0-mn964\" (UID: \"fc981614-0ce4-468c-9457-c972e4caf4d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c3362e0-mn964"
Oct 30 19:17:58.896031 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:58.895950    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc981614-0ce4-468c-9457-c972e4caf4d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c3362e0-mn964\" (UID: \"fc981614-0ce4-468c-9457-c972e4caf4d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c3362e0-mn964"
Oct 30 19:17:58.896031 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:58.895993    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc981614-0ce4-468c-9457-c972e4caf4d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c3362e0-mn964\" (UID: \"fc981614-0ce4-468c-9457-c972e4caf4d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c3362e0-mn964"
Oct 30 19:17:58.896243 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:58.896044    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrc9q\" (UniqueName: \"kubernetes.io/projected/fc981614-0ce4-468c-9457-c972e4caf4d7-kube-api-access-nrc9q\") pod \"maas-api-revoke-sub-1c3362e0-mn964\" (UID: \"fc981614-0ce4-468c-9457-c972e4caf4d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c3362e0-mn964"
Oct 30 19:17:58.896621 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:58.896599    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc981614-0ce4-468c-9457-c972e4caf4d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c3362e0-mn964\" (UID: \"fc981614-0ce4-468c-9457-c972e4caf4d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c3362e0-mn964"
Oct 30 19:17:58.898452 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:58.898433    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc981614-0ce4-468c-9457-c972e4caf4d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c3362e0-mn964\" (UID: \"fc981614-0ce4-468c-9457-c972e4caf4d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c3362e0-mn964"
Oct 30 19:17:58.903910 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:58.903891    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrc9q\" (UniqueName: \"kubernetes.io/projected/fc981614-0ce4-468c-9457-c972e4caf4d7-kube-api-access-nrc9q\") pod \"maas-api-revoke-sub-1c3362e0-mn964\" (UID: \"fc981614-0ce4-468c-9457-c972e4caf4d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c3362e0-mn964"
Oct 30 19:17:58.961238 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:58.961210    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c3362e0-mn964"
Oct 30 19:17:59.251131 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:59.251099    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c3362e0-mn964"]
Oct 30 19:17:59.251824 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:17:59.251800    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc981614_0ce4_468c_9457_c972e4caf4d7.slice/crio-f258057e9bc430af577b8359616f64cd2850457a0fd2959347a3211724a664d7 WatchSource:0}: Error finding container f258057e9bc430af577b8359616f64cd2850457a0fd2959347a3211724a664d7: Status 404 returned error can't find the container with id f258057e9bc430af577b8359616f64cd2850457a0fd2959347a3211724a664d7
Oct 30 19:17:59.354402 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:17:59.354366    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c3362e0-mn964" event={"ID":"fc981614-0ce4-468c-9457-c972e4caf4d7","Type":"ContainerStarted","Data":"f258057e9bc430af577b8359616f64cd2850457a0fd2959347a3211724a664d7"}
Oct 30 19:18:00.364943 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:00.364902    4525 generic.go:356] "Generic (PLEG): container finished" podID="fc981614-0ce4-468c-9457-c972e4caf4d7" containerID="3035ec79edd632ae1d1b39b42fe2e05b6df8869a00e9c4475b82e69e22b87e93" exitCode=0
Oct 30 19:18:00.365354 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:00.364985    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c3362e0-mn964" event={"ID":"fc981614-0ce4-468c-9457-c972e4caf4d7","Type":"ContainerDied","Data":"3035ec79edd632ae1d1b39b42fe2e05b6df8869a00e9c4475b82e69e22b87e93"}
Oct 30 19:18:01.529598 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:01.529568    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c3362e0-mn964"
Oct 30 19:18:01.622170 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:01.622135    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc981614-0ce4-468c-9457-c972e4caf4d7-kube-api-access-nrc9q\" (UniqueName: \"kubernetes.io/projected/fc981614-0ce4-468c-9457-c972e4caf4d7-kube-api-access-nrc9q\") pod \"fc981614-0ce4-468c-9457-c972e4caf4d7\" (UID: \"fc981614-0ce4-468c-9457-c972e4caf4d7\") "
Oct 30 19:18:01.622374 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:01.622193    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc981614-0ce4-468c-9457-c972e4caf4d7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc981614-0ce4-468c-9457-c972e4caf4d7-maas-api-xks-ca\") pod \"fc981614-0ce4-468c-9457-c972e4caf4d7\" (UID: \"fc981614-0ce4-468c-9457-c972e4caf4d7\") "
Oct 30 19:18:01.624496 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:01.624425    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc981614-0ce4-468c-9457-c972e4caf4d7-maas-api-xks-ca" pod "fc981614-0ce4-468c-9457-c972e4caf4d7" (UID: "fc981614-0ce4-468c-9457-c972e4caf4d7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:18:01.624496 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:01.624462    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc981614-0ce4-468c-9457-c972e4caf4d7-kube-api-access-nrc9q" pod "fc981614-0ce4-468c-9457-c972e4caf4d7" (UID: "fc981614-0ce4-468c-9457-c972e4caf4d7"). InnerVolumeSpecName "kube-api-access-nrc9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:18:01.723011 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:01.722974    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc981614-0ce4-468c-9457-c972e4caf4d7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc981614-0ce4-468c-9457-c972e4caf4d7-maas-api-openshift-ca\") pod \"fc981614-0ce4-468c-9457-c972e4caf4d7\" (UID: \"fc981614-0ce4-468c-9457-c972e4caf4d7\") "
Oct 30 19:18:01.723231 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:01.723220    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrc9q\" (UniqueName: \"kubernetes.io/projected/fc981614-0ce4-468c-9457-c972e4caf4d7-kube-api-access-nrc9q\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:18:01.723322 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:01.723232    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc981614-0ce4-468c-9457-c972e4caf4d7-maas-api-xks-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:18:01.723390 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:01.723329    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc981614-0ce4-468c-9457-c972e4caf4d7-maas-api-openshift-ca" pod "fc981614-0ce4-468c-9457-c972e4caf4d7" (UID: "fc981614-0ce4-468c-9457-c972e4caf4d7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:18:01.824236 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:01.824195    4525 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc981614-0ce4-468c-9457-c972e4caf4d7-maas-api-openshift-ca\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:18:02.381857 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:02.381821    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c3362e0-mn964"
Oct 30 19:18:02.382045 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:02.381854    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c3362e0-mn964" event={"ID":"fc981614-0ce4-468c-9457-c972e4caf4d7","Type":"ContainerDied","Data":"f258057e9bc430af577b8359616f64cd2850457a0fd2959347a3211724a664d7"}
Oct 30 19:18:02.382045 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:02.381924    4525 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f258057e9bc430af577b8359616f64cd2850457a0fd2959347a3211724a664d7"
Oct 30 19:18:07.022063 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:07.022027    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7501d6cf-xbk8t"]
Oct 30 19:18:07.024974 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:07.024948    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7501d6cf-xbk8t"]
Oct 30 19:18:07.545626 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:07.545584    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f526772-60e6-44cf-a476-de704cd4e3e9" path="/var/lib/kubelet/pods/1f526772-60e6-44cf-a476-de704cd4e3e9/volumes"
Oct 30 19:18:07.712927 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:07.712872    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts" podUID="be6d4969-285b-4da0-883e-4847f2cbf3af" containerName="main" containerID="cri-o://8e453ac3401f70c8b0ba71c23f7db0ce5e80f6dc9e49daa3ff4ab12da9b514f0" gracePeriod=45
Oct 30 19:18:07.965868 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:07.965775    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts"
Oct 30 19:18:07.981764 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:07.981735    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be6d4969-285b-4da0-883e-4847f2cbf3af-tls-certs\" (UniqueName: \"kubernetes.io/secret/be6d4969-285b-4da0-883e-4847f2cbf3af-tls-certs\") pod \"be6d4969-285b-4da0-883e-4847f2cbf3af\" (UID: \"be6d4969-285b-4da0-883e-4847f2cbf3af\") "
Oct 30 19:18:07.981916 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:07.981779    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be6d4969-285b-4da0-883e-4847f2cbf3af-home\" (UniqueName: \"kubernetes.io/empty-dir/be6d4969-285b-4da0-883e-4847f2cbf3af-home\") pod \"be6d4969-285b-4da0-883e-4847f2cbf3af\" (UID: \"be6d4969-285b-4da0-883e-4847f2cbf3af\") "
Oct 30 19:18:07.981916 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:07.981802    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be6d4969-285b-4da0-883e-4847f2cbf3af-kube-api-access-42ksl\" (UniqueName: \"kubernetes.io/projected/be6d4969-285b-4da0-883e-4847f2cbf3af-kube-api-access-42ksl\") pod \"be6d4969-285b-4da0-883e-4847f2cbf3af\" (UID: \"be6d4969-285b-4da0-883e-4847f2cbf3af\") "
Oct 30 19:18:07.981916 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:07.981900    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be6d4969-285b-4da0-883e-4847f2cbf3af-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be6d4969-285b-4da0-883e-4847f2cbf3af-model-cache\") pod \"be6d4969-285b-4da0-883e-4847f2cbf3af\" (UID: \"be6d4969-285b-4da0-883e-4847f2cbf3af\") "
Oct 30 19:18:07.982129 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:07.981955    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be6d4969-285b-4da0-883e-4847f2cbf3af-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be6d4969-285b-4da0-883e-4847f2cbf3af-tmp-dir\") pod \"be6d4969-285b-4da0-883e-4847f2cbf3af\" (UID: \"be6d4969-285b-4da0-883e-4847f2cbf3af\") "
Oct 30 19:18:07.982129 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:07.981982    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be6d4969-285b-4da0-883e-4847f2cbf3af-dshm\" (UniqueName: \"kubernetes.io/empty-dir/be6d4969-285b-4da0-883e-4847f2cbf3af-dshm\") pod \"be6d4969-285b-4da0-883e-4847f2cbf3af\" (UID: \"be6d4969-285b-4da0-883e-4847f2cbf3af\") "
Oct 30 19:18:07.982129 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:07.982108    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be6d4969-285b-4da0-883e-4847f2cbf3af-home" pod "be6d4969-285b-4da0-883e-4847f2cbf3af" (UID: "be6d4969-285b-4da0-883e-4847f2cbf3af"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:18:07.982295 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:07.982219    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be6d4969-285b-4da0-883e-4847f2cbf3af-model-cache" pod "be6d4969-285b-4da0-883e-4847f2cbf3af" (UID: "be6d4969-285b-4da0-883e-4847f2cbf3af"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:18:07.982408 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:07.982390    4525 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be6d4969-285b-4da0-883e-4847f2cbf3af-model-cache\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:18:07.982467 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:07.982417    4525 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be6d4969-285b-4da0-883e-4847f2cbf3af-home\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:18:07.982467 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:07.982424    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be6d4969-285b-4da0-883e-4847f2cbf3af-tmp-dir" pod "be6d4969-285b-4da0-883e-4847f2cbf3af" (UID: "be6d4969-285b-4da0-883e-4847f2cbf3af"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:18:07.984588 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:07.984561    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be6d4969-285b-4da0-883e-4847f2cbf3af-kube-api-access-42ksl" pod "be6d4969-285b-4da0-883e-4847f2cbf3af" (UID: "be6d4969-285b-4da0-883e-4847f2cbf3af"). InnerVolumeSpecName "kube-api-access-42ksl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:18:07.984745 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:07.984569    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be6d4969-285b-4da0-883e-4847f2cbf3af-dshm" pod "be6d4969-285b-4da0-883e-4847f2cbf3af" (UID: "be6d4969-285b-4da0-883e-4847f2cbf3af"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:18:07.984951 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:07.984932    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be6d4969-285b-4da0-883e-4847f2cbf3af-tls-certs" pod "be6d4969-285b-4da0-883e-4847f2cbf3af" (UID: "be6d4969-285b-4da0-883e-4847f2cbf3af"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:18:08.083373 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:08.083294    4525 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be6d4969-285b-4da0-883e-4847f2cbf3af-tmp-dir\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:18:08.083373 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:08.083322    4525 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be6d4969-285b-4da0-883e-4847f2cbf3af-dshm\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:18:08.083373 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:08.083331    4525 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be6d4969-285b-4da0-883e-4847f2cbf3af-tls-certs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:18:08.083373 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:08.083340    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42ksl\" (UniqueName: \"kubernetes.io/projected/be6d4969-285b-4da0-883e-4847f2cbf3af-kube-api-access-42ksl\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:18:08.431869 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:08.431834    4525 generic.go:356] "Generic (PLEG): container finished" podID="be6d4969-285b-4da0-883e-4847f2cbf3af" containerID="8e453ac3401f70c8b0ba71c23f7db0ce5e80f6dc9e49daa3ff4ab12da9b514f0" exitCode=0
Oct 30 19:18:08.432057 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:08.431882    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts"
Oct 30 19:18:08.432057 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:08.431926    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts" event={"ID":"be6d4969-285b-4da0-883e-4847f2cbf3af","Type":"ContainerDied","Data":"8e453ac3401f70c8b0ba71c23f7db0ce5e80f6dc9e49daa3ff4ab12da9b514f0"}
Oct 30 19:18:08.432057 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:08.431970    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts" event={"ID":"be6d4969-285b-4da0-883e-4847f2cbf3af","Type":"ContainerDied","Data":"06f0935f6ac358ac4bb171a70bf8b1e6097ff634cc5e95143fea57091d6523db"}
Oct 30 19:18:08.432057 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:08.431986    4525 scope.go:122] "RemoveContainer" containerID="8e453ac3401f70c8b0ba71c23f7db0ce5e80f6dc9e49daa3ff4ab12da9b514f0"
Oct 30 19:18:08.443322 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:08.443303    4525 scope.go:122] "RemoveContainer" containerID="8e453ac3401f70c8b0ba71c23f7db0ce5e80f6dc9e49daa3ff4ab12da9b514f0"
Oct 30 19:18:08.443602 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:18:08.443579    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e453ac3401f70c8b0ba71c23f7db0ce5e80f6dc9e49daa3ff4ab12da9b514f0\": container with ID starting with 8e453ac3401f70c8b0ba71c23f7db0ce5e80f6dc9e49daa3ff4ab12da9b514f0 not found: ID does not exist" containerID="8e453ac3401f70c8b0ba71c23f7db0ce5e80f6dc9e49daa3ff4ab12da9b514f0"
Oct 30 19:18:08.443657 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:08.443611    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e453ac3401f70c8b0ba71c23f7db0ce5e80f6dc9e49daa3ff4ab12da9b514f0"} err="failed to get container status \"8e453ac3401f70c8b0ba71c23f7db0ce5e80f6dc9e49daa3ff4ab12da9b514f0\": rpc error: code = NotFound desc = could not find container \"8e453ac3401f70c8b0ba71c23f7db0ce5e80f6dc9e49daa3ff4ab12da9b514f0\": container with ID starting with 8e453ac3401f70c8b0ba71c23f7db0ce5e80f6dc9e49daa3ff4ab12da9b514f0 not found: ID does not exist"
Oct 30 19:18:08.454208 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:08.454186    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts"]
Oct 30 19:18:08.460326 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:08.460301    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-85cb41/premium-w1-85cb41-kserve-86b576cb74-7zzts"]
Oct 30 19:18:09.543880 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:09.543846    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be6d4969-285b-4da0-883e-4847f2cbf3af" path="/var/lib/kubelet/pods/be6d4969-285b-4da0-883e-4847f2cbf3af/volumes"
Oct 30 19:18:13.878234 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:13.878196    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-85cb41-5f5wm"]
Oct 30 19:18:13.883724 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:13.883662    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-85cb41-5f5wm"]
Oct 30 19:18:15.542847 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:15.542815    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0c4cdba-b018-48f1-9db4-8c9a802632ef" path="/var/lib/kubelet/pods/d0c4cdba-b018-48f1-9db4-8c9a802632ef/volumes"
Oct 30 19:18:17.023993 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:17.023949    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e703ad97-92hq4"]
Oct 30 19:18:17.026917 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:17.026890    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e703ad97-92hq4"]
Oct 30 19:18:17.397515 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:18:17.397475    4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d49l" containerName="extractor"
Oct 30 19:18:17.544117 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:17.544084    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b46ae00-fd8c-4575-b843-7ca560d0f12c" path="/var/lib/kubelet/pods/2b46ae00-fd8c-4575-b843-7ca560d0f12c/volumes"
Oct 30 19:18:20.723108 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:20.723055    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-fz9ck"]
Oct 30 19:18:20.723861 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:20.723428    4525 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-fz9ck" podUID="f236b7d6-975d-447f-bcaa-8e51d6ee0fc9" containerName="istio-proxy" containerID="cri-o://4f207c5afaead434d93f0cb10b449b06795b0d9268e5256d05849de231369eb5" gracePeriod=30
Oct 30 19:18:21.825391 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:21.825354    4525 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-58966b5867-lsc7q_30bd1d87-7569-4d72-9898-13990914ca4a/authorino/0.log"
Oct 30 19:18:22.027651 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:22.027609    4525 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-xwr2j_c5d9d0df-f991-42dc-8889-1bed5e55ed4e/manager/0.log"
Oct 30 19:18:22.232953 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:22.232926    4525 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-k569b_6eaca66e-2608-40c2-a989-ea16d6b4eb80/manager/0.log"
Oct 30 19:18:23.980357 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:23.980326    4525 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-w68l9_ef0c72de-55ce-4860-89a0-e4bad58e11bb/manager/0.log"
Oct 30 19:18:24.908805 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:24.908779    4525 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-qtvgz_1e8fda7d-c348-4818-92bb-89fb11836b50/manager/0.log"
Oct 30 19:18:25.095025 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:25.095000    4525 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-ls657_0c027763-f795-4d38-8c7f-f366f8ae47c6/manager/0.log"
Oct 30 19:18:25.776766 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:25.776741    4525 scope.go:122] "RemoveContainer" containerID="8535394a809a648f1856e311f611c42f46389279f6e1bb1c4b926d9548528fa3"
Oct 30 19:18:25.837017 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:25.836988    4525 scope.go:122] "RemoveContainer" containerID="05ed619cb3a472a3e5421f05c6e998770f19035f900d1ca5c48e89d8ace4ce66"
Oct 30 19:18:25.846658 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:25.846627    4525 scope.go:122] "RemoveContainer" containerID="5ba9ca05a407e818624c09d442b6e49af8bacb1de7ec962516e79e121990b961"
Oct 30 19:18:25.982859 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:25.982834    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-fz9ck"
Oct 30 19:18:26.068049 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:26.067972    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f236b7d6-975d-447f-bcaa-8e51d6ee0fc9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f236b7d6-975d-447f-bcaa-8e51d6ee0fc9-workload-certs\") pod \"f236b7d6-975d-447f-bcaa-8e51d6ee0fc9\" (UID: \"f236b7d6-975d-447f-bcaa-8e51d6ee0fc9\") "
Oct 30 19:18:26.068049 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:26.068030    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f236b7d6-975d-447f-bcaa-8e51d6ee0fc9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f236b7d6-975d-447f-bcaa-8e51d6ee0fc9-credential-socket\") pod \"f236b7d6-975d-447f-bcaa-8e51d6ee0fc9\" (UID: \"f236b7d6-975d-447f-bcaa-8e51d6ee0fc9\") "
Oct 30 19:18:26.068288 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:26.068081    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f236b7d6-975d-447f-bcaa-8e51d6ee0fc9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f236b7d6-975d-447f-bcaa-8e51d6ee0fc9-istio-envoy\") pod \"f236b7d6-975d-447f-bcaa-8e51d6ee0fc9\" (UID: \"f236b7d6-975d-447f-bcaa-8e51d6ee0fc9\") "
Oct 30 19:18:26.068288 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:26.068125    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f236b7d6-975d-447f-bcaa-8e51d6ee0fc9-kube-api-access-xg8zb\" (UniqueName: \"kubernetes.io/projected/f236b7d6-975d-447f-bcaa-8e51d6ee0fc9-kube-api-access-xg8zb\") pod \"f236b7d6-975d-447f-bcaa-8e51d6ee0fc9\" (UID: \"f236b7d6-975d-447f-bcaa-8e51d6ee0fc9\") "
Oct 30 19:18:26.068288 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:26.068156    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f236b7d6-975d-447f-bcaa-8e51d6ee0fc9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f236b7d6-975d-447f-bcaa-8e51d6ee0fc9-workload-socket\") pod \"f236b7d6-975d-447f-bcaa-8e51d6ee0fc9\" (UID: \"f236b7d6-975d-447f-bcaa-8e51d6ee0fc9\") "
Oct 30 19:18:26.068288 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:26.068196    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f236b7d6-975d-447f-bcaa-8e51d6ee0fc9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f236b7d6-975d-447f-bcaa-8e51d6ee0fc9-istiod-ca-cert\") pod \"f236b7d6-975d-447f-bcaa-8e51d6ee0fc9\" (UID: \"f236b7d6-975d-447f-bcaa-8e51d6ee0fc9\") "
Oct 30 19:18:26.068288 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:26.068226    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f236b7d6-975d-447f-bcaa-8e51d6ee0fc9-istio-token\" (UniqueName: \"kubernetes.io/projected/f236b7d6-975d-447f-bcaa-8e51d6ee0fc9-istio-token\") pod \"f236b7d6-975d-447f-bcaa-8e51d6ee0fc9\" (UID: \"f236b7d6-975d-447f-bcaa-8e51d6ee0fc9\") "
Oct 30 19:18:26.068288 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:26.068254    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f236b7d6-975d-447f-bcaa-8e51d6ee0fc9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f236b7d6-975d-447f-bcaa-8e51d6ee0fc9-istio-data\") pod \"f236b7d6-975d-447f-bcaa-8e51d6ee0fc9\" (UID: \"f236b7d6-975d-447f-bcaa-8e51d6ee0fc9\") "
Oct 30 19:18:26.068650 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:26.068340    4525 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f236b7d6-975d-447f-bcaa-8e51d6ee0fc9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f236b7d6-975d-447f-bcaa-8e51d6ee0fc9-istio-podinfo\") pod \"f236b7d6-975d-447f-bcaa-8e51d6ee0fc9\" (UID: \"f236b7d6-975d-447f-bcaa-8e51d6ee0fc9\") "
Oct 30 19:18:26.068650 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:26.068333    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f236b7d6-975d-447f-bcaa-8e51d6ee0fc9-workload-certs" pod "f236b7d6-975d-447f-bcaa-8e51d6ee0fc9" (UID: "f236b7d6-975d-447f-bcaa-8e51d6ee0fc9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:18:26.068650 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:26.068468    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f236b7d6-975d-447f-bcaa-8e51d6ee0fc9-credential-socket" pod "f236b7d6-975d-447f-bcaa-8e51d6ee0fc9" (UID: "f236b7d6-975d-447f-bcaa-8e51d6ee0fc9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:18:26.068650 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:26.068546    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f236b7d6-975d-447f-bcaa-8e51d6ee0fc9-workload-socket" pod "f236b7d6-975d-447f-bcaa-8e51d6ee0fc9" (UID: "f236b7d6-975d-447f-bcaa-8e51d6ee0fc9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:18:26.068922 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:26.068712    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f236b7d6-975d-447f-bcaa-8e51d6ee0fc9-istio-data" pod "f236b7d6-975d-447f-bcaa-8e51d6ee0fc9" (UID: "f236b7d6-975d-447f-bcaa-8e51d6ee0fc9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:18:26.068922 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:26.068885    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f236b7d6-975d-447f-bcaa-8e51d6ee0fc9-istiod-ca-cert" pod "f236b7d6-975d-447f-bcaa-8e51d6ee0fc9" (UID: "f236b7d6-975d-447f-bcaa-8e51d6ee0fc9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:18:26.069174 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:26.068886    4525 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f236b7d6-975d-447f-bcaa-8e51d6ee0fc9-workload-certs\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:18:26.069174 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:26.069073    4525 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f236b7d6-975d-447f-bcaa-8e51d6ee0fc9-credential-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:18:26.069174 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:26.069088    4525 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f236b7d6-975d-447f-bcaa-8e51d6ee0fc9-workload-socket\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:18:26.069174 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:26.069102    4525 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f236b7d6-975d-447f-bcaa-8e51d6ee0fc9-istio-data\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:18:26.070500 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:26.070473    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f236b7d6-975d-447f-bcaa-8e51d6ee0fc9-istio-envoy" pod "f236b7d6-975d-447f-bcaa-8e51d6ee0fc9" (UID: "f236b7d6-975d-447f-bcaa-8e51d6ee0fc9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:18:26.070720 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:26.070663    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f236b7d6-975d-447f-bcaa-8e51d6ee0fc9-kube-api-access-xg8zb" pod "f236b7d6-975d-447f-bcaa-8e51d6ee0fc9" (UID: "f236b7d6-975d-447f-bcaa-8e51d6ee0fc9"). InnerVolumeSpecName "kube-api-access-xg8zb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:18:26.071110 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:26.071086    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f236b7d6-975d-447f-bcaa-8e51d6ee0fc9-istio-podinfo" pod "f236b7d6-975d-447f-bcaa-8e51d6ee0fc9" (UID: "f236b7d6-975d-447f-bcaa-8e51d6ee0fc9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:18:26.071110 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:26.071101    4525 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f236b7d6-975d-447f-bcaa-8e51d6ee0fc9-istio-token" pod "f236b7d6-975d-447f-bcaa-8e51d6ee0fc9" (UID: "f236b7d6-975d-447f-bcaa-8e51d6ee0fc9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:18:26.169630 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:26.169601    4525 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f236b7d6-975d-447f-bcaa-8e51d6ee0fc9-istio-podinfo\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:18:26.169630 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:26.169625    4525 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f236b7d6-975d-447f-bcaa-8e51d6ee0fc9-istio-envoy\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:18:26.169630 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:26.169637    4525 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xg8zb\" (UniqueName: \"kubernetes.io/projected/f236b7d6-975d-447f-bcaa-8e51d6ee0fc9-kube-api-access-xg8zb\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:18:26.170078 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:26.169647    4525 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f236b7d6-975d-447f-bcaa-8e51d6ee0fc9-istiod-ca-cert\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:18:26.170078 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:26.169655    4525 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f236b7d6-975d-447f-bcaa-8e51d6ee0fc9-istio-token\") on node \"ip-10-0-137-135.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 19:18:26.318963 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:26.318892    4525 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-lxnnq_6a9d3618-5a4c-40aa-839e-002d218bd69b/manager/0.log"
Oct 30 19:18:26.565263 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:26.565231    4525 generic.go:356] "Generic (PLEG): container finished" podID="f236b7d6-975d-447f-bcaa-8e51d6ee0fc9" containerID="4f207c5afaead434d93f0cb10b449b06795b0d9268e5256d05849de231369eb5" exitCode=0
Oct 30 19:18:26.565452 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:26.565276    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-fz9ck" event={"ID":"f236b7d6-975d-447f-bcaa-8e51d6ee0fc9","Type":"ContainerDied","Data":"4f207c5afaead434d93f0cb10b449b06795b0d9268e5256d05849de231369eb5"}
Oct 30 19:18:26.565452 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:26.565288    4525 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-fz9ck"
Oct 30 19:18:26.565452 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:26.565315    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-fz9ck" event={"ID":"f236b7d6-975d-447f-bcaa-8e51d6ee0fc9","Type":"ContainerDied","Data":"aefef72c23ac7e9a4862c88887060e09a76a2cf3e2e4f54a777f343952fb2fde"}
Oct 30 19:18:26.565452 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:26.565338    4525 scope.go:122] "RemoveContainer" containerID="4f207c5afaead434d93f0cb10b449b06795b0d9268e5256d05849de231369eb5"
Oct 30 19:18:26.577018 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:26.576998    4525 scope.go:122] "RemoveContainer" containerID="d181f6236df07f838dc96fb118ef6348173a3112ca2f7a7930028c1a496140db"
Oct 30 19:18:26.587208 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:26.587190    4525 scope.go:122] "RemoveContainer" containerID="4f207c5afaead434d93f0cb10b449b06795b0d9268e5256d05849de231369eb5"
Oct 30 19:18:26.587514 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:18:26.587496    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f207c5afaead434d93f0cb10b449b06795b0d9268e5256d05849de231369eb5\": container with ID starting with 4f207c5afaead434d93f0cb10b449b06795b0d9268e5256d05849de231369eb5 not found: ID does not exist" containerID="4f207c5afaead434d93f0cb10b449b06795b0d9268e5256d05849de231369eb5"
Oct 30 19:18:26.587572 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:26.587524    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f207c5afaead434d93f0cb10b449b06795b0d9268e5256d05849de231369eb5"} err="failed to get container status \"4f207c5afaead434d93f0cb10b449b06795b0d9268e5256d05849de231369eb5\": rpc error: code = NotFound desc = could not find container \"4f207c5afaead434d93f0cb10b449b06795b0d9268e5256d05849de231369eb5\": container with ID starting with 4f207c5afaead434d93f0cb10b449b06795b0d9268e5256d05849de231369eb5 not found: ID does not exist"
Oct 30 19:18:26.587572 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:26.587543    4525 scope.go:122] "RemoveContainer" containerID="d181f6236df07f838dc96fb118ef6348173a3112ca2f7a7930028c1a496140db"
Oct 30 19:18:26.587865 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:18:26.587845    4525 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d181f6236df07f838dc96fb118ef6348173a3112ca2f7a7930028c1a496140db\": container with ID starting with d181f6236df07f838dc96fb118ef6348173a3112ca2f7a7930028c1a496140db not found: ID does not exist" containerID="d181f6236df07f838dc96fb118ef6348173a3112ca2f7a7930028c1a496140db"
Oct 30 19:18:26.587920 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:26.587870    4525 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d181f6236df07f838dc96fb118ef6348173a3112ca2f7a7930028c1a496140db"} err="failed to get container status \"d181f6236df07f838dc96fb118ef6348173a3112ca2f7a7930028c1a496140db\": rpc error: code = NotFound desc = could not find container \"d181f6236df07f838dc96fb118ef6348173a3112ca2f7a7930028c1a496140db\": container with ID starting with d181f6236df07f838dc96fb118ef6348173a3112ca2f7a7930028c1a496140db not found: ID does not exist"
Oct 30 19:18:26.589306 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:26.589287    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-fz9ck"]
Oct 30 19:18:26.596980 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:26.596959    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-85cb41-openshift-default-74bfbc4465-fz9ck"]
Oct 30 19:18:27.284531 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:27.284498    4525 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-58966b5867-lsc7q_30bd1d87-7569-4d72-9898-13990914ca4a/authorino/0.log"
Oct 30 19:18:27.543033 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:27.542949    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f236b7d6-975d-447f-bcaa-8e51d6ee0fc9" path="/var/lib/kubelet/pods/f236b7d6-975d-447f-bcaa-8e51d6ee0fc9/volumes"
Oct 30 19:18:31.027192 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:31.027155    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f009fff9-76lm9"]
Oct 30 19:18:31.029997 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:31.029968    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f009fff9-76lm9"]
Oct 30 19:18:31.543395 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:31.543360    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52c00ed7-ee9c-422c-964d-e6c41b832d71" path="/var/lib/kubelet/pods/52c00ed7-ee9c-422c-964d-e6c41b832d71/volumes"
Oct 30 19:18:37.931970 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:37.931940    4525 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-qtvgz_1e8fda7d-c348-4818-92bb-89fb11836b50/manager/0.log"
Oct 30 19:18:38.112432 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:38.112397    4525 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-ls657_0c027763-f795-4d38-8c7f-f366f8ae47c6/manager/0.log"
Oct 30 19:18:39.297791 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:39.297760    4525 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-lxnnq_6a9d3618-5a4c-40aa-839e-002d218bd69b/manager/0.log"
Oct 30 19:18:40.460623 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:40.460585    4525 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-14c2b681-cvbkd_bb17e9a7-c339-431a-bac6-f5d21ae854a9/revoke-keys/0.log"
Oct 30 19:18:40.640578 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:40.640545    4525 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-1c3362e0-mn964_fc981614-0ce4-468c-9457-c972e4caf4d7/revoke-keys/0.log"
Oct 30 19:18:40.816738 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:40.816643    4525 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-2641515c-vf5vk_399c5bbf-b0b3-49a1-ae99-6bc544052d99/revoke-keys/0.log"
Oct 30 19:18:40.994855 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:40.994827    4525 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-2f3a3b99-57p29_eda0d3b8-cfbd-44bf-940d-200d449dec9a/revoke-keys/0.log"
Oct 30 19:18:41.350138 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:41.350108    4525 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-743e4446-jtlb9_0f675df4-123a-412a-8aa3-5a61497b06bd/revoke-keys/0.log"
Oct 30 19:18:41.704048 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:41.704010    4525 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-856e5336-gxwh4_e25fa0f4-be7f-4c10-83f8-ce8b479668cb/revoke-keys/0.log"
Oct 30 19:18:41.873136 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:41.873101    4525 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-868e180c-xqlh2_f2ae470b-e56c-49f2-ae97-f0a4ec3a98fa/revoke-keys/0.log"
Oct 30 19:18:42.048190 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:42.048102    4525 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-90343bb1-k4hvz_9cafe781-d9a6-4a40-b632-9dc7baf6a243/revoke-keys/0.log"
Oct 30 19:18:42.227444 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:42.227418    4525 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-a5e80b74-9sj46_70641967-e202-48ef-bdbd-d03939412444/revoke-keys/0.log"
Oct 30 19:18:42.394611 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:42.394577    4525 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-c8684c15-vf9cs_16cc85e9-28d0-45d4-ac48-6f15f18399a3/revoke-keys/0.log"
Oct 30 19:18:42.569818 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:42.569788    4525 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f2b0c944-ctpw6_b0781052-ad77-4573-bfea-16b96fbc3c5b/revoke-keys/0.log"
Oct 30 19:18:42.745671 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:42.745593    4525 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-fb4b9fc4-x5727_0a2b88cf-5484-4765-ac9e-87917569f9f4/revoke-keys/0.log"
Oct 30 19:18:42.920461 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:42.920430    4525 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-vw6gn_834bfbdb-e855-429d-9699-60f3b7cf82c9/postgres/0.log"
Oct 30 19:18:44.346832 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:44.346799    4525 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-58966b5867-lsc7q_30bd1d87-7569-4d72-9898-13990914ca4a/authorino/0.log"
Oct 30 19:18:44.535889 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:44.535863    4525 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-xwr2j_c5d9d0df-f991-42dc-8889-1bed5e55ed4e/manager/0.log"
Oct 30 19:18:44.729056 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:44.729026    4525 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-k569b_6eaca66e-2608-40c2-a989-ea16d6b4eb80/manager/0.log"
Oct 30 19:18:46.087879 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:46.087847    4525 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-w68l9_ef0c72de-55ce-4860-89a0-e4bad58e11bb/manager/0.log"
Oct 30 19:18:46.848326 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:46.848293    4525 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7d4ffc574frqzst_b7b2003d-590d-4a27-a88d-3fb30e1133c4/istio-proxy/0.log"
Oct 30 19:18:47.227835 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:47.227793    4525 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-hncld_ee5fed99-83c8-46d3-a07c-b1db0da31534/discovery/0.log"
Oct 30 19:18:47.397969 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:18:47.397928    4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d49l" containerName="extractor"
Oct 30 19:18:47.636744 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:47.636711    4525 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-wlrxd_aa550fe6-5b53-439d-9c72-391ed15e1fb3/discovery/0.log"
Oct 30 19:18:48.877833 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:48.877804    4525 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-68cb9b4b95-sgtll_85ce0a7f-b7fb-4909-bfd6-affca6da6d49/istio-proxy/1.log"
Oct 30 19:18:49.052153 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:49.052126    4525 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-68cb9b4b95-tckhm_23ad1447-7842-4b64-b99f-adf61a803291/istio-proxy/0.log"
Oct 30 19:18:49.232381 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:49.232292    4525 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-864cbfbb6c-qhbfx_d9a5292b-dacb-4a0f-af6d-f3da1b7843c2/istio-proxy/0.log"
Oct 30 19:18:50.229139 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:50.229105    4525 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-68cb9b4b95-sgtll_85ce0a7f-b7fb-4909-bfd6-affca6da6d49/istio-proxy/0.log"
Oct 30 19:18:50.900920 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:50.900883    4525 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-zgw99_72bab614-8d2f-4640-9d16-b54f05759572/main/0.log"
Oct 30 19:18:51.443611 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:51.443581    4525 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fh28d_e396973d-d477-43a8-a781-4d3b9d2044ad/main/0.log"
Oct 30 19:18:51.794540 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:51.794465    4525 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-nmscq_787150b8-33c2-41a6-b54c-e35fd0d27bef/main/0.log"
Oct 30 19:18:51.975263 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:18:51.975205    4525 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-v6bvb_12ebc8b9-c794-47b5-b87b-a63c4bd993e6/main/0.log"
Oct 30 19:19:07.754428 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:07.754351    4525 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-vhtmb_7c3ae3e0-ddf1-414f-bb79-e4afc0527b26/global-pull-secret-syncer/0.log"
Oct 30 19:19:08.030792 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:08.030708    4525 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-dbbbq_bd08a0fa-46bc-4250-985c-d2580a2225b8/konnectivity-agent/0.log"
Oct 30 19:19:08.185439 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:08.185406    4525 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-135.us-east-2.compute.internal_826c9de932b6871da80993c953a89ebf/haproxy/0.log"
Oct 30 19:19:15.030066 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:15.030027    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8684c15-vf9cs"]
Oct 30 19:19:15.032702 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:15.032661    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8684c15-vf9cs"]
Oct 30 19:19:15.544484 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:15.544444    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16cc85e9-28d0-45d4-ac48-6f15f18399a3" path="/var/lib/kubelet/pods/16cc85e9-28d0-45d4-ac48-6f15f18399a3/volumes"
Oct 30 19:19:15.704448 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:15.704415    4525 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-58966b5867-lsc7q_30bd1d87-7569-4d72-9898-13990914ca4a/authorino/0.log"
Oct 30 19:19:15.739731 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:15.739696    4525 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-xwr2j_c5d9d0df-f991-42dc-8889-1bed5e55ed4e/manager/0.log"
Oct 30 19:19:15.767249 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:15.767216    4525 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-k569b_6eaca66e-2608-40c2-a989-ea16d6b4eb80/manager/0.log"
Oct 30 19:19:16.025206 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:16.025169    4525 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14c2b681-cvbkd"]
Oct 30 19:19:16.028463 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:16.028439    4525 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14c2b681-cvbkd"]
Oct 30 19:19:17.087691 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:17.087646    4525 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-w68l9_ef0c72de-55ce-4860-89a0-e4bad58e11bb/manager/0.log"
Oct 30 19:19:17.397781 ip-10-0-137-135 kubenswrapper[4525]: E1030 19:19:17.397744    4525 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4d49l" containerName="extractor"
Oct 30 19:19:17.544324 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:17.544293    4525 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb17e9a7-c339-431a-bac6-f5d21ae854a9" path="/var/lib/kubelet/pods/bb17e9a7-c339-431a-bac6-f5d21ae854a9/volumes"
Oct 30 19:19:18.992287 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:18.992259    4525 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-xl7v4_47a5f9fe-b519-43a6-9973-4479f504421f/monitoring-plugin/0.log"
Oct 30 19:19:19.190643 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:19.190612    4525 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rwkfn_62619ba0-8477-4864-a107-7c2fd2de3273/node-exporter/0.log"
Oct 30 19:19:19.215249 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:19.215227    4525 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rwkfn_62619ba0-8477-4864-a107-7c2fd2de3273/kube-rbac-proxy/0.log"
Oct 30 19:19:19.234947 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:19.234911    4525 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rwkfn_62619ba0-8477-4864-a107-7c2fd2de3273/init-textfile/0.log"
Oct 30 19:19:19.602104 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:19.602080    4525 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-86qts_7815c72b-c24d-4720-9400-2482edce2255/prometheus-operator-admission-webhook/0.log"
Oct 30 19:19:19.926195 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:19.925642    4525 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-47xdf/perf-node-gather-daemonset-p2mm6"]
Oct 30 19:19:19.926396 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:19.926265    4525 cpu_manager.go:414] "containerMap: removing container" podUID="f236b7d6-975d-447f-bcaa-8e51d6ee0fc9" containerName="istio-proxy"
Oct 30 19:19:19.926396 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:19.926284    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f236b7d6-975d-447f-bcaa-8e51d6ee0fc9" containerName="istio-proxy"
Oct 30 19:19:19.926396 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:19.926298    4525 cpu_manager.go:414] "containerMap: removing container" podUID="fc981614-0ce4-468c-9457-c972e4caf4d7" containerName="revoke-keys"
Oct 30 19:19:19.926396 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:19.926306    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc981614-0ce4-468c-9457-c972e4caf4d7" containerName="revoke-keys"
Oct 30 19:19:19.926396 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:19.926318    4525 cpu_manager.go:414] "containerMap: removing container" podUID="f236b7d6-975d-447f-bcaa-8e51d6ee0fc9" containerName="istio-proxy"
Oct 30 19:19:19.926396 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:19.926325    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f236b7d6-975d-447f-bcaa-8e51d6ee0fc9" containerName="istio-proxy"
Oct 30 19:19:19.926396 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:19.926343    4525 cpu_manager.go:414] "containerMap: removing container" podUID="be6d4969-285b-4da0-883e-4847f2cbf3af" containerName="main"
Oct 30 19:19:19.926396 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:19.926349    4525 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be6d4969-285b-4da0-883e-4847f2cbf3af" containerName="main"
Oct 30 19:19:19.926876 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:19.926437    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="f236b7d6-975d-447f-bcaa-8e51d6ee0fc9" containerName="istio-proxy"
Oct 30 19:19:19.926876 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:19.926449    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc981614-0ce4-468c-9457-c972e4caf4d7" containerName="revoke-keys"
Oct 30 19:19:19.926876 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:19.926459    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="f236b7d6-975d-447f-bcaa-8e51d6ee0fc9" containerName="istio-proxy"
Oct 30 19:19:19.926876 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:19.926469    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="be6d4969-285b-4da0-883e-4847f2cbf3af" containerName="main"
Oct 30 19:19:19.926876 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:19.926479    4525 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ec64e5e-2c2b-47d8-8070-296ce41ada09" containerName="istio-proxy"
Oct 30 19:19:19.930149 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:19.930124    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-47xdf/perf-node-gather-daemonset-p2mm6"
Oct 30 19:19:19.938974 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:19.938589    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-47xdf/perf-node-gather-daemonset-p2mm6"]
Oct 30 19:19:20.013511 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:20.013272    4525 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-47xdf\"/\"default-dockercfg-xpwfn\""
Oct 30 19:19:20.027001 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:20.026977    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-47xdf\"/\"kube-root-ca.crt\""
Oct 30 19:19:20.027149 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:20.027006    4525 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-47xdf\"/\"openshift-service-ca.crt\""
Oct 30 19:19:20.071269 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:20.071242    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/85411bf7-a1f2-49c8-a6e5-8f25c4cb997e-lib-modules\") pod \"perf-node-gather-daemonset-p2mm6\" (UID: \"85411bf7-a1f2-49c8-a6e5-8f25c4cb997e\") " pod="openshift-must-gather-47xdf/perf-node-gather-daemonset-p2mm6"
Oct 30 19:19:20.071403 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:20.071272    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pk8c\" (UniqueName: \"kubernetes.io/projected/85411bf7-a1f2-49c8-a6e5-8f25c4cb997e-kube-api-access-9pk8c\") pod \"perf-node-gather-daemonset-p2mm6\" (UID: \"85411bf7-a1f2-49c8-a6e5-8f25c4cb997e\") " pod="openshift-must-gather-47xdf/perf-node-gather-daemonset-p2mm6"
Oct 30 19:19:20.071403 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:20.071304    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/85411bf7-a1f2-49c8-a6e5-8f25c4cb997e-podres\") pod \"perf-node-gather-daemonset-p2mm6\" (UID: \"85411bf7-a1f2-49c8-a6e5-8f25c4cb997e\") " pod="openshift-must-gather-47xdf/perf-node-gather-daemonset-p2mm6"
Oct 30 19:19:20.071486 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:20.071396    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/85411bf7-a1f2-49c8-a6e5-8f25c4cb997e-proc\") pod \"perf-node-gather-daemonset-p2mm6\" (UID: \"85411bf7-a1f2-49c8-a6e5-8f25c4cb997e\") " pod="openshift-must-gather-47xdf/perf-node-gather-daemonset-p2mm6"
Oct 30 19:19:20.071486 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:20.071466    4525 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/85411bf7-a1f2-49c8-a6e5-8f25c4cb997e-sys\") pod \"perf-node-gather-daemonset-p2mm6\" (UID: \"85411bf7-a1f2-49c8-a6e5-8f25c4cb997e\") " pod="openshift-must-gather-47xdf/perf-node-gather-daemonset-p2mm6"
Oct 30 19:19:20.172582 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:20.172546    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/85411bf7-a1f2-49c8-a6e5-8f25c4cb997e-lib-modules\") pod \"perf-node-gather-daemonset-p2mm6\" (UID: \"85411bf7-a1f2-49c8-a6e5-8f25c4cb997e\") " pod="openshift-must-gather-47xdf/perf-node-gather-daemonset-p2mm6"
Oct 30 19:19:20.172582 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:20.172582    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pk8c\" (UniqueName: \"kubernetes.io/projected/85411bf7-a1f2-49c8-a6e5-8f25c4cb997e-kube-api-access-9pk8c\") pod \"perf-node-gather-daemonset-p2mm6\" (UID: \"85411bf7-a1f2-49c8-a6e5-8f25c4cb997e\") " pod="openshift-must-gather-47xdf/perf-node-gather-daemonset-p2mm6"
Oct 30 19:19:20.172842 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:20.172606    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/85411bf7-a1f2-49c8-a6e5-8f25c4cb997e-podres\") pod \"perf-node-gather-daemonset-p2mm6\" (UID: \"85411bf7-a1f2-49c8-a6e5-8f25c4cb997e\") " pod="openshift-must-gather-47xdf/perf-node-gather-daemonset-p2mm6"
Oct 30 19:19:20.172842 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:20.172624    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/85411bf7-a1f2-49c8-a6e5-8f25c4cb997e-proc\") pod \"perf-node-gather-daemonset-p2mm6\" (UID: \"85411bf7-a1f2-49c8-a6e5-8f25c4cb997e\") " pod="openshift-must-gather-47xdf/perf-node-gather-daemonset-p2mm6"
Oct 30 19:19:20.172842 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:20.172669    4525 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/85411bf7-a1f2-49c8-a6e5-8f25c4cb997e-sys\") pod \"perf-node-gather-daemonset-p2mm6\" (UID: \"85411bf7-a1f2-49c8-a6e5-8f25c4cb997e\") " pod="openshift-must-gather-47xdf/perf-node-gather-daemonset-p2mm6"
Oct 30 19:19:20.172842 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:20.172700    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/85411bf7-a1f2-49c8-a6e5-8f25c4cb997e-lib-modules\") pod \"perf-node-gather-daemonset-p2mm6\" (UID: \"85411bf7-a1f2-49c8-a6e5-8f25c4cb997e\") " pod="openshift-must-gather-47xdf/perf-node-gather-daemonset-p2mm6"
Oct 30 19:19:20.172842 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:20.172761    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/85411bf7-a1f2-49c8-a6e5-8f25c4cb997e-podres\") pod \"perf-node-gather-daemonset-p2mm6\" (UID: \"85411bf7-a1f2-49c8-a6e5-8f25c4cb997e\") " pod="openshift-must-gather-47xdf/perf-node-gather-daemonset-p2mm6"
Oct 30 19:19:20.172842 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:20.172804    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/85411bf7-a1f2-49c8-a6e5-8f25c4cb997e-proc\") pod \"perf-node-gather-daemonset-p2mm6\" (UID: \"85411bf7-a1f2-49c8-a6e5-8f25c4cb997e\") " pod="openshift-must-gather-47xdf/perf-node-gather-daemonset-p2mm6"
Oct 30 19:19:20.173076 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:20.172880    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/85411bf7-a1f2-49c8-a6e5-8f25c4cb997e-sys\") pod \"perf-node-gather-daemonset-p2mm6\" (UID: \"85411bf7-a1f2-49c8-a6e5-8f25c4cb997e\") " pod="openshift-must-gather-47xdf/perf-node-gather-daemonset-p2mm6"
Oct 30 19:19:20.180126 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:20.180078    4525 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pk8c\" (UniqueName: \"kubernetes.io/projected/85411bf7-a1f2-49c8-a6e5-8f25c4cb997e-kube-api-access-9pk8c\") pod \"perf-node-gather-daemonset-p2mm6\" (UID: \"85411bf7-a1f2-49c8-a6e5-8f25c4cb997e\") " pod="openshift-must-gather-47xdf/perf-node-gather-daemonset-p2mm6"
Oct 30 19:19:20.247888 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:20.247857    4525 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-47xdf/perf-node-gather-daemonset-p2mm6"
Oct 30 19:19:20.526444 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:20.526420    4525 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-47xdf/perf-node-gather-daemonset-p2mm6"]
Oct 30 19:19:20.528001 ip-10-0-137-135 kubenswrapper[4525]: W1030 19:19:20.527964    4525 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod85411bf7_a1f2_49c8_a6e5_8f25c4cb997e.slice/crio-c0dc8cdd5387c7cbb9f2ac7ca8081b6390339b812d4693f2ca02b912416aba6b WatchSource:0}: Error finding container c0dc8cdd5387c7cbb9f2ac7ca8081b6390339b812d4693f2ca02b912416aba6b: Status 404 returned error can't find the container with id c0dc8cdd5387c7cbb9f2ac7ca8081b6390339b812d4693f2ca02b912416aba6b
Oct 30 19:19:20.954328 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:20.954284    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-47xdf/perf-node-gather-daemonset-p2mm6" event={"ID":"85411bf7-a1f2-49c8-a6e5-8f25c4cb997e","Type":"ContainerStarted","Data":"f94495c6060f0a835becd5eee1014f80460173c5f5bd78bd777f4b8d973f221a"}
Oct 30 19:19:20.954547 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:20.954350    4525 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-47xdf/perf-node-gather-daemonset-p2mm6" event={"ID":"85411bf7-a1f2-49c8-a6e5-8f25c4cb997e","Type":"ContainerStarted","Data":"c0dc8cdd5387c7cbb9f2ac7ca8081b6390339b812d4693f2ca02b912416aba6b"}
Oct 30 19:19:20.954547 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:20.954372    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-47xdf/perf-node-gather-daemonset-p2mm6"
Oct 30 19:19:20.972477 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:20.972429    4525 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-47xdf/perf-node-gather-daemonset-p2mm6" podStartSLOduration=1.97241396 podStartE2EDuration="1.97241396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:19:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:19:20.970168192 +0000 UTC m=+2583.595838157" watchObservedRunningTime="2026-10-30 19:19:20.97241396 +0000 UTC m=+2583.598083969"
Oct 30 19:19:22.968344 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:22.968313    4525 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-l98wz_356d85d5-c222-49e7-916b-bfd94d2438e1/download-server/0.log"
Oct 30 19:19:24.839757 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:24.839728    4525 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-qtvgz_1e8fda7d-c348-4818-92bb-89fb11836b50/manager/0.log"
Oct 30 19:19:24.895874 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:24.895845    4525 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-ls657_0c027763-f795-4d38-8c7f-f366f8ae47c6/manager/0.log"
Oct 30 19:19:25.131081 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:25.131054    4525 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-qtvgz_1e8fda7d-c348-4818-92bb-89fb11836b50/manager/0.log"
Oct 30 19:19:25.179080 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:25.179052    4525 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-ls657_0c027763-f795-4d38-8c7f-f366f8ae47c6/manager/0.log"
Oct 30 19:19:25.906583 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:25.906554    4525 scope.go:122] "RemoveContainer" containerID="537f0c8c39c9527732edc6d6f161d66f89e01d1f7ad124d0907c52b8824d22a8"
Oct 30 19:19:25.916855 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:25.916837    4525 scope.go:122] "RemoveContainer" containerID="5238ec5221253566eb50004f3a20db3418f46abaa2a8c7737bae9f1e6d0bfcd5"
Oct 30 19:19:25.929418 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:25.929390    4525 scope.go:122] "RemoveContainer" containerID="9b5893c5e59d54301b4ecb3892e96a125537d808ef6b62980a281faffc11aa78"
Oct 30 19:19:25.938415 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:25.938393    4525 scope.go:122] "RemoveContainer" containerID="8509ef207b63c1ec00f3b4c296da65415e618169653d2494cba3b10c7bb56063"
Oct 30 19:19:26.417493 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:26.417467    4525 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lcst7_5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a/dns/0.log"
Oct 30 19:19:26.446271 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:26.446242    4525 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lcst7_5e44ab8a-1e33-4c96-a4d4-9f3c71e5883a/kube-rbac-proxy/0.log"
Oct 30 19:19:26.512394 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:26.512364    4525 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7z9gn_42b71e2d-a8f2-43cf-8487-0e5da2338c54/dns-node-resolver/0.log"
Oct 30 19:19:26.732551 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:26.732472    4525 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-lxnnq_6a9d3618-5a4c-40aa-839e-002d218bd69b/manager/0.log"
Oct 30 19:19:26.913267 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:26.913232    4525 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-lxnnq_6a9d3618-5a4c-40aa-839e-002d218bd69b/manager/0.log"
Oct 30 19:19:26.974973 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:26.974946    4525 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-47xdf/perf-node-gather-daemonset-p2mm6"
Oct 30 19:19:27.456428 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:27.456392    4525 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-b89865498-mr5st_64563635-ed73-4e02-9333-2527c071bb00/registry/0.log"
Oct 30 19:19:27.522245 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:27.522211    4525 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mhtvp_093a372e-74ea-4820-82c8-0dff7c201f10/node-ca/0.log"
Oct 30 19:19:28.027630 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:28.027589    4525 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-qtvgz_1e8fda7d-c348-4818-92bb-89fb11836b50/manager/0.log"
Oct 30 19:19:28.074224 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:28.074193    4525 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-ls657_0c027763-f795-4d38-8c7f-f366f8ae47c6/manager/0.log"
Oct 30 19:19:28.262898 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:28.262870    4525 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-qtvgz_1e8fda7d-c348-4818-92bb-89fb11836b50/manager/0.log"
Oct 30 19:19:28.306895 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:28.306825    4525 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-ls657_0c027763-f795-4d38-8c7f-f366f8ae47c6/manager/0.log"
Oct 30 19:19:29.435578 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:29.435534    4525 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7d4ffc574frqzst_b7b2003d-590d-4a27-a88d-3fb30e1133c4/istio-proxy/0.log"
Oct 30 19:19:29.960664 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:29.960634    4525 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-lxnnq_6a9d3618-5a4c-40aa-839e-002d218bd69b/manager/0.log"
Oct 30 19:19:30.243325 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:30.243252    4525 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-lxnnq_6a9d3618-5a4c-40aa-839e-002d218bd69b/manager/0.log"
Oct 30 19:19:31.606079 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:31.606042    4525 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-qtvgz_1e8fda7d-c348-4818-92bb-89fb11836b50/manager/0.log"
Oct 30 19:19:31.665263 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:31.665228    4525 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-ls657_0c027763-f795-4d38-8c7f-f366f8ae47c6/manager/0.log"
Oct 30 19:19:31.804606 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:31.804575    4525 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-qtvgz_1e8fda7d-c348-4818-92bb-89fb11836b50/manager/0.log"
Oct 30 19:19:31.864052 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:31.863971    4525 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-ls657_0c027763-f795-4d38-8c7f-f366f8ae47c6/manager/0.log"
Oct 30 19:19:32.636887 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:32.636847    4525 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-hncld_ee5fed99-83c8-46d3-a07c-b1db0da31534/discovery/0.log"
Oct 30 19:19:34.051788 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:34.051754    4525 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-lxnnq_6a9d3618-5a4c-40aa-839e-002d218bd69b/manager/0.log"
Oct 30 19:19:34.370004 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:34.369964    4525 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-lxnnq_6a9d3618-5a4c-40aa-839e-002d218bd69b/manager/0.log"
Oct 30 19:19:34.477511 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:34.477484    4525 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7d4ffc574frqzst_b7b2003d-590d-4a27-a88d-3fb30e1133c4/istio-proxy/0.log"
Oct 30 19:19:35.938441 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:35.938406    4525 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7d4ffc574frqzst_b7b2003d-590d-4a27-a88d-3fb30e1133c4/istio-proxy/0.log"
Oct 30 19:19:36.290658 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:36.290572    4525 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7d4ffc574frqzst_b7b2003d-590d-4a27-a88d-3fb30e1133c4/istio-proxy/0.log"
Oct 30 19:19:37.396224 ip-10-0-137-135 kubenswrapper[4525]: I1030 19:19:37.396192    4525 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-hncld_ee5fed99-83c8-46d3-a07c-b1db0da31534/discovery/0.log"